{"id":"ac70c7a8-9474-482d-953c-90a10822ca85","arxiv_id":"2508.14789","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A study's contribution to science can be quantified as the Wasserstein-2 distance between the community's prior and posterior distributions over the effect of interest.","lead":"Researchers propose measuring how much a study teaches a scientific field by taking the Wasserstein-2 distance between the field's prior beliefs and its updated posterior beliefs. The metric gives a quantitative alternative to statistical significance for judging past and future research, but its value depends on how the prior is chosen.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central metric is undefined without a formal community prior: W2 depends so strongly on prior scale that a single study can receive arbitrarily different 'learning' scores, so the headline claim lacks a stable object of measurement.","rationale":"Read in good faith, the paper is a clear proposal with standard math and honest limitations. The strongest claim is that W2 between a community prior and posterior quantifies how much has been learned. For that claim to hold, the community prior must be well-defined; otherwise W2 is not a property of the study. The paper itself exposes the problem in Appendix G and in Section 2. My concern does not contradict the mathematics; it targets the interpretation. The decision-theoretic appendix shows only that small W2 guarantees similar expected utilities for a class of losses; it does not establish that W2 uniquely measures learning, nor does it resolve prior construction. I agree with the reader's weakest assumption: the burden rests on the unelicited community prior. Because this is a gap in operationalization rather than an internal inconsistency, the appropriate verdict remains CONDITIONAL; no adjustment to the reader's verdict is needed.","tokens_in":15530,"tokens_out":5348,"duration_ms":63845,"concrete_test":"Re-analyze the Appendix G example over a grid of defensible priors (μ0 ∈ [0, 0.5], σ0 ∈ [0.2, 0.5]) using Eq. (2) for the normal prior and numerical Wasserstein computation for the truncated-normal prior. Record the resulting W2 values. If the spread exceeds the 0.27–0.34 gap used in the paper, or if study rankings change, the metric is not a stable property of the study. Analytically, the same conclusion follows immediately from W2(N(0,σ0), N(0,1)) = σ0−1, showing unbounded sensitivity to prior dispersion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 defines the learning score as W2(π_old, π_new), but π_old is never operationalized. The paper acknowledges that 'the exact specification of the prior matters' (Section 2) and suggests possible prior sources in the conclusion, yet provides no protocol for constructing the community prior. Appendix G illustrates the consequence: two 'reasonable' priors for the same Hainmueller et al. study give W2 = 0.27 and 0.34, a 26% difference. The problem is structural, not merely elicitation noise. In the normal-normal case, Eq. (2) yields W2 = sqrt((μ1−μ0)^2 + (σ1−σ0)^2). For a fixed study with posterior N(0,1), choosing priors N(0,σ0) gives W2 = σ0−1 for σ0 ≥ 1; as the prior becomes more diffuse, W2 grows without bound. Thus the metric conflates prior ignorance with study-induced learning and is not even defined in the improper-prior limit. Because W2 is input-dependent, the paper's central claim that the metric 'quantifies how much a research community learns' is conditional on an unavailable, possibly arbitrary, prior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Wasserstein-2 distance between a research community's prior distribution and its posterior distribution as a quantitative learning metric, arguing that this distance captures both changes in the location and changes in the dispersion of beliefs after a study. The authors derive the closed-form normal-normal formula, illustrate the metric on stylized examples and on Green et al.'s lawn-sign experiments, and extend the framework to prospective learning by averaging the metric over the prior predictive distribution. They compare the proposal with Lindley's information measure and the KL divergence, and discuss limitations such as scale-dependence and non-cumulativity. The central claim is that W2 offers a simple, interpretable, decision-relevant summary of how much a study contributes to a literature.","tokens_in":15861,"tokens_out":7693,"duration_ms":83695,"significance":"If the proposal is accepted, it would provide editors, funders, and researchers with a single-number supplement to significance testing, and it connects research evaluation to a formal Bayesian decision-theoretic framework. Strengths include the correct and transparent normal-normal formula (Eq. 2), the use of reproducible empirical applications with data and code provided, a clear decision-theoretic motivation in Appendix B, and an unusually candid discussion of the metric's limitations. The prospective-learning extension is a natural and potentially useful contribution. However, the usefulness of the metric hinges on the specification of the community prior, and the manuscript currently lacks a protocol for obtaining that prior; the paper's own Appendix G shows that different reasonable priors give materially different learning scores for the same study. The central prospective formula in Eq. (3) is also garbled as printed. These issues are load-bearing but correctable, so the paper warrants major revision rather than rejection.","major_comments":[{"comment":"The prospective-learning formula is not correctly stated. The displayed expression includes the term σ_prior^2 σ^2 / (n σ_prior^2 + 1), which has incompatible dimensions and does not equal the variance of the posterior mean. The correct term is n σ_prior^4 / (σ^2 + n σ_prior^2); the full expected squared W2 is n σ_prior^4/(σ^2 + n σ_prior^2) + σ_prior^2 (1 − 1/sqrt(1 + n σ_prior^2/σ^2))^2. The notation in Eq. (3) also mixes π_c and π_p before these objects are defined, making the expression unverifiable. Because this is the main analytic result of the prospective section, it must be corrected.","section":"§3.3, Eq. (3)"},{"comment":"The central claim that W2 'quantifies how much a research community learns' is not fully supported without an operational definition of the community prior π_old. Section 2 correctly warns that the prior matters, and the conclusion suggests possible sources, but no elicitation, aggregation, or sensitivity protocol is provided. Appendix G illustrates the consequence: two plausible priors for the Hainmueller et al. study give W2 = 0.27 and W2 = 0.34 for the same study, a 26% difference, and the normal-normal case is even more extreme: for a fixed posterior N(0,1) and priors N(0,σ0), W2 = σ0 − 1, which is unbounded as σ0 grows. The paper's recommendation to normalize by σ0 is mentioned but not made central; without such a normalization, the metric conflates prior ignorance with study-induced learning. The authors need either to provide a concrete procedure for constructing the community pri","section":"§2 and Appendix G"}],"minor_comments":[{"comment":"Lindley's information measure is defined as H(π(θ|y)) − H(π(θ)), which is negative for an informative experiment (e.g., log 5 − log 10 = −0.69), yet Table 2 reports positive values. The authors should either define I as H(π(θ)) − H(π(θ|y)) or use absolute values, and note the convention.","section":"Appendix D, Eq. (6)"},{"comment":"The text states that the KL divergence is 'upper bounded by the surprisal of the study’s outcome S(y) = 1/p(y)'. Surprisal is normally −log p(y), and the stated bound is not generally valid for KL(π(θ|y), π(θ)). This is not central to the W2 recommendation but should be corrected.","section":"Appendix E"},{"comment":"The notation for the prior is inconsistent: the main text uses π(θ), πold := π(ϑ), and π0 interchangeably. Using θ consistently throughout would improve readability.","section":"§2.1 and Appendix A"},{"comment":"Figure 5 is described as showing E_{p_{dc}(y)}[W2], but the analytic bound in Eq. (3) is for E[W2^2]. The figure caption should state whether this quantity is computed exactly, by simulation, or by an approximation such as sqrt(E[W2^2]).","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The self-citation to Mikhaeil et al. (2025) is used to support within-domain comparability claims, but it is an unpublished self-authored manuscript. It is not load-bearing, but in the revision the authors should either provide the relevant results in an appendix or clarify its status. The manuscript is better framed as a measurement proposal with a clear sensitivity-analysis caveat than as a definitive study-level score, given the prior-dependence demonstrated in Appendix G."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the take: the paper is a clearly written proposal to use Wasserstein-2 distance between prior and posterior as a 'learning metric' for evaluating individual studies. The math is simple, the illustrative examples do their job, and the limitations section is honest. What you should know before reading: the metric is not an intrinsic property of a study. It depends on the prior, and the paper doesn't solve the prior-elicitation problem.\n\nThe genuinely new thing is the framing: using W2 for research evaluation rather than the usual KL/Lindley information measures. The prospective extension with consensus and pioneer priors is a nice addition. The paper earns credit for being explicit about non-cumulativity and unit-dependence.\n\nThe load-bearing weakness is exactly what the stress test says. For a fixed posterior N(0,1), choosing prior N(0,σ0) gives W2 = σ0−1 for σ0≥1, which grows without bound as the prior gets more diffuse. The paper says 'the exact specification of the prior matters' (Section 2) and then in Appendix G shows two reasonable priors for the same Hainmueller et al. study producing 0.27 versus 0.34, a 26% difference. That is not a minor detail; it means the headline claim that the metric quantifies how much the community learns is only true conditional on a prior you have to supply from somewhere. The decision-theoretic justification doesn't rescue this either; Theorem 1 just says expectation functionals are continuous in W2, which holds for many other metrics and doesn't uniquely single out W2. Minor: Equation 3 is garbled—the first term should be n σ_prior^4 / (σ^2 + n σ_prior^2)—and the self-citation to Mikhaeil et al. 2025 is not load-bearing.\n\nWho is this for? People in metascience, funding evaluation, Bayesian experimental design. It is a legitimate methodological proposal with a central weakness that is fixable if the authors reframe it as a conditional summary and add a protocol for prior elicitation or comparison across priors. I'd send it to a serious referee; it's not a desk reject.\n\nI would peer review it, and I'd bring it to reading group for the debate about whether such a metric is useful.","headline":"A clean, honest proposal for a W2-based learning metric that is conditional on an unformalized community prior.","tokens_in":16290,"tokens_out":2469,"would_cite":false,"duration_ms":26634,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62C10","62B10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper recommends the Wasserstein-2 distance between a field's prior and posterior as a single number for how much a study teaches the field.","keywords":["Bayesian updating","Wasserstein distance","knowledge accumulation","optimal transport","prior elicitation","research evaluation","prospective learning"],"falsifier":"Take a published study and elicit priors from two independent panels of domain experts using the same elicitation protocol. If the two panels' W2 scores for the same study disagree enough to reverse a judgment about whether the study contributed substantially (as the paper's own citizenship example shows 0.27 versus 0.34 under two reasonable priors), then the metric does not capture an intrinsic property of the study.","tokens_in":15470,"feed_emoji":"📊","tokens_out":5847,"duration_ms":60612,"temperature":0.7,"pith_summary":"This paper proposes a way to put a number on how much a research study contributes to a field: the distance between what the research community believed before the study and what it believes after. The recommended measure is the Wasserstein-2 distance between the prior and the posterior distribution, which in the common normal case reduces to a single formula combining the shift in the estimated effect and the change in uncertainty. If the approach works, editors and funders could compare studies by a transparent learning score rather than by statistical significance or informal impressions. The paper also extends the metric forward, letting decision-makers compute how much they should expect to learn from a proposed study before paying for it.","feed_headline":"One number can rank how much a study teaches a field","feed_subtitle":"Wasserstein-2 distance between prior and posterior turns belief change into a transparent learning score.","key_machinery":"The load-bearing object is the Wasserstein-2 distance W2(πold, πnew), the minimal cost of transporting the probability mass of the prior into the shape of the posterior. For one-dimensional parameters with normal prior and posterior, W2 = sqrt((µ1−µ0)² + (σ1−σ0)²), making learning a function of both the change in the estimated magnitude and the change in uncertainty. The same closed form holds for any location-scale family. The distance also carries a decision-theoretic justification: by Villani's Theorem 6.9, small W2 implies that expected utilities over a broad class of loss functions will be close, so the metric measures how differently the community would judge decisions under prior vers","core_discovery":"The central claim is that learning from a study can be quantified as the Wasserstein-2 distance between the research community's prior distribution over a parameter of interest and the posterior distribution after seeing the study's data. When both distributions are normal this equals the square root of the squared change in the mean plus the squared change in the standard deviation, so moving an estimate and sharpening it both count as learning, and even a null result can produce large learning if it collapses a diffuse prior or overturns a confident one. The paper argues this metric is preferable to Lindley's information measure, which is insensitive to changes in the mean, and it shows th","pith_inferences":["A testable consequence the paper does not run: elicit priors from two independent expert panels for the same study; if the W2 scores disagree enough to change a funding decision, the metric is not an objective property of the study.","The non-cumulative nature of W2 means a literature's total learning is not the sum of its studies' scores; portfolio-level evaluation would need an explicit aggregation rule, such as distance from initial prior to final posterior.","The prospective framework implicitly defines 'surprise' as the Wasserstein distance induced by data under one prior while measuring against another prior; this could be linked to the value-of-information literature, though the paper does not make that connection.","Because the metric is scale-dependent, cross-domain comparisons of learning scores are meaningless; any public ranking of studies would need to be domain-specific or normalized."],"forward_implications":["A journal's significance statement could be supplemented with a one-number learning score computed from an elicited prior and the posterior.","Funders could rank prospective studies by expected W2 over the prior predictive distribution, replacing or complementing power calculations that target significance.","Studies with null results can legitimately score as high-learning—the citizenship experiment scores 0.27 or 0.34 under two reasonable priors—because they compress a divided literature.","Within a substantive domain, learning scores can be compared across studies, and dividing by the prior standard deviation gives a scale-free benchmark where 1 means all prior uncertainty was eliminated.","Because W2 is sensitive to increases in uncertainty, revelations of fraud or bias count as learning just as much as new confirmatory evidence does."],"supporting_citations":[{"why":"Supplies the definition of Wasserstein distance and the continuity theorem used for the decision-theoretic justification that small W2 implies similar expected judgments across utility functions.","marker":"[Villani, 2008]"},{"why":"Defines the additive information measure that the paper contrasts with its own and rejects as insensitive to changes in the posterior mean.","marker":"[Lindley, 1956]"},{"why":"Provides the prior-elicitation methodology the paper relies on for obtaining the community prior that W2 measures from.","marker":"[O'Hagan et al., 2006]"},{"why":"Reviews prior-elicitation methods, supporting the claim that community priors can be extracted from literature or experts.","marker":"[Mikkola et al., 2023]"},{"why":"Supplies the four lawn-sign field experiments used as the main retrospective illustration of stepwise learning scores.","marker":"[Green et al., 2016]"},{"why":"Supplies the citizenship RCT used to show that null results can produce large learning and that different reasonable priors change the score (0.27 vs 0.34).","marker":"[Hainmueller et al., 2023]"},{"why":"Gives the closed-form Wasserstein result for location-scale families, extending the learning metric beyond normal priors and posteriors.","marker":"[Panaretos and Zemel, 2019]"},{"why":"Companion manuscript proposing within-domain comparisons of learning values across studies.","marker":"[Mikhaeil et al., 2025]"}],"fun_headline_variants":["A single number scores how much a study teaches","Wasserstein metric turns belief change into a study score","What a paper adds, in one distance measure","Rank research by how much it shifts beliefs","When a null result still teaches a lot"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire number depends on the prior distribution being a faithful picture of what the research community actually believed before the study; if that prior cannot be elicited or is chosen arbitrarily, the learning score changes arbitrarily.","fun_headline_variants_meta":{"raw":{"variants":["A single number scores how much a study teaches","Wasserstein metric turns belief change into a study score","What a paper adds, in one distance measure","Rank research by how much it shifts beliefs","When a null result still teaches a lot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1188,"prompt_tokens":677,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":421,"completion_tokens_details":{"reasoning_tokens":440}},"tokens_in":421,"tokens_out":511,"duration_ms":5912,"temperature":1.0,"reasoning_tokens":440,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:15:26.825461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a published study and elicit priors from two independent panels of domain experts using the same elicitation protocol. If the two panels' W2 scores for the same study disagree enough to reverse a judgment about whether the study contributed substantially (as the paper's own citizenship example shows 0.27 versus 0.34 under two reasonable priors), then the metric does not capture an intrinsic property of the study.","supporting_citations":[{"cited_title":"[2006], Mikkola et al","cited_arxiv_id":null,"evidence_quote":"Provides the prior-elicitation methodology the paper relies on for obtaining the community prior that W2 measures from."},{"cited_title":"In these cases, practitioners can separately report the contribution of the change in mean and change in standard deviation to the overall amount of learning, see Table","cited_arxiv_id":null,"evidence_quote":"Gives the closed-form Wasserstein result for location-scale families, extending the learning metric beyond normal priors and posteriors."}],"review_version":1}