Quantifying How Much Has Been Learned from a Research Study

Donald P. Green; Jonas M. Mikhaeil

REVIEW 2 major objections 4 minor 14 references

Quantifying How Much Has Been Learned from a Research Study

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper recommends the Wasserstein-2 distance between a field's prior and posterior as a single number for how much a study teaches the field.

desk verdict A clean, honest proposal for a W2-based learning metric that is conditional on an unformalized community prior. read the letter →

arxiv 2508.14789 v1 pith:BMISOZOY submitted 2025-08-20 stat.ME stat.AP

Jonas M. Mikhaeil , Donald P. Green This is my paper · ORCID

classification stat.MEstat.AP MSC 62F1562C1062B10

keywords BayesianupdatingWassersteindistanceknowledgeaccumulationoptimaltransportpriorelicitationresearchevaluationprospectivelearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to put a number on how much a research study contributes to a field: the distance between what the research community believed before the study and what it believes after. The recommended measure is the Wasserstein-2 distance between the prior and the posterior distribution, which in the common normal case reduces to a single formula combining the shift in the estimated effect and the change in uncertainty. If the approach works, editors and funders could compare studies by a transparent learning score rather than by statistical significance or informal impressions. The paper also extends the metric forward, letting decision-makers compute how much they should expect to learn from a proposed study before paying for it.

What carries the argument

The load-bearing object is the Wasserstein-2 distance W2(πold, πnew), the minimal cost of transporting the probability mass of the prior into the shape of the posterior. For one-dimensional parameters with normal prior and posterior, W2 = sqrt((µ1−µ0)² + (σ1−σ0)²), making learning a function of both the change in the estimated magnitude and the change in uncertainty. The same closed form holds for any location-scale family. The distance also carries a decision-theoretic justification: by Villani's Theorem 6.9, small W2 implies that expected utilities over a broad class of loss functions will be close, so the metric measures how differently the community would judge decisions under prior vers

What would settle it

Take a published study and elicit priors from two independent panels of domain experts using the same elicitation protocol. If the two panels' W2 scores for the same study disagree enough to reverse a judgment about whether the study contributed substantially (as the paper's own citizenship example shows 0.27 versus 0.34 under two reasonable priors), then the metric does not capture an intrinsic property of the study.

Watch

Extended reading notes

Core claim

The central claim is that learning from a study can be quantified as the Wasserstein-2 distance between the research community's prior distribution over a parameter of interest and the posterior distribution after seeing the study's data. When both distributions are normal this equals the square root of the squared change in the mean plus the squared change in the standard deviation, so moving an estimate and sharpening it both count as learning, and even a null result can produce large learning if it collapses a diffuse prior or overturns a confident one. The paper argues this metric is preferable to Lindley's information measure, which is insensitive to changes in the mean, and it shows th

Load-bearing premise

The entire number depends on the prior distribution being a faithful picture of what the research community actually believed before the study; if that prior cannot be elicited or is chosen arbitrarily, the learning score changes arbitrarily.

Editorial extensions

If this is right

A journal's significance statement could be supplemented with a one-number learning score computed from an elicited prior and the posterior.
Funders could rank prospective studies by expected W2 over the prior predictive distribution, replacing or complementing power calculations that target significance.
Studies with null results can legitimately score as high-learning—the citizenship experiment scores 0.27 or 0.34 under two reasonable priors—because they compress a divided literature.
Within a substantive domain, learning scores can be compared across studies, and dividing by the prior standard deviation gives a scale-free benchmark where 1 means all prior uncertainty was eliminated.
Because W2 is sensitive to increases in uncertainty, revelations of fraud or bias count as learning just as much as new confirmatory evidence does.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the paper does not run: elicit priors from two independent expert panels for the same study; if the W2 scores disagree enough to change a funding decision, the metric is not an objective property of the study.
The non-cumulative nature of W2 means a literature's total learning is not the sum of its studies' scores; portfolio-level evaluation would need an explicit aggregation rule, such as distance from initial prior to final posterior.
The prospective framework implicitly defines 'surprise' as the Wasserstein distance induced by data under one prior while measuring against another prior; this could be linked to the value-of-information literature, though the paper does not make that connection.
Because the metric is scale-dependent, cross-domain comparisons of learning scores are meaningless; any public ranking of studies would need to be domain-specific or normalized.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the take: the paper is a clearly written proposal to use Wasserstein-2 distance between prior and posterior as a 'learning metric' for evaluating individual studies. The math is simple, the illustrative examples do their job, and the limitations section is honest. What you should know before reading: the metric is not an intrinsic property of a study. It depends on the prior, and the paper doesn't solve the prior-elicitation problem.

The genuinely new thing is the framing: using W2 for research evaluation rather than the usual KL/Lindley information measures. The prospective extension with consensus and pioneer priors is a nice addition. The paper earns credit for being explicit about non-cumulativity and unit-dependence.

The load-bearing weakness is exactly what the stress test says. For a fixed posterior N(0,1), choosing prior N(0,σ0) gives W2 = σ0−1 for σ0≥1, which grows without bound as the prior gets more diffuse. The paper says 'the exact specification of the prior matters' (Section 2) and then in Appendix G shows two reasonable priors for the same Hainmueller et al. study producing 0.27 versus 0.34, a 26% difference. That is not a minor detail; it means the headline claim that the metric quantifies how much the community learns is only true conditional on a prior you have to supply from somewhere. The decision-theoretic justification doesn't rescue this either; Theorem 1 just says expectation functionals are continuous in W2, which holds for many other metrics and doesn't uniquely single out W2. Minor: Equation 3 is garbled—the first term should be n σ_prior^4 / (σ^2 + n σ_prior^2)—and the self-citation to Mikhaeil et al. 2025 is not load-bearing.

Who is this for? People in metascience, funding evaluation, Bayesian experimental design. It is a legitimate methodological proposal with a central weakness that is fixable if the authors reframe it as a conditional summary and add a protocol for prior elicitation or comparison across priors. I'd send it to a serious referee; it's not a desk reject.

I would peer review it, and I'd bring it to reading group for the debate about whether such a metric is useful.

Referee Report

2 major / 4 minor

Summary. The paper proposes the Wasserstein-2 distance between a research community's prior distribution and its posterior distribution as a quantitative learning metric, arguing that this distance captures both changes in the location and changes in the dispersion of beliefs after a study. The authors derive the closed-form normal-normal formula, illustrate the metric on stylized examples and on Green et al.'s lawn-sign experiments, and extend the framework to prospective learning by averaging the metric over the prior predictive distribution. They compare the proposal with Lindley's information measure and the KL divergence, and discuss limitations such as scale-dependence and non-cumulativity. The central claim is that W2 offers a simple, interpretable, decision-relevant summary of how much a study contributes to a literature.

Significance. If the proposal is accepted, it would provide editors, funders, and researchers with a single-number supplement to significance testing, and it connects research evaluation to a formal Bayesian decision-theoretic framework. Strengths include the correct and transparent normal-normal formula (Eq. 2), the use of reproducible empirical applications with data and code provided, a clear decision-theoretic motivation in Appendix B, and an unusually candid discussion of the metric's limitations. The prospective-learning extension is a natural and potentially useful contribution. However, the usefulness of the metric hinges on the specification of the community prior, and the manuscript currently lacks a protocol for obtaining that prior; the paper's own Appendix G shows that different reasonable priors give materially different learning scores for the same study. The central prospective formula in Eq. (3) is also garbled as printed. These issues are load-bearing but correctable, so the paper warrants major revision rather than rejection.

major comments (2)

[§3.3, Eq. (3)] The prospective-learning formula is not correctly stated. The displayed expression includes the term σ_prior^2 σ^2 / (n σ_prior^2 + 1), which has incompatible dimensions and does not equal the variance of the posterior mean. The correct term is n σ_prior^4 / (σ^2 + n σ_prior^2); the full expected squared W2 is n σ_prior^4/(σ^2 + n σ_prior^2) + σ_prior^2 (1 − 1/sqrt(1 + n σ_prior^2/σ^2))^2. The notation in Eq. (3) also mixes π_c and π_p before these objects are defined, making the expression unverifiable. Because this is the main analytic result of the prospective section, it must be corrected.
[§2 and Appendix G] The central claim that W2 'quantifies how much a research community learns' is not fully supported without an operational definition of the community prior π_old. Section 2 correctly warns that the prior matters, and the conclusion suggests possible sources, but no elicitation, aggregation, or sensitivity protocol is provided. Appendix G illustrates the consequence: two plausible priors for the Hainmueller et al. study give W2 = 0.27 and W2 = 0.34 for the same study, a 26% difference, and the normal-normal case is even more extreme: for a fixed posterior N(0,1) and priors N(0,σ0), W2 = σ0 − 1, which is unbounded as σ0 grows. The paper's recommendation to normalize by σ0 is mentioned but not made central; without such a normalization, the metric conflates prior ignorance with study-induced learning. The authors need either to provide a concrete procedure for constructing the community pri

minor comments (4)

[Appendix D, Eq. (6)] Lindley's information measure is defined as H(π(θ|y)) − H(π(θ)), which is negative for an informative experiment (e.g., log 5 − log 10 = −0.69), yet Table 2 reports positive values. The authors should either define I as H(π(θ)) − H(π(θ|y)) or use absolute values, and note the convention.
[Appendix E] The text states that the KL divergence is 'upper bounded by the surprisal of the study’s outcome S(y) = 1/p(y)'. Surprisal is normally −log p(y), and the stated bound is not generally valid for KL(π(θ|y), π(θ)). This is not central to the W2 recommendation but should be corrected.
[§2.1 and Appendix A] The notation for the prior is inconsistent: the main text uses π(θ), πold := π(ϑ), and π0 interchangeably. Using θ consistently throughout would improve readability.
[§3.3] Figure 5 is described as showing E_{p_{dc}(y)}[W2], but the analytic bound in Eq. (3) is for E[W2^2]. The figure caption should state whether this quantity is computed exactly, by simulation, or by an approximation such as sqrt(E[W2^2]).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the W2 learning metric is a proposed operationalization and all calculations follow from stated inputs.

full rationale

The paper's derivation chain is self-contained: W2(π_old, π_new) is introduced as a proposed measure of belief change, not as a quantity derived from a hidden premise (Section 2.1, Eqs. 1–2). The closed-form normal expression is a standard optimal-transport result (Eq. 2). The prospective analysis (Section 3.3, Eq. 3) computes the prior-predictive expectation of this explicitly defined metric; it is an analytic calculation under the stated prior, not an empirical prediction from fitted parameters. Empirical examples use clearly labeled illustrative priors, e.g., 'Suppose, for example, that the prior distribution were π0 = N(0,5)' (Section 3.2), and Appendix G candidly shows that different reasonable priors produce different W2 values, which is a measurement-dependence caveat rather than a circular step. The sole self-citation, [Mikhaeil et al., 2025], supports only a secondary interpretive suggestion about within-domain comparisons and is not load-bearing for the central metric, which is justified independently via Villani's theorem and the paper's own examples. Acknowledged limitations such as non-cumulativity, unit-dependence, and prior sensitivity are validity concerns, not instances of the derivation reducing to its own inputs.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim introduces no fitted parameters or new entities. It rests on assumptions that community beliefs are representable and elicitable as a one-dimensional distribution. The illustrative applications choose priors by hand, making the specific learning values dependent on those choices.

assumptions (4)

domain assumption Community beliefs about a quantity of interest are representable as a probability distribution over a one-dimensional parameter θ.
Section 2: 'Suppose our beliefs... provide us with a prior π(θ) about a parameter θ.' The entire metric operates on such distributions.
domain assumption A consensus prior can be elicited from the literature or experts and represents the shared beliefs of the field.
Section 2 and 3.3 rely on prior elicitation methods (O'Hagan et al. 2006; Mikkola et al. 2023); the metric's value depends entirely on this prior.
standard math Normality (or same location-scale family) of prior and posterior for the closed-form W2 expression.
Equation 2 and Appendix A use the normal-normal W2 formula; the paper acknowledges general distributions require numerical computation.
domain assumption For prospective learning, the data-generating model y ∼ N(θ, σ) with known σ is correctly specified.
Section 3.3 computes expected W2 under this model; misspecification would change the predictive distribution and expected learning.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quantifying How Much Has Been Learned from a Research Study." pith.science (2026). https://pith.science/paper/BMISOZOY

@misc{pith2026250814789,
  author       = {Pith},
  title        = {Pith review of: Quantifying How Much Has Been Learned from a Research Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMISOZOY}},
  note         = {Machine review of arXiv:2508.14789}
}

read the original abstract

How much does a research study contribute to a scientific literature? We propose a learning metric to quantify how much a research community learns from a given study. To do so, we adopt a Bayesian perspective and assess changes in the community's beliefs once updated with a new study's evidence. We recommend the Wasserstein-2 distance as a way to describe how the research community's prior beliefs change to incorporate a study's findings. We illustrate this approach through stylized examples and empirical applications, showing how it differs from more traditional evaluative standards, such as statistical significance. We then extend the framework to the prospective setting, offering a way for decision-makers to evaluate the expected amount of learning from a proposed study. While assessments about what has or could be learned from a research program are often expressed informally, our learning metric provides a principled tool for judging scientific contributions. By formalizing these judgments, our measure has the potential to allow for more transparent assessments of past and prospective research contributions.

Figures

Figures reproduced from arXiv: 2508.14789 by the authors.

**Figure 2.** Comparison of four posterior distributions to a prior. How much is learned [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Comparison of the posteriors for the series of four field experiments presented [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Illustration of prospective learning: Based on weighing the expert’s judgment, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Degree of prospective learning for different study sizes [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Comparison of posterior updating and learning in terms of our Wasserstein [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

[1]

[2006], Mikkola et al

In situations in which prior elicitation O’Hagan et al. [2006], Mikkola et al

work page 2006
[2]

In these cases, practitioners can separately report the contribution of the change in mean and change in standard deviation to the overall amount of learning, see Table

The same closed-form solution obtains more general when prior and posterior are members of the same location-scale family [Panaretos and Zemel, 2019]. In these cases, practitioners can separately report the contribution of the change in mean and change in standard deviation to the overall amount of learning, see Table

work page 2019
[10]

compliers

and ascribes significantly more learning in this case. Lindley’s learning metric is unable to capture learning because it is completely insen- sitive to changes in the mean – it, for example, ascribes the same amount of learning for scenario 1 and scenario 3 even though the mean has further from prior beliefs in the first example. G Empirical Example: Mea...

work page 2023
[11]

cash-value

0 81 81 9.0 Table 1: Contribution of the change in mean and the change in standard deviation to the Wasserstein-2 learning metric for the illustrative examples from section 3.1. linear programming. Off-the-shelf packages (such as the transport package in R [Schuh- macher et al., 2024]) make calculations of the Wasserstein distance straightforward. Figure ...

work page 2024
[12]

16 Generally, learning will not be captured by changes in expected utility for a specific utility function

We thus see that if the prior π(θ) is close to the posterior π(θ|y) in the Wasserstein-2 distance, we can guarantee that our evaluations of potential decisionsa will be very similar. 16 Generally, learning will not be captured by changes in expected utility for a specific utility function. Our beliefs need to be useful and dependable across many situation...

work page 1956
[13]

See also the discussion in the previous section (Section C)

as they are sensitive to comprehensive changes in the posterior distribution. See also the discussion in the previous section (Section C). E Kullback-Leibler Divergence The Kullback-Leibler divergence KL(π(θ|y), π(θ)) = Z π(θ|y) log π(θ|y) π(θ) dθ. is frequently used to compare how similar two distributions are [Nielsen, 2020]. How much is learned in term...

work page 2020
[2002]

ISBN 978-1-58113-495-7

Association for Computing Machinery. ISBN 978-1-58113-495-7. doi: 10.1145/509907.510005. URL https://doi. org/10.1145/509907.510005. P . Bevelander and R. Pendakur. Citizenship, co-ethnic populations and employment probabilities of immigrants in sweden. Journal of International Migration and Integration, 13: 203–222,

work page arXiv
[2006]

doi: DOI:10.1002/0470033312. V . M. Panaretos and Y. Zemel. Statistical aspects of wasserstein distances. An- nual Review of Statistics and Its Application , 6(Volume 6, 2019):405–431,

work page doi:10.1002/0470033312 2019

Show all 14 references

[2008]

The left facet shows how a normal prior is updated; the right facet, how a truncated normal is updated

13 A The Wasserstein Distance for Non-Normal Distributions Prior: ⇡0 W2(⇡0,⇡1)=1 .5 0.00 0.05 0.10 0.15 0.20 0.25 −10 0 10 Density 0.0 0.3 0.6 0.9 0 5 10 15 20 Update 1: ⇡1 Update 1: ⇡1 W2(⇡0,⇡1)=4 .0 Prior: ⇡0 Figure 6: Comparison of posterior updating and learning in terms o...

2016
[2016]

doi: https://doi.org/10.1016/j

ISSN 0261-3794. doi: https://doi.org/10.1016/j. electstud.2015.12.002. URL https://www.sciencedirect.com/science/article/pii/ S0261379415002310. J. Hainmueller, D. Hangartner, and D. Ward. The effect of citizenship on the long-term earnings of marginalized immigrants: Quasi-ex...

2015 doi
[2019]

doi: https://doi.org/10.1146/annurev-statistics-030718-104938

ISSN 2326-831X. doi: https://doi.org/10.1146/annurev-statistics-030718-104938. URL https://www.annualreviews.org/content/journals/10.1146/ annurev-statistics-030718-104938 . G. Parmigiani and D. A. Berry. Applications of lindley information measure to the design of clinical ex...

doi
[2020]

doi: 10.3390/e22101100

ISSN 1099-4300. doi: 10.3390/e22101100. arXiv:1808.08271 [cs]. A. O’Hagan, C. E. Buck, A. Daneshkhah, J. R. Eiser, D. J. J. Paul H. Garthwaite, J. E. Oakley, and T. Rakow. Uncertain Judgements: Eliciting Experts’ Probabilities . John Wiley & Sons, Ltd,

arXiv
[2023]

URL https://arxiv.org/abs/2112.01380. F. Nielsen. An elementary introduction to information geometry. Entropy, 22(10):1100, Sept

arXiv
[2024]

W. James. Pragmatism: A New Name for Some Old Ways of Thinking. New York and London: Longmans, Green & Co.; reprinted Cambridge, MA: Harvard University Press, 1975,

1975

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

[2006], Mikkola et al

In situations in which prior elicitation O’Hagan et al. [2006], Mikkola et al

work page 2006

[2] [2]

In these cases, practitioners can separately report the contribution of the change in mean and change in standard deviation to the overall amount of learning, see Table

The same closed-form solution obtains more general when prior and posterior are members of the same location-scale family [Panaretos and Zemel, 2019]. In these cases, practitioners can separately report the contribution of the change in mean and change in standard deviation to the overall amount of learning, see Table

work page 2019

[3] [10]

compliers

and ascribes significantly more learning in this case. Lindley’s learning metric is unable to capture learning because it is completely insen- sitive to changes in the mean – it, for example, ascribes the same amount of learning for scenario 1 and scenario 3 even though the mean has further from prior beliefs in the first example. G Empirical Example: Mea...

work page 2023

[4] [11]

cash-value

0 81 81 9.0 Table 1: Contribution of the change in mean and the change in standard deviation to the Wasserstein-2 learning metric for the illustrative examples from section 3.1. linear programming. Off-the-shelf packages (such as the transport package in R [Schuh- macher et al., 2024]) make calculations of the Wasserstein distance straightforward. Figure ...

work page 2024

[5] [12]

16 Generally, learning will not be captured by changes in expected utility for a specific utility function

We thus see that if the prior π(θ) is close to the posterior π(θ|y) in the Wasserstein-2 distance, we can guarantee that our evaluations of potential decisionsa will be very similar. 16 Generally, learning will not be captured by changes in expected utility for a specific utility function. Our beliefs need to be useful and dependable across many situation...

work page 1956

[6] [13]

See also the discussion in the previous section (Section C)

as they are sensitive to comprehensive changes in the posterior distribution. See also the discussion in the previous section (Section C). E Kullback-Leibler Divergence The Kullback-Leibler divergence KL(π(θ|y), π(θ)) = Z π(θ|y) log π(θ|y) π(θ) dθ. is frequently used to compare how similar two distributions are [Nielsen, 2020]. How much is learned in term...

work page 2020

[7] [2002]

ISBN 978-1-58113-495-7

Association for Computing Machinery. ISBN 978-1-58113-495-7. doi: 10.1145/509907.510005. URL https://doi. org/10.1145/509907.510005. P . Bevelander and R. Pendakur. Citizenship, co-ethnic populations and employment probabilities of immigrants in sweden. Journal of International Migration and Integration, 13: 203–222,

work page arXiv

[8] [2006]

doi: DOI:10.1002/0470033312. V . M. Panaretos and Y. Zemel. Statistical aspects of wasserstein distances. An- nual Review of Statistics and Its Application , 6(Volume 6, 2019):405–431,

work page doi:10.1002/0470033312 2019