pith. sign in

arxiv: 2604.22793 · v1 · submitted 2026-04-13 · 📊 stat.AP

Research Funding as a Decision Problem Under Heavy-Tailed Uncertainty

Carlos Oscar S. Sorzano , B. Pueche-Granados This is my paper

Pith reviewed 2026-05-10 15:10 UTC · model grok-4.3

classification 📊 stat.AP
keywords research fundingheavy-tailed distributionsbiased lotteriespeer review alternativesbibliometric predictionallocation mechanismsdecision theoryexploration-exploitation
0
0 comments X

The pith

Biased lottery mechanisms allocate research funding more transparently and scalably than peer review when impacts follow heavy-tailed distributions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper models research funding as a decision problem where future impacts are highly uncertain and heavy-tailed, so a small fraction of projects accounts for most value. Large-scale bibliometric analysis shows that an aggregated percentile-normalised signal from past performance gives statistically useful, though imperfect, predictions of future productivity across domains. Purely deterministic or stochastic mechanisms that exploit this signal produce extreme concentration, awarding nearly all funds to a tiny set of top performers. To temper this while respecting budgets, the authors define a biased lottery derived from a regularised objective that trades off exploitation of high performers against exploration of others.

Core claim

Past performance supplies meaningful predictive structure for future output, yet both deterministic and stochastic impact-based allocations converge to highly concentrated distributions. A biased lottery framework based on a regularised decision-theoretic objective balances exploration and exploitation under practical constraints and supplies a transparent, efficient, scalable alternative to conventional peer review.

What carries the argument

The biased lottery framework, which regularises a decision-theoretic objective to balance exploration of new researchers against exploitation of high past performers while enforcing funding constraints.

Load-bearing premise

That the aggregated percentile-normalised proxy from past performance captures enough predictive structure to guide allocations across research domains and future periods.

What would settle it

Run a controlled funding round in which one cohort is selected by the biased lottery and another by standard peer review, then compare their subsequent publication, citation, and grant records after five years.

Figures

Figures reproduced from arXiv: 2604.22793 by B. Pueche-Granados, Carlos Oscar S. Sorzano.

Figure 1
Figure 1. Figure 1: Smoothed joint distributions of percentile-normalised future outcomes [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Deterministic allocation curves showing the fraction of the total bud [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Smoothed two-dimensional histograms showing the relationship be [PITH_FULL_IMAGE:figures/full_fig_p025_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Complementary cumulative distribution functions (CCDFs) of future [PITH_FULL_IMAGE:figures/full_fig_p026_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Empirical cumulative distribution functions (CDFs) of productivity [PITH_FULL_IMAGE:figures/full_fig_p027_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Future performance as a function of aggregated past perfor [PITH_FULL_IMAGE:figures/full_fig_p031_4.png] view at source ↗
read the original abstract

Heavy-tailed impact distributions, intrinsic uncertainty, and the high costs of proposal-based peer review increasingly challenge research funding decisions. Using large-scale bibliometric data, we show that past scientific performance provides statistically meaningful, though imperfect, information about future productivity and impact across multiple dimensions. An aggregated, percentile-normalised proxy signal captures this predictive structure robustly across research domains. We analyse deterministic and stochastic funding allocation mechanisms under impact-based objectives and find that both converge to highly concentrated allocations that favour a small number of top-performing researchers. To address the limitations of pure exploitation, we introduce a biased lottery framework based on a regularised decision-theoretic objective that explicitly balances exploration and exploitation while accounting for practical funding constraints. Our results suggest that biased lottery mechanisms offer a transparent, efficient, and scalable alternative to conventional peer review in environments characterised by heavy-tailed scientific returns. Additionally, we provide a web application, available at http://scilottery.biocomputingunit.es, that implements the deterministic allocation method presented in this work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript uses large-scale bibliometric data to argue that past performance yields statistically meaningful (though imperfect) predictive information about future productivity and impact, captured robustly by an aggregated percentile-normalised proxy across domains. Deterministic and stochastic allocation mechanisms under impact-based objectives are shown to converge to highly concentrated allocations favouring top performers. A biased lottery is introduced via a regularised decision-theoretic objective that balances exploration/exploitation under funding constraints, with the claim that such mechanisms provide a transparent, efficient, and scalable alternative to conventional peer review in heavy-tailed environments. A web application implementing the deterministic method is provided.

Significance. If the predictive structure and mechanism analyses hold, the work could inform funding policy by quantifying the concentration effects of pure exploitation and offering a principled, scalable lottery framework that explicitly incorporates heavy-tailed uncertainty. The provision of a publicly available web application for the deterministic allocation method is a concrete strength that supports reproducibility and practical testing.

major comments (2)
  1. [Abstract] Abstract and results description: the central claim that biased lottery mechanisms constitute a transparent, efficient, and scalable alternative to peer review rests on an untested translation from the decision-theoretic model to superior real-world performance; no simulation outputs, sensitivity checks, head-to-head metrics (e.g., total future impact, tail coverage, or regret), or direct comparisons against a peer-review baseline are reported.
  2. [Results] The analysis of deterministic and stochastic mechanisms: while convergence to concentrated allocations is asserted, the manuscript provides no reported error bars, robustness checks under alternative heavy-tailed parameterisations, or validation details on how the percentile-normalised proxy was aggregated and tested across domains, limiting assessment of whether the predictive signal is load-bearing for the allocation conclusions.
minor comments (1)
  1. [Abstract] The web application URL is given but no description of its inputs, outputs, or underlying assumptions is provided in the text, which would aid readers in reproducing the deterministic allocations.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for their detailed and constructive report. We address each major comment point by point below and indicate planned revisions to the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract and results description: the central claim that biased lottery mechanisms constitute a transparent, efficient, and scalable alternative to peer review rests on an untested translation from the decision-theoretic model to superior real-world performance; no simulation outputs, sensitivity checks, head-to-head metrics (e.g., total future impact, tail coverage, or regret), or direct comparisons against a peer-review baseline are reported.

    Authors: We agree that the manuscript does not include direct head-to-head comparisons against a modeled peer-review baseline or associated metrics such as regret or tail coverage. Such comparisons would require explicit assumptions about the peer-review process and proposal-level outcome data that are not present in the bibliometric dataset used here. The claims are grounded in the decision-theoretic analysis of allocation mechanisms given the observed predictive structure and heavy-tailed returns. We will revise the abstract to clarify that the biased lottery is advanced as a transparent, scalable mechanism within this modeling framework. We will also add simulation outputs and sensitivity checks for the allocation mechanisms to the results section. revision: partial

  2. Referee: [Results] The analysis of deterministic and stochastic mechanisms: while convergence to concentrated allocations is asserted, the manuscript provides no reported error bars, robustness checks under alternative heavy-tailed parameterisations, or validation details on how the percentile-normalised proxy was aggregated and tested across domains, limiting assessment of whether the predictive signal is load-bearing for the allocation conclusions.

    Authors: We acknowledge that additional methodological transparency and robustness analyses would improve the presentation. In the revised manuscript we will report error bars on figures showing allocation concentrations, expand the description of how the percentile-normalised proxy was constructed and validated across domains, and include robustness checks under alternative heavy-tailed distributions (e.g., log-normal and Pareto with varied shape parameters) to confirm that the convergence results are not sensitive to the specific distributional assumptions. revision: yes

standing simulated objections not resolved
  • Direct empirical validation of biased-lottery performance against real-world peer-review outcomes, which would require a separate controlled experiment or longitudinal proposal-evaluation dataset outside the scope of the current bibliometric and decision-theoretic study.

Circularity Check

0 steps flagged

No circularity: proxy derived from external data; mechanisms analyzed independently

full rationale

The paper extracts an aggregated percentile-normalised proxy from large-scale bibliometric data to capture predictive structure in past performance for future impact. It then separately examines deterministic and stochastic allocation mechanisms under impact objectives and introduces a biased lottery via a regularised objective. No equations or steps reduce a claimed prediction or result to a fitted parameter by construction, nor do any load-bearing claims rest on self-citations whose content is unverified or tautological. The derivation chain remains self-contained against external bibliometric benchmarks and decision-theoretic modeling without self-referential loops.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review prevents identification of specific free parameters, axioms, or invented entities; no explicit new entities or ad-hoc assumptions are named.

pith-pipeline@v0.9.0 · 5472 in / 1109 out tokens · 62752 ms · 2026-05-10T15:10:51.833608+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

  1. [1]

    Aagaard, K., Kladakis, A., and Nielsen, M. W. (2020). Concentration or dis- persal of research funding?Quantitative Science Studies, 1(1):117–149. Bornmann, L. (2011). Peer review and bibliometric: potentials and problems. InUniversity rankings: Theoretical basis, methodology and impacts on global higher education, pages 145–164. Springer. De Peuter, S. a...

    work page 2020

  2. [2]

    lottery tickets

    11 Fang, F. C., Bowen, A., and Casadevall, A. (2016). Nih peer review percentile scores are poorly predictive of grant productivity.Elife, 5:e13323. Fang, F. C. and Casadevall, A. (2016). Research funding: The case for a modi- fied lottery.MBio, 7(2):10–1128. Gigerenzer, G., Allen, C., Gaillard, S., Goldstone, R. L., Haaf, J., Holmes, W. R., Kashima, Y., ...

    work page 2016