pith. sign in

arxiv: 2408.12932 · v1 · submitted 2024-08-23 · 📊 stat.AP

Broad versus narrow research questions in evidence synthesis: a parallel to (and plea for) estimands

Antonio Remiro-Az\'ocar , Anders Gorst-Rasmussen This is my paper

Pith reviewed 2026-05-23 21:55 UTC · model grok-4.3

classification 📊 stat.AP
keywords estimandsnetwork meta-analysisevidence synthesishealth technology assessmentresearch questionsclinical trialsnarrow questions
0
0 comments X

The pith

The estimand framework from single trials offers a way to make network meta-analysis questions more precise and useful for health technology assessment.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Network meta-analysis is moving from broad to narrow research questions about specific treatments and populations, much like individual clinical trials have done with the estimand framework. This framework defines exactly what is being estimated, including the target population, the treatment of interest, and how to handle events that interrupt treatment. The paper argues that adopting this language in NMA would help align evidence synthesis with the narrow needs of health technology assessment. A reader would care because it promises clearer, more relevant results from combining multiple studies without changing the underlying data.

Core claim

The transition toward narrow research questions in network meta-analysis parallels the introduction of the estimand framework in registrational trials, and the language of estimands can clarify the precise quantity being estimated when synthesizing evidence across studies for health technology assessment purposes.

What carries the argument

The estimand framework, which specifies the target population, treatment, outcome, and handling of intercurrent events to define the quantity of interest.

If this is right

  • NMA analyses would specify exact target populations and treatments to match HTA decision needs.
  • Data collection and analysis plans in evidence synthesis would focus on the defined estimand rather than broad averages.
  • Interpretation of NMA results would include explicit statements on applicability to particular patient groups.
  • Future trials could be designed to address specific estimands missing from existing networks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • NMA guidelines might start requiring explicit estimand descriptions in published reports.
  • This approach could help resolve inconsistencies in networks by making population differences explicit.
  • Decision makers in HTA could more easily judge whether an NMA result applies to their specific setting.

Load-bearing premise

The benefits seen when using estimands in single trials will transfer to network meta-analysis without needing major new methods or creating new biases.

What would settle it

A demonstration that applying the estimand framework to an NMA network requires substantial new bias-correction techniques or produces results that conflict with standard NMA outputs on the same data would falsify the claim.

read the original abstract

There has been a transition from broad to more specific research questions in the practice of network meta-analysis (NMA). Such convergence is also taking place in the context of individual registrational trials, following the recent introduction of the estimand framework, which is impacting the design, data collection strategy, analysis and interpretation of clinical trials. The language of estimands has much to offer to NMA, particularly given the "narrow" perspective of treatments and target populations taken in health technology assessment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript observes a shift from broad to narrow research questions in network meta-analysis (NMA) and draws a parallel to the estimand framework now used in individual registrational trials. It argues that the estimand language can usefully inform NMA, especially for the narrow treatment and population perspectives typical in health technology assessment (HTA).

Significance. If the suggested parallel holds, the manuscript could encourage more precise question formulation in evidence synthesis, improving alignment between NMA outputs and HTA decision needs. The piece correctly notes an observed trend in the literature and gives credit to the ICH E9(R1) estimand framework for sharpening single-trial practice; however, the absence of any worked mapping or example leaves the claimed transferability untested.

major comments (1)
  1. [Abstract] Abstract: the central claim that 'the language of estimands has much to offer to NMA' is presented without any concrete illustration, mapping of estimand attributes (population, treatment, endpoint, intercurrent events, summary measure) onto an NMA setting, or discussion of whether new sources of bias would arise in the multi-study context; this absence is load-bearing for the plea because the manuscript itself acknowledges that NMA is more complex than single registrational trials.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the observed trend toward narrower questions in NMA as well as the potential value of the estimand framework. The major comment correctly identifies that the manuscript is a concise conceptual plea rather than a methods paper; we address it directly below and will revise accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that 'the language of estimands has much to offer to NMA' is presented without any concrete illustration, mapping of estimand attributes (population, treatment, endpoint, intercurrent events, summary measure) onto an NMA setting, or discussion of whether new sources of bias would arise in the multi-study context; this absence is load-bearing for the plea because the manuscript itself acknowledges that NMA is more complex than single registrational trials.

    Authors: We accept that the absence of an explicit mapping weakens the central claim for readers seeking immediate transferability. In revision we will insert a short illustrative paragraph (approximately 150–200 words) that maps the five ICH E9(R1) attributes onto a hypothetical NMA for an HTA-relevant decision (e.g., a three-treatment network in a specific subpopulation). The example will show how the target population, treatment definitions, endpoint, intercurrent-event strategies, and summary measure can be stated more precisely than is typical in current NMA protocols. On new sources of bias, we will add one sentence noting that NMA-specific issues (inconsistency, transitivity violations) constitute additional threats to the validity of the chosen estimand; however, the framework’s primary contribution remains the explicit articulation of the research question rather than the resolution of all multi-study biases. A fuller treatment of bias implications lies outside the scope of this short communication. We therefore view the revision as strengthening rather than altering the manuscript’s intent. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The manuscript is a short conceptual discussion and plea for applying the estimand framework to network meta-analysis, drawing a parallel to its use in single registrational trials. It advances no equations, formal derivations, fitted parameters, or technical results. The argument rests on observed trends in the literature and the external ICH E9(R1) estimand framework rather than any self-referential definitions, self-citation chains, or renamings that reduce claims to their own inputs. No load-bearing steps match the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The argument assumes without demonstration that the estimand framework's benefits in trials generalize to NMA and that narrower questions are inherently preferable for health technology assessment.

axioms (2)
  • domain assumption The estimand framework has produced net benefits in the design and interpretation of individual clinical trials.
    Invoked in the abstract when stating that the framework is 'impacting the design, data collection strategy, analysis and interpretation of clinical trials.'
  • domain assumption Narrower research questions are more appropriate than broad ones for health technology assessment purposes.
    Stated in the abstract as the motivation for applying estimands to NMA.

pith-pipeline@v0.9.0 · 5607 in / 1233 out tokens · 26262 ms · 2026-05-23T21:55:14.952866+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages

  1. [1]

    Twenty years of network meta-analysis: Continuing controversies and recent developments

    Ades A, Welton NJ, Dias S, Phillippo DM, Caldwell DM. Twenty years of network meta-analysis: Continuing controversies and recent developments. Research Synthesis Methods 2024

    work page 2024

  2. [2]

    ICH E9 (R1) addendum on estimands and sensiti v- ity analysis in clinical trials to the guideline on statistical principl es for clinical trials

    European Medicines Agency. ICH E9 (R1) addendum on estimands and sensiti v- ity analysis in clinical trials to the guideline on statistical principl es for clinical trials. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e9-r1-addendum-estimands-sensitivity-analysis-clinical-trials-gu ideline-

    work page

  3. [3]

    Accessed: 2024-03-04

    work page 2024

  4. [4]

    The estimands fr amework: a primer on the ICH E9 (R1) addendum

    Kahan BC, Hindley J, Edwards M, Cro S, Morris TP . The estimands fr amework: a primer on the ICH E9 (R1) addendum. bmj 2024; 384

    work page 2024

  5. [5]

    Seeking har mony: estimands and sensitivity analyses for confirmatory clinical trials

    Mehrotra DV , Hemmings RJ, Russek-Cohen E, Group IEEW. Seeking har mony: estimands and sensitivity analyses for confirmatory clinical trials. Clinical trials 2016; 13(4): 456–458

    work page 2016

  6. [6]

    ICH Guideline Implementation

    International Council for Harmonisation of Technical Requirements fo r Pharmaceuticals for Human Use. ICH Guideline Implementation. https://www.ich.org/page/ich-guideline-implementation; 2024. Accessed: 2024-03-04

    work page 2024

  7. [7]

    Early phase clinical trials extension to guidelines for the content of statistical analysis plans

    Homer V , Yap C, Bond S, et al. Early phase clinical trials extension to guidelines for the content of statistical analysis plans. bmj 2022; 376

    work page 2022

  8. [8]

    Reporting of factorial randomized tri als: extension of the CONSORT 2010 Statement

    Kahan BC, Hall SS, Beller EM, et al. Reporting of factorial randomized tri als: extension of the CONSORT 2010 Statement. JAMA 2023; 330(21): 2106–2114

    work page 2010

  9. [9]

    Consensus Statement for Protocols o f Factorial Randomized Trials: Extension of the SPIRIT 2013 Statement

    Kahan BC, Hall SS, Beller EM, et al. Consensus Statement for Protocols o f Factorial Randomized Trials: Extension of the SPIRIT 2013 Statement. JAMA network open 2023; 6(12): e2346121–e2346121

    work page 2013

  10. [10]

    Why estimands ar e needed to define treatment effects in clinical trials

    Keene ON, Lynggaard H, Englert S, Lanius V , Wright D. Why estimands ar e needed to define treatment effects in clinical trials. BMC medicine 2023; 21(1): 276

    work page 2023

  11. [11]

    Estimands: bringing clarity and focus to research questions in clinical trials

    Clark TP, Kahan BC, Phillips A, White I, Carpenter JR. Estimands: bringing clarity and focus to research questions in clinical trials. BMJ open 2022; 12(1): e052953

    work page 2022

  12. [12]

    A narrat ive review of estimands in drug development and regulatory evaluation: old wine in new barrels?

    Mitroiu M, Oude Rengerink K, Teerenstra S, Pétavy F, Roes K. A narrat ive review of estimands in drug development and regulatory evaluation: old wine in new barrels?. Trials 2020; 21(1): 1–14

    work page 2020

  13. [13]

    Esti mation of treatment effects in short-term depression studies

    Mitroiu M, Teerenstra S, Oude Rengerink K, Pétavy F, Roes KC. Esti mation of treatment effects in short-term depression studies. An evaluation based on the ICH E9 (R1) estimands framework. Pharmaceutical Statistics 2022; 21(5): 1037–1057

    work page 2022

  14. [14]

    Utilization of th e PICO framework to improve searching PubMed for clinical questions

    Schardt C, Adams MB, Owens T, Keitz S, Fontelo P . Utilization of th e PICO framework to improve searching PubMed for clinical questions. BMC medical informatics and decision making 2007; 7(1): 1–6. 6 REMIRO-AZÓCAR & GORST-RASMUSSEN

    work page 2007

  15. [15]

    Preferred reporting items for systematic reviews and meta- analyses: the PRISMA statement

    Moher D, Liberati A, Tetzlaff J, Altman DG, Group P, others . Preferred reporting items for systematic reviews and meta- analyses: the PRISMA statement. International journal of surgery 2010; 8(5): 336–341

    work page 2010

  16. [16]

    The PRISMA statement extension f or systematic reviews incorporating network meta- analysis: PRISMA-NMA

    Hutton B, Catala-Lopez F, Moher D. The PRISMA statement extension f or systematic reviews incorporating network meta- analysis: PRISMA-NMA. Medicina Clínica (English Edition) 2016; 147(6): 262–266

    work page 2016

  17. [17]

    Morga A, Latimer NR, Scott M, Hawkins N, Schlichting M, Wang J. Is Intention to Treat Still the Gold Standard or Should Health Technology Assessment Agencies Embrace a Broader Estimands Framewo rk?: Insights and Perspectives From the National Institute for Health and Care Excellence and Institut für Qualit ät und Wirtschaftlichkeit im Gesundheitswesen on ...

    work page 2023

  18. [18]

    Experimental and quasi-experimental designs for generalized causal inference

    Cook TD, Campbell DT, Shadish W. Experimental and quasi-experimental designs for generalized causal inference. 1195. Houghton Mifflin Boston, MA . 2002

    work page 2002

  19. [19]

    Missing data in clinical trials: from clinical assumptions to statistical analysis using pattern mixture models

    Ratitch B, O’Kelly M, Tosiello R. Missing data in clinical trials: from clinical assumptions to statistical analysis using pattern mixture models. Pharmaceutical statistics 2013; 12(6): 337–347

    work page 2013

  20. [20]

    Target estimands for population-adjusted indirect comparisons

    Remiro-Azócar A. Target estimands for population-adjusted indirect comparisons. Statistics in Medicine 2022; 41(28): 5558–5569

    work page 2022

  21. [21]

    Transportability of model-based estimands in evid ence synthesis

    Remiro-Azócar A. Transportability of model-based estimands in evid ence synthesis. In press, Statistics in Medicine 2024

    work page 2024

  22. [22]

    How choice of effect measure influences minimall y sufficient adjustment sets for external validity

    Webster-Clark M, Keil AP . How choice of effect measure influences minimall y sufficient adjustment sets for external validity. American Journal of Epidemiology 2023; 192(7): 1148–1154

    work page 2023

  23. [23]

    portable

    Xiao M, Chu H, Cole S, et al. Odds Ratios are far from “portable”—A call t o use realistic models for effect variation in meta-analysis.. Journal of clinical epidemiology 2022; 142: 294

    work page 2022

  24. [24]

    On the logic of collapsibility for causal effect measures

    Didelez V , Stensrud MJ. On the logic of collapsibility for causal effect measures. Biometrical Journal 2022; 64(2): 235–242

    work page 2022

  25. [25]

    Some considerations on target estimands for health t echnology assessment

    Remiro-Azócar A. Some considerations on target estimands for health t echnology assessment. Statistics in Medicine 2022; 41(28): 5592–5596

    work page 2022

  26. [26]

    target estimands for population- adjusted indirect comparisons

    Russek-Cohen E. Discussion of “target estimands for population- adjusted indirect comparisons” by Antonio Remiro-Azocar. Statistics in medicine 2022; 41(28): 5573–5576

    work page 2022

  27. [27]

    Julian E, Gianfrate F, Sola-Morales O, et al. How can a joint European H ealth Technology Assessment provide an ‘addi- tional benefit’over the current standard of national assessments? Insight s generated from a multi-stakeholder survey in hematology/oncology. Health economics review 2022; 12(1): 30

    work page 2022