Transportability of model-based estimands in evidence synthesis
Pith reviewed 2026-05-24 10:21 UTC · model grok-4.3
The pith
For non-collapsible measures, purely prognostic variables modify marginal effects through their joint distribution with effect modifiers, complicating transport between studies.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In evidence synthesis, marginal effect estimates for non-collapsible measures cannot be expressed in terms of marginal covariate moments when there is heterogeneity, and generally depend on the joint distribution of conditional effect measure modifiers and purely prognostic variables. Unadjusted anchored indirect comparisons can be biased in the absence of individual-level treatment effect homogeneity, or when marginal covariate moments are balanced across studies. Covariate adjustment may be necessary to account for cross-study imbalances in joint covariate distributions involving purely prognostic variables. In the absence of individual patient data for the target, covariate adjustment is,
What carries the argument
The distinction between directly collapsible and non-collapsible effect measures together with the dependence of the latter on joint covariate distributions rather than marginal moments alone.
If this is right
- Unadjusted anchored indirect comparisons can be biased without individual-level treatment effect heterogeneity.
- Unadjusted anchored indirect comparisons can be biased when marginal covariate moments are balanced across studies.
- Covariate adjustment may be necessary to account for cross-study imbalances in joint covariate distributions involving purely prognostic variables.
- In the absence of individual patient data for the target, covariate adjustment approaches are inherently limited for measures that are not directly collapsible.
- Directly collapsible measures reduce dependence on model-based covariate adjustment when there is individual-level homogeneity or marginal covariate moments are balanced.
Where Pith is reading between the lines
- Synthesis protocols could incorporate explicit checks of joint covariate distributions rather than relying only on marginal balance tests.
- When transport across populations is central, analysts may prefer collapsible measures such as risk differences over non-collapsible ones to limit adjustment requirements.
- The argument implies that current recommendations for covariate selection in network meta-analysis may need revision to prioritize variables that affect joint distributions.
Load-bearing premise
Marginal effect estimates rather than conditional ones are the target quantities for population-level decisions, and purely prognostic variables modify those marginal effects through their joint distribution with effect modifiers.
What would settle it
A simulation in which individual-level treatment response is homogeneous, marginal covariate moments match across studies, yet changing only the joint distribution of a prognostic variable and an effect modifier produces a different marginal odds ratio or risk ratio.
read the original abstract
In evidence synthesis, effect modifiers are typically described as variables that induce treatment effect heterogeneity at the individual level, through treatment-covariate interactions in an outcome model parametrized at such level. As such, effect modification is defined with respect to a conditional measure, but marginal effect estimates are required for population-level decisions in health technology assessment. For non-collapsible measures, purely prognostic variables that are not determinants of treatment response at the individual level may modify marginal effects, even where there is individual-level treatment effect homogeneity. With heterogeneity, marginal effects for measures that are not directly collapsible cannot be expressed in terms of marginal covariate moments, and generally depend on the joint distribution of conditional effect measure modifiers and purely prognostic variables. There are implications for recommended practices in evidence synthesis. Unadjusted anchored indirect comparisons can be biased in the absence of individual-level treatment effect heterogeneity, or when marginal covariate moments are balanced across studies. Covariate adjustment may be necessary to account for cross-study imbalances in joint covariate distributions involving purely prognostic variables. In the absence of individual patient data for the target, covariate adjustment approaches are inherently limited in their ability to remove bias for measures that are not directly collapsible. Directly collapsible measures would facilitate the transportability of marginal effects between studies by: (1) reducing dependence on model-based covariate adjustment where there is individual-level treatment effect homogeneity or marginal covariate moments are balanced; and (2) facilitating the selection of baseline covariates for adjustment where there is individual-level treatment effect heterogeneity.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript argues that for non-collapsible effect measures, marginal estimands depend on the full joint distribution of covariates (including purely prognostic variables) even under individual-level treatment effect homogeneity, whereas directly collapsible measures allow marginal effects to be expressed via marginal covariate moments. This has direct implications for bias in unadjusted anchored indirect comparisons, the reach of covariate adjustment without IPD, and the selection of baseline covariates when heterogeneity is present.
Significance. If the definitional argument is accepted, the paper supplies a useful conceptual distinction that can guide choice of effect measure and adjustment strategy in evidence synthesis for health technology assessment, where marginal rather than conditional effects are the policy-relevant quantities.
minor comments (2)
- The abstract is dense; the main text would benefit from a short numerical illustration (e.g., a 2×2 covariate table) showing how a purely prognostic variable alters a non-collapsible marginal effect under conditional homogeneity.
- A brief simulation or analytic example demonstrating the claimed bias in unadjusted indirect comparison would make the practical implications more concrete without altering the central claim.
Simulated Author's Rebuttal
We thank the referee for their supportive assessment of the manuscript and for recommending minor revision. No specific major comments were provided in the report, so we have no individual points to address point-by-point. We will incorporate any minor editorial suggestions during revision.
Circularity Check
No significant circularity identified
full rationale
The paper derives its claims about transportability of marginal effects directly from the standard mathematical definitions of collapsibility (marginal effect equals covariate-averaged conditional effect under homogeneity) and effect modification, without any reduction to fitted parameters, self-referential equations, or load-bearing self-citations. The abstract explicitly states that for non-collapsible measures purely prognostic variables can modify marginal effects even under individual-level homogeneity, and that marginal effects depend on joint distributions when heterogeneity is present; these are direct consequences of the definitions rather than constructed predictions or ansatzes. No equations or steps in the provided text reduce the central result to its own inputs by construction, and the implications for indirect comparisons and covariate adjustment follow logically from the premises without circularity.
Axiom & Free-Parameter Ledger
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
For non-collapsible measures, purely prognostic variables... may modify marginal effects, even where there is individual-level treatment effect homogeneity.
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
Directly collapsible measures would facilitate the transportability of marginal effects between studies
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Forward citations
Cited by 1 Pith paper
-
Model-based standardization using multiple imputation
MIM generates synthetic datasets via multiple imputation to average predictions from a parametric outcome model over the target covariate distribution, providing marginal effect estimates with proper uncertainty quant...
Reference graph
Works this paper leans on
-
[1]
BucherHC,GuyattGH,GriffithLE,WalterSD.Theresultsofdirectandindirecttreatmentcomparisonsinmeta-analysis of randomized controlled trials.Journal of clinical epidemiology1997; 50(6): 683–691
-
[2]
Use of indirect and mixed treatment comparisons for technology assessment
Sutton A, Ades A, Cooper N, Abrams K. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics 2008; 26(9): 753–767
work page 2008
-
[3]
DiasS,SuttonAJ,AdesA,WeltonNJ.Evidencesynthesisfordecisionmaking2:ageneralizedlinearmodelingframework for pairwise and network meta-analysis of randomized controlled trials.Medical Decision Making2013; 33(5): 607–617
-
[4]
Lumley T. Network meta-analysis for indirect treatment comparisons.Statistics in medicine2002; 21(16): 2313–2324
-
[5]
Lu G, Ades A. Combination of direct and indirect evidence in mixed treatment comparisons.Statistics in medicine2004; 23(20): 3105–3124
-
[6]
JansenJP,FleurenceR,DevineB,etal.Interpretingindirecttreatmentcomparisonsandnetworkmeta-analysisforhealth- caredecisionmaking:reportoftheISPORTaskForceonIndirectTreatmentComparisonsGoodResearchPractices:part
-
[7]
Value in health2011; 14(4): 417–428
-
[8]
Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirect treatment comparison/network meta-analysis study questionnaire to assess relevance andcredibility to informhealth care decision making:an ISPOR-AMCP-NPC Good PracticeTask Force report. Value in Health2014; 17(2): 157–173
-
[9]
Meta-analysis in clinical trials.Controlled clinical trials1986; 7(3): 177–188
DerSimonian R, Laird N. Meta-analysis in clinical trials.Controlled clinical trials1986; 7(3): 177–188
-
[10]
DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update.Contemporary clinical trials 2007; 28(2): 105–114
work page 2007
-
[11]
Stijnen T, Hamza TH, Özdemir P. Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data.Statistics in medicine2010; 29(29): 3046–3067
-
[12]
Bowden J, Tierney JF, Simmonds M, Copas AJ, Higgins JP. Individual patient data meta-analysis of time-to-event out- comes: one-stage versus two-stage approaches for estimating the hazard ratio under a random effects model.Research synthesis methods2011; 2(3): 150–162
-
[13]
MedicalDecisionMaking 2005; 25(6): 646–654
AdesA,LuG,HigginsJ.Theinterpretationofrandom-effectsmeta-analysisindecisionmodels. MedicalDecisionMaking 2005; 25(6): 646–654
work page 2005
-
[14]
Dahabreh IJ, Petito LC, Robertson SE, Hernán MA, Steingrimsson JA. Towards causally interpretable meta-analysis: transporting inferences from multiple randomized trials to a new target population.Epidemiology (Cambridge, Mass.) 2020; 31(3): 334. REMIRO-AZÓCAR 29
work page 2020
-
[15]
Sobel M, Madigan D, Wang W. Causal inference for meta-analysis and multi-level data structures, with application to randomized studies of Vioxx.Psychometrika2017; 82(2): 459–474
-
[16]
Barker DH, Dahabreh IJ, Steingrimsson JA, et al. Causally interpretable meta-analysis: Application in adolescent HIV prevention.Prevention Science2022; 23(3): 403–414
-
[17]
Schardt C, Adams MB, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions.BMC medical informatics and decision making2007; 7(1): 1–6
-
[18]
JulianE,GianfrateF,Sola-MoralesO,etal.HowcanajointEuropeanhealthtechnologyassessmentprovidean‘additional benefit’over the current standard of national assessments?.Health Economics Review2022; 12(1): 1–12
-
[19]
Comparative effectiveness without head-to-head trials.Pharmacoeconomics 2010; 28(10): 935–945
Signorovitch JE, Wu EQ, Andrew PY, et al. Comparative effectiveness without head-to-head trials.Pharmacoeconomics 2010; 28(10): 935–945
work page 2010
-
[20]
PhillippoDM,AdesAE,DiasS,PalmerS,AbramsKR,WeltonNJ.Methodsforpopulation-adjustedindirectcomparisons in health technology appraisal.Medical Decision Making2018; 38(2): 200–211
-
[21]
Phillippo D, Ades T, Dias S, Palmer S, Abrams KR, Welton N. NICE DSU technical support document 18: methods for population-adjusted indirect comparisons in submissions to NICE. 2016
work page 2016
-
[22]
Remiro-AzócarA,HeathA,BaioG.ParametricG-computationforcompatibleindirecttreatmentcomparisonswithlimited individual patient data.Research synthesis methods2022; 13(6): 716–744
-
[23]
Two-stage matching-adjusted indirect comparison.BMC medical research methodology2022; 22(1): 1–16
Remiro-Azócar A. Two-stage matching-adjusted indirect comparison.BMC medical research methodology2022; 22(1): 1–16
-
[24]
Multilevel network meta-regression for population-adjusted treatment comparisons
Phillippo DM, Dias S, Ades A, et al. Multilevel network meta-regression for population-adjusted treatment comparisons. Journal of the Royal Statistical Society: Series A (Statistics in Society)2020; 183(3): 1189–1210
work page 2020
-
[25]
BMC medical research methodology2012; 12(1): 1–16
JansenJP,CopeS.Meta-regressionmodelstoaddressheterogeneityandinconsistencyinnetworkmeta-analysisofsurvival outcomes. BMC medical research methodology2012; 12(1): 1–16
-
[26]
Statistics in medicine2009; 28(14): 1861–1881
CooperNJ,SuttonAJ,MorrisD,AdesA,WeltonNJ.Addressingbetween-studyheterogeneityandinconsistencyinmixed treatment comparisons: application to stroke prevention treatments in individuals with non-rheumatic atrial fibrillation. Statistics in medicine2009; 28(14): 1861–1881
-
[27]
Dias S, Sutton AJ, Welton NJ, Ades A. Evidence synthesis for decision making 3: heterogeneity—subgroups, meta- regression, bias, and bias-adjustment.Medical Decision Making2013; 33(5): 618–640
-
[28]
Dias S, Sutton AJ, Welton NJ, Ades A. NICE DSU technical support document 3: heterogeneity: subgroups, meta- regression, bias and bias-adjustment. 2011
work page 2011
-
[29]
Directed acyclic graphs can help understand bias in indirect and mixed treatment comparisons
Jansen JP, Schmid CH, Salanti G. Directed acyclic graphs can help understand bias in indirect and mixed treatment comparisons. Journal of clinical epidemiology2012; 65(7): 798–807
-
[30]
Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2.Value in health 2011; 14(4): 429–437
work page 2011
-
[31]
Remiro-Azócar A, Heath A, Baio G. Conflating marginal and conditional treatment effects: Comments on “Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study”.Statistics in Medicine 2021; 40(11): 2753–2758
work page 2021
-
[32]
Phillippo DM, Dias S, Ades AE, Welton NJ. Target estimands for efficient decision making: Response to comments on “Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study”. Statistics in Medicine2021; 40(11): 2759–2763. 30 REMIRO-AZÓCAR
-
[33]
Remiro-Azócar A. Target estimands for population-adjusted indirect comparisons.Statistics in medicine2022; 41(28): 5558–5569
-
[34]
Greenland S. Interpretation and choice of effect measures in epidemiologic analyses.American journal of epidemiology 1987; 125(5): 761–768
work page 1987
-
[35]
Does Cox analysis of a randomized survival study yield a causal treatment effect?
Aalen OO, Cook RJ, Røysland K. Does Cox analysis of a randomized survival study yield a causal treatment effect?. Lifetime data analysis2015; 21(4): 579–593
-
[36]
Greenland S, Pearl J. Adjustments and their consequences—collapsibility analysis using graphical models.International Statistical Review2011; 79(3): 401–426
-
[37]
Marginalia: comparing adjusted effect measures.Epidemiology2010; 21(4): 490–493
Kaufman JS. Marginalia: comparing adjusted effect measures.Epidemiology2010; 21(4): 490–493
-
[38]
Internationaljournalofepidemiology 2011;40(3): 780–785
HernánMA,ClaytonD,KeidingN.TheSimpson’sparadoxunraveled. Internationaljournalofepidemiology 2011;40(3): 780–785
work page 2011
-
[39]
On the collapsibility of measures of effect in the counterfactual causal framework
Huitfeldt A, Stensrud MJ, Suzuki E. On the collapsibility of measures of effect in the counterfactual causal framework. Emerging themes in epidemiology2019; 16: 1–5
-
[40]
Risk ratio, odds ratio, risk difference
Colnet B, Josse J, Varoquaux G, Scornet E. Risk ratio, odds ratio, risk difference... Which causal measure is easier to generalize?.arXiv preprint arXiv:2303.160082023
-
[41]
target estimands for population-adjusted indirect comparisons
Russek-Cohen E. Discussion of “target estimands for population-adjusted indirect comparisons” by Antonio Remiro- Azocar.Statistics in medicine2022; 41(28): 5573–5576
-
[42]
Target estimands for population-adjusted indirect comparisons
Schiel A. Commentary on" Target estimands for population-adjusted indirect comparisons"..Statistics in medicine2022; 41(28): 5570–5572
-
[43]
Spieker AJ. Comments on the debate between marginal and conditional estimands.Statistics in medicine2022; 41(28): 5589–5591
-
[44]
Senn S. Conditions for success and margins of error: estimation in clinical trials.Statistics in medicine2022; 41(28): 5586–5588
-
[45]
Van Lancker K, Vo TT, Akacha M. Estimands in heath technology assessment: a causal inference perspective.Statistics in medicine2022; 41(28): 5577–5585
-
[46]
StatisticsinMedicine 2022; 41(28): 5592–5596
Remiro-AzócarA.Someconsiderationsontargetestimandsforhealthtechnologyassessment. StatisticsinMedicine 2022; 41(28): 5592–5596
work page 2022
-
[47]
Mehrotra DV, Hemmings RJ, Russek-Cohen E, Group IEEW. Seeking harmony: estimands and sensitivity analyses for confirmatory clinical trials.Clinical trials2016; 13(4): 456–458
-
[48]
Akacha M, Bretz F, Ohlssen D, Rosenkranz G, Schmidli H. Estimands and their role in clinical trials.Statistics in Biopharmaceutical Research2017; 9(3): 268–271
-
[49]
Estimands in clinical trials–broadening the perspective.Statistics in medicine2017; 36(1): 5–19
Akacha M, Bretz F, Ruberg S. Estimands in clinical trials–broadening the perspective.Statistics in medicine2017; 36(1): 5–19
-
[50]
Rubin DB. Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of educational Psychology1974; 66(5): 688
-
[51]
The Use of Covariate Adjustment in Randomized Controlled Trials: An Overview
Van Lancker K, Bretz F, Dukes O. The Use of Covariate Adjustment in Randomized Controlled Trials: An Overview. arXiv preprint arXiv:2306.058232023
-
[52]
rank reversal in indirect comparisons
Valkenhoef vG, Ades A. Evidence synthesis assumes additivity on the scale of measurement: response to “rank reversal in indirect comparisons” by Norton et al..Value in Health2013; 16(2): 449–451. REMIRO-AZÓCAR 31
-
[53]
Caldwell DM, Welton NJ, Dias S, Ades A. Selecting the best scale for measuring treatment effect in a network meta- analysis: a case study in childhood nocturnal enuresis.Research Synthesis Methods2012; 3(2): 126–141
-
[54]
Networkmeta-analysisfordecision-making
DiasS,AdesAE,WeltonNJ,JansenJP,SuttonAJ. Networkmeta-analysisfordecision-making . JohnWiley&Sons. 2018
work page 2018
-
[55]
Vansteelandt S, Dukes O. Assumption-lean inference for generalised linear model parameters.Journal of the Royal Statistical Society Series B: Statistical Methodology2022; 84(3): 657–685
-
[56]
Remiro-Azócar A, Heath A, Baio G. Methods for population adjustment with limited access to individual patient data: A review and simulation study.Research synthesis methods2021; 12(6): 750–775
-
[57]
Liu Y, Wang B, Yang M, et al. Correct and logical causal inference for binary and time-to-event outcomes in randomized controlled trials.Biometrical Journal2022; 64(2): 198–224
-
[58]
DanielR,ZhangJ,FarewellD.Makingapplesfromoranges:Comparingnoncollapsibleeffectestimatorsandtheirstandard errors after adjustment for different covariate sets.Biometrical Journal2021; 63(3): 528–557
-
[59]
Vellaisamy P, Vijay V. Collapsibility of regression coefficients and its extensions.Journal of statistical planning and inference2008; 138(4): 982–994
-
[60]
Randomization does not justify logistic regression.Statistical Science2008; 23(2): 237–249
Freedman DA. Randomization does not justify logistic regression.Statistical Science2008; 23(2): 237–249
-
[61]
Confounding and collapsibility in causal inference.Statistical science1999: 29–46
Greenland S, Robins JM, Pearl J. Confounding and collapsibility in causal inference.Statistical science1999: 29–46
-
[62]
MorrisTP, WalkerAS,WilliamsonEJ, WhiteIR.Planninga methodforcovariate adjustmentinindividuallyrandomised trials: a practical guide.Trials2022; 23(1): 328
-
[63]
A geometric approach to assess bias due to omitted covariates in generalized linear models
Neuhaus JM, Jewell NP. A geometric approach to assess bias due to omitted covariates in generalized linear models. Biometrika 1993; 80(4): 807–815
work page 1993
-
[64]
Austin PC, Stafford J. The performance of two data-generation processes for data with specified marginal treatment odds ratios.Communications in Statistics—Simulation and Computation2008; 37(6): 1039–1051
-
[65]
Statistical methods in medical research2016; 25(5): 1925–1937
PangM,KaufmanJS,PlattRW.Studyingnoncollapsibilityoftheoddsratiowithmarginalstructuralandlogisticregression models. Statistical methods in medical research2016; 25(5): 1925–1937
work page 1925
-
[66]
Lifetimedata analysis2013; 19(3): 279–296
MartinussenT,VansteelandtS.OncollapsibilityandconfoundingbiasinCoxandAalenregressionmodels. Lifetimedata analysis2013; 19(3): 279–296
-
[67]
On the distinction between interaction and effect modification.Epidemiology 2009; 20(6): 863–871
VanderWeele TJ. On the distinction between interaction and effect modification.Epidemiology 2009; 20(6): 863–871
work page 2009
-
[68]
Four types of effect modification: a classification based on directed acyclic graphs
VanderWeele TJ, Robins JM. Four types of effect modification: a classification based on directed acyclic graphs. Epidemiology 2007; 18(5): 561–568
work page 2007
-
[69]
Boca Raton: Chapman & Hall/CRC
Hernán MA, Robins JM.Causal inference: what if. Boca Raton: Chapman & Hall/CRC . 2020
work page 2020
-
[70]
VanderWeele TJ, Knol MJ. Interpretation of subgroup analyses in randomized trials: heterogeneity versus secondary interventions.Annals of internal medicine2011; 154(10): 680–683
-
[71]
Statisticsinmedicine 1999;18(12):1467–1474
LongfordNT.Selectionbiasandtreatmentheterogeneityinclinicaltrials. Statisticsinmedicine 1999;18(12):1467–1474
work page 1999
-
[72]
Brumback B, Berg A. On effect-measure modification: Relationships among changes in the relative risk, odds ratio, and risk difference.Statistics in Medicine2008; 27(18): 3453–3465
-
[73]
KieferC,MayerA.Averageeffectsbasedonregressionswithalogarithmiclinkfunction:Anewapproachwithstochastic covariates.psychometrika2019; 84(2): 422–446
-
[74]
Degtiar I, Rose S. A review of generalizability and transportability.Annual Review of Statistics and Its Application2023; 10: 501–524. 32 REMIRO-AZÓCAR
-
[75]
Happich M, Brnabic A, Faries D, et al. Reweighting randomized controlled trial evidence to better reflect real life–a case study of the Innovative Medicines Initiative.Clinical Pharmacology & Therapeutics2020; 108(4): 817–825
-
[76]
Lu G, Ades A. Assessing evidence inconsistency in mixed treatment comparisons.Journal of the American Statistical Association 2006; 101(474): 447–459
work page 2006
-
[77]
Webster-Clark M, Keil AP. How Choice of Effect Measure Influences Minimally Sufficient Adjustment Sets for External Validity.American Journal of Epidemiology2023: kwad041
-
[78]
Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods.Statistics in medicine2019; 38(11): 2074–2102
work page 2074
-
[79]
R: A language and environment for statistical computing
Team RC, others . R: A language and environment for statistical computing. 2013
work page 2013
-
[80]
Song F, Altman DG, Glenny AM, Deeks JJ. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analyses.Bmj 2003; 326(7387): 472
work page 2003
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.