Pith. sign in

REVIEW 3 major objections 8 minor 71 references

Imputing unreported outcomes to debias meta-analyses

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 08:31 UTC pith:H2VZVYKL

load-bearing objection Bivariate MI+importance sampling for ORB is a genuine extension; simulation evidence is optimistic due to fixed variance components. the 3 major comments →

arxiv 2607.07509 v1 pith:H2VZVYKL submitted 2026-07-08 stat.ME

Adjusting for Outcome Reporting Bias in Meta-analysis: A Multiple Imputation Approach

Cora Burgwinkel , Saverio Fontana , Leonhard Held This is my paper
classification stat.ME
keywords outcomesapproachadjustmentmeta-analysesbiasdatamultivariateunreported
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes a method to correct outcome reporting bias (ORB) in meta-analysis by treating unreported study outcomes as missing data. The approach works in two stages: first, missing outcome effect estimates and their standard errors are imputed under a missing-at-random assumption using a multivariate normal model, which can borrow strength across correlated outcomes in a bivariate extension; second, the imputed datasets are reweighted via importance sampling under a logistic selection model that assumes studies with larger effects or z-scores are more likely to be reported. The selection weight delta is not identifiable from the data, so the method is designed as a sensitivity analysis: varying delta traces how the pooled treatment estimate shifts as one assumes stronger or weaker selective reporting. The authors apply the method to a Cochrane review of topiramate for epilepsy and conduct a large simulation study.

Core claim

The bivariate extension of the multiple imputation approach reduces bias and improves coverage relative to naive (complete-case) meta-analysis by borrowing information across correlated outcomes, provided the outcomes are genuinely correlated and heterogeneity is not extreme. In the epilepsy application, ORB-adjusted estimates shifted systematically toward the null for the outcome with substantial non-reporting. The simulations show that the adjustment reduces bias across moderate heterogeneity levels, but residual bias persists at I²=90%, and the univariate version performs no better than the naive estimate under high heterogeneity. The method is robust to moderate misspecification of the選選

What carries the argument

The central mechanism is a two-stage pipeline: (1) multiple imputation of unreported study outcomes from a conditional multivariate normal distribution, where the covariance structure encodes within-study and between-study correlations, and (2) importance sampling reweighting of the imputed datasets using weights derived from a logistic selection model Pr(report) = expit(alpha + delta * theta_hat), where delta controls the strength of selective reporting. The adjusted pooled estimate is the weighted average of meta-analysis estimates across imputations, with variance computed via Rubin's rules.

Load-bearing premise

The simulation study fixes the between-study variance components (tau-squared and the between-study correlation) at their true generating values rather than estimating them from data, so the reported performance does not capture the additional uncertainty and potential degradation that would arise in real meta-analyses with few studies where these parameters are poorly estimated.

What would settle it

A simulation or empirical comparison showing that, when between-study variance components are estimated rather than fixed at truth, the importance sampling weights become unstable enough that the bivariate adjusted estimate has higher mean squared error than the naive estimate, particularly with few studies (K=6) and moderate-to-high heterogeneity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Meta-analysts can use this method as a sensitivity tool: by sweeping delta from 0 (no selection) to higher values, they can visualize how robust their pooled estimate is to assumed selective outcome reporting, analogous to how trim-and-fill or Egger regression are used for publication bias.
  • The bivariate extension is most valuable when one outcome is well-reported and a correlated outcome is poorly reported: the reported outcome informs imputation for the unreported one, but only if the within-study correlation can be estimated or plausibly approximated.
  • The method does not require ORBIT risk-of-bias classifications, unlike the Copas adjustment, making it applicable when such classifications are unavailable—but at the cost of requiring the analyst to assume a selection mechanism without external validation.
  • Under extreme heterogeneity (I²=90%), the method fails to fully remove bias, suggesting that ORB adjustment in highly heterogeneous evidence bases remains an open problem.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. This manuscript proposes a multiple imputation approach combined with importance sampling to adjust for outcome reporting bias (ORB) in both univariate and bivariate random-effects meta-analysis. Unreported study outcomes are imputed under a MAR assumption using conditional multivariate normal distributions, then reweighted under a logistic selection model (MNAR) via importance sampling. The bivariate extension borrows strength across correlated outcomes by jointly modeling their covariance structure. The method is applied to a Cochrane review on topiramate for epilepsy and evaluated in an extensive simulation study varying heterogeneity, number of studies, selection strength, and missingness proportions. The authors find that the bivariate ORB-adjustment reduces bias and improves coverage relative to naive estimates, particularly under moderate heterogeneity and moderate correlation, while residual bias persists under high heterogeneity (I²=90%).

Significance. The paper addresses a practically important and underdeveloped problem. ORB is common in clinical trials and few adjustment methods exist. The integration of multiple imputation with importance sampling for the bivariate setting is a natural and useful extension of Carpenter et al. (2011). The authors provide reproducible code and a prespecified simulation protocol on OSF, which is commendable. The method is framed as a sensitivity analysis (delta swept over a range), which is an appropriate and honest framing given the non-identifiability of the selection parameter. The real-data application demonstrates the practical relevance of the adjustment. The simulation study is extensive in its factorial design.

major comments (3)
  1. Section 4.1.1: The simulation study fixes variance components (τ² and ρ_B) at their true generating values throughout. This is load-bearing for the central performance claim because the imputation covariance matrix Σ_j (Eq. 2, Section 2.3.1) and the bivariate Σ (Section 2.3.2) depend directly on τ²_j and ρ_B, and the importance weights (Eq. 5) depend on imputed values that are functions of these parameters. Under ORB, the reported studies are a selected subset; τ² estimated from these studies could be biased (typically inflated under selection favoring larger effects), and ρ_B could differ from the true value. For K=6, τ² estimation is highly uncertain even without selection. The authors acknowledge this in Section 5, but the simulation evidence as presented does not support the broad performance claims under realistic conditions. A supplementary simulation with estimated (rather than固定)
  2. Section 4.2.5: The bivariate model non-convergence rate of 26–33% for K=6 is substantial. The authors handle this by redrawing failed replicates until 1900 successful fits are obtained, which introduces survivorship bias: the reported performance measures are conditional on convergence. This is load-bearing because K=6 is a common real-world scenario, and the methods that fail to converge may be precisely those where the adjustment is most needed (e.g., sparse data, high heterogeneity). The authors should report the characteristics of failed replicates and discuss the potential impact on the generalizability of the results.
  3. Section 2.4, Eq. (5): The bivariate importance weight uses a common delta across both outcomes. The authors note that outcome-specific delta_j can in principle be considered, but the simulation and application use a common delta. Since the selection mechanism in the simulation is imposed only on outcome 1 (Section 4.1), the common-delta assumption is misspecified by construction for outcome 2. The authors should clarify whether the reported bivariate performance gains (Figures 5–8) reflect this misspecification or whether outcome 2 is always fully reported (in which case the common delta is irrelevant for outcome 2). This affects the interpretation of the borrowing-strength claim.
minor comments (8)
  1. Section 2.1.1: The REM is written as θ̂_i ~ N(θ + ν_i, σ²_i) with ν_i ~ N(0, τ²). This is a two-level formulation; it would be clearer to write the marginal distribution θ̂_i ~ N(θ, σ²_i + τ²) to connect directly to the weighting formula that follows.
  2. Section 2.3.1: The covariance matrix Σ_j includes the term SE(θ̂_{j,MA})² · J_K. This term accounts for uncertainty in the naive MA estimate. It would help to state explicitly that this treats θ̂_{j,MA} as a fixed (estimated) quantity rather than integrating over its sampling distribution.
  3. Section 3.2: The estimated Pearson correlation between observed study-level effect estimates is r = −0.33 (95% CI: −0.90 to 0.66). This is used as a proxy for the within-study correlation ρ_W. The within-study and between-study correlations are conceptually different quantities; using one as a proxy for the other is a strong assumption. The authors could comment on this.
  4. Figure 1: The forest plot caption mentions 'r denotes the assumed within-study correlation' but the figure itself is difficult to parse. Consider enlarging or splitting into panels for readability.
  5. Section 4.1: The simulation uses continuous outcomes for 'computational convenience' but the application uses binary outcomes (log OR/RR). A brief comment on whether the method's performance is expected to differ for binary outcomes would be appropriate.
  6. Section 4.2.2: The text states 'The bias for the univariate ORB-adjusted estimate is as large as for the naive estimate and is around 0.2 for I²=90%.' This is an important finding that somewhat undercuts the value of the univariate adjustment and could be discussed more prominently in the discussion.
  7. The reference list includes arXiv preprints (e.g., Ref [41] Bai et al.). Authors should check whether published versions are now available.
  8. Section 2.2, Eq. (1): The standard error imputation formula uses k̂_j derived from reported studies. For studies with very different sample sizes from the reported set, this imputation may be poor. A brief comment on this limitation would be useful, though the functional form is appropriate.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. All three major comments identify legitimate concerns about the realism and interpretability of our simulation evidence. We agree with the substance of each comment and will revise the manuscript accordingly. Specifically: (1) we will add a supplementary simulation with estimated variance components and temper our performance claims; (2) we will report characteristics of failed replicates and add a discussion of survivorship bias; (3) we will clarify the role of the common-delta assumption in the bivariate simulation and its implications for the borrowing-strength claim. No standing objections remain.

read point-by-point responses

  1. Referee: Section 4.1.1: The simulation study fixes variance components (τ² and ρ_B) at their true generating values throughout. This is load-bearing for the central performance claim because the imputation covariance matrix and importance weights depend directly on these parameters. Under ORB, τ² estimated from reported studies could be biased, and for K=6 estimation is highly uncertain. A supplementary simulation with estimated rather than fixed variance components is needed.

    Authors: The referee is correct that fixing variance components at their true values is a substantive simplification that limits the generalizability of our performance claims. We acknowledge this explicitly in Section 5, but we agree that the current framing of the simulation results does not sufficiently temper the claims. We will address this in two ways. First, we will add a supplementary simulation in which τ² and ρ_B are estimated from the reported studies via REML rather than fixed at generating values. This will be conducted for a representative subset of scenarios (K=6 and K=12; I²=0%, 60%, 90%; δ=0, 0.4, 0.8; z-score and effect-estimate selection), which is computationally feasible within the revised scope. Second, we will revise the language in Sections 4.2 and 6 to state clearly that the main simulation results represent an upper bound on performance under known variance components, and that the supplementary results with estimated variance components provide a more realistic assessment. We expect, based on preliminary exploration, that performance will degrade modestly for K=6 under high heterogeneity, but we will report whatever the simulation shows honestly. revision: yes

  2. Referee: Section 4.2.5: The bivariate model non-convergence rate of 26–33% for K=6 is substantial. Redrawing failed replicates until 1900 successful fits introduces survivorship bias. The authors should report characteristics of failed replicates and discuss the potential impact on generalizability.

    Authors: We agree that the redrawing procedure introduces a form of survivorship bias and that the referee's concern about the generalizability of results for K=6 is well placed. We will make three changes. First, we will report the characteristics of failed replicates (distribution of number of reported studies, heterogeneity level, and selection strength among failed versus successful fits) in a supplementary table. Second, we will add a paragraph in Section 4.2.5 discussing the potential direction and magnitude of survivorship bias: if non-convergence is concentrated in scenarios with sparse data and high heterogeneity, the reported performance measures for K=6 are likely optimistic. Third, we will add a corresponding caveat in the Discussion (Section 5) noting that the method's practical applicability for small K may be more limited than the simulation results suggest, and that convergence diagnostics should be checked in applied work. We note that the non-convergence rate for K=12 and K=25 is substantially lower (below 1% for K=25), so the concern is primarily relevant to the K=6 setting. revision: yes

  3. Referee: Section 2.4, Eq. (5): The bivariate importance weight uses a common delta across both outcomes. Since selection in the simulation is imposed only on outcome 1, the common-delta assumption is misspecified by construction for outcome 2. The authors should clarify whether the reported bivariate performance gains reflect this misspecification or whether outcome 2 is always fully reported, and how this affects the borrowing-strength claim.

    Authors: We thank the referee for identifying this important point of ambiguity. To clarify: in the simulation, selective reporting is imposed only on outcome 1, while outcome 2 is always fully reported for all K studies. This means that outcome 2 has no unreported study outcomes, so the common delta in the bivariate importance weight (Eq. 5) does not introduce misspecification for outcome 2 in the simulation, because the summation over unreported studies (i∈U_j) is empty for j=2. The borrowing-strength benefit arises through the imputation step (Section 2.3.2), where the covariance structure between outcomes 1 and 2 informs the imputed values for the unreported outcome 1 studies, not through the importance weights for outcome 2. We will revise the manuscript to state explicitly that outcome 2 is fully reported in the simulation, that the common delta therefore only acts on outcome 1's unreported studies, and that the bivariate performance gains reflect borrowing through the imputation model rather than through differential weighting of outcome 2. We will also note that in settings where both outcomes have unreported studies and different selection mechanisms, outcome-specific delta_j would be needed, and this remains a direction for future work. This clarification does not change the simulation results but makes the borrowing-strength mechanism transparent. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the method is a transparent sensitivity analysis with δ swept, not estimated, and the simulation's fixed τ² is a limitation but not a definitional reduction.

full rationale

The paper proposes a multiple imputation + importance sampling approach for ORB adjustment. The derivation chain is: (1) compute naive MA estimate from reported studies, (2) impute unreported outcomes from a conditional multivariate normal centered on the naive MA estimate (Eq. 3), (3) apply importance weights from a logistic selection model with user-specified δ (Eq. 5), (4) compute weighted average of completed-data MA estimates (Eq. 6). At δ=0, weights are uniform and the adjusted estimate reduces to the naive estimate by construction — but this is explicitly acknowledged as the MAR special case, not presented as a finding. The selection weight δ is swept over a range and the authors state in Section 5 that 'δ is generally not identifiable from the data, necessitating a sensitivity analysis approach.' The simulation study fixes τ² and ρ_B at true generating values (Section 4.1.1), which the authors acknowledge makes performance 'somewhat optimistic' — this is a limitation of the simulation design, not a circularity in the method or derivation. Self-citations are to Held's textbook [53] for standard results (conditional normal distribution, Fisher z-transform, Wishart distribution) and to Saracini and Held [45] for a related selection model approach; neither is load-bearing for the central claim. The methodological foundation comes from Carpenter et al. [47], an external citation. No step in the derivation chain reduces to its own inputs by construction in a way that would constitute circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new entities, particles, forces, or dimensions are introduced. The method operates entirely within the standard framework of random-effects meta-analysis, multivariate normal distributions, and logistic selection models. The free parameters (delta, rho_W, M, k_hat_j) are standard sensitivity parameters or data-derived quantities, not new theoretical constructs.

free parameters (4)
  • delta (selection weight) = swept over {0.0, 0.1, ..., 1.3} in application; {0, 0.2, ..., 1.0} in simulation
    Controls the strength of selective reporting in the logistic selection model (Eq. 4). Not identifiable from data; the method is a sensitivity analysis over delta.
  • rho_W (within-study correlation) = r = -0.33 (Pearson correlation from observed study-level effects in clinical application); 0 or 0.4 in simulation
    Used in the bivariate within-study covariance matrix V_i (Section 2.1.2). Rarely reported in practice; approximated from studies reporting both outcomes or set as a sensitivity parameter.
  • M (number of imputations) = 1000 in clinical application; 200 in simulation
    Chosen for computational feasibility; authors state results were insensitive to reducing from 1000 to 200.
  • k_hat_j (standard error imputation constant) = Computed from reported studies via Eq. 1
    Used to impute missing standard errors for unreported outcomes. Derived from the inverse-variance-weighted average of sample sizes among reporting studies.
axioms (5)
  • domain assumption Unreported study outcomes follow a MAR mechanism conditional on the reported outcomes, which is then reweighted to MNAR via importance sampling.
    Section 2.3-2.4. The imputation model assumes the joint vector of study effects is multivariate normal centered on the naive MA estimate. The MNAR correction is applied post-hoc through weights.
  • domain assumption Study sample sizes are reported even when treatment effects and standard errors are selectively unreported.
    Section 1 and Section 2.2. This is required to impute missing standard errors via Eq. 1. The authors note this matches the Copas assumption.
  • ad hoc to paper The selection probability depends on the study-level effect estimate or z-score through a logistic function (Eq. 4).
    Section 2.4. The logistic form is inherited from Carpenter et al. [47]. Alternative functional forms for the selection mechanism are not explored.
  • ad hoc to paper Between-study variance components are known in the simulation study.
    Section 4.1.1. The authors acknowledge this is a simplification for computational feasibility and that it makes results 'somewhat optimistic.'
  • ad hoc to paper The bivariate selection mechanism uses a common delta across both outcomes (Eq. 5).
    Section 2.4. The authors note that 'in principle, outcome-specific selection weights delta_j can also be considered' but do not implement this.

pith-pipeline@v1.1.0-glm · 25414 in / 3458 out tokens · 547058 ms · 2026-07-09T08:31:53.719355+00:00 · methodology

0 comments
read the original abstract

Background: Outcome reporting bias (ORB) occurs when study outcomes are selectively reported based on their results. ORB potentially undermines the credibility and validity of meta-analyses and contributes to research waste by distorting overall treatment effects. ORB can be viewed as a missing data problem in which unreported study outcomes introduce bias. Despite the serious implications ORB poses, it remains an underrecognized issue, with only a few adjustment methods available. Methods: We propose an approach that addresses unreported study outcomes in meta-analyses through multiple imputation for univariate and multivariate meta-analysis. To assess the impact of ORB in meta-analyses, we apply our proposed methodology to real clinical data affected by ORB, and conduct a simulation study to evaluate the method's performance under a range of scenarios. Results: The proposed method provides bias-adjusted estimates under assumed selective non-reporting mechanisms. In the application to clinical data, ORB-adjusted estimates were systematically shifted towards less extreme treatment effects compared with naive analyses, highlighting the potential magnitude of ORB in practice. The simulation study shows that the extent of adjustment depends on the assumed selection mechanism and the degree of heterogeneity, with stronger selection leading to larger adjustment. Conclusions: Imputing unreported study outcomes provides a promising approach to address ORB in meta-analyses. The multivariate approach extends ORB adjustment to jointly model correlated outcomes, allowing borrowing of strength across outcomes. Overall, we propose a practical and flexible approach for evaluating the sensitivity of univariate and multivariate meta-analytic conclusions to ORB.

Figures

Figures reproduced from arXiv: 2607.07509 by Cora Burgwinkel, Leonhard Held, Saverio Fontana.

Figure 1
Figure 1. Figure 1: Forest plot for univariate and bivariate ORB-adjustment applied to Topiramate data for [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Outcome seizure freedom: comparison of within-study correlations for bivariate ORB￾adjustment for selection on the log OR/log RR and selection on the z-score over increasing selection [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Outcome seizure freedom: comparison of univariate and multivariate ORB-adjustment (for a correlation of r = −0.3) for selection on the log OR/log RR and selection on the z-score over increasing selection [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Simulation results for the confidence interval width of four MA estimates under ORB for [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Simulation results for the coverage of four MA estimates under ORB for [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Outcome 50% seizure reduction: comparison of naive and univariate ORB-adjusted esti￾mates for selection on the log OR/ log RR and selection on the z-score over increasing selection. As expected, the ORB-adjusted treatment effects and corresponding CIs are generally shifted toward the null compared to the naive MA estimates based only on the reported studies ( [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Outcome 50% seizure reduction: comparison of univariate and multivariate ORB￾adjustment (for a correlation of r = −0.3) for selection on the log OR/ log RR and selection on the z-score over increasing selection. Next, we compared the univariate and multivariate ORB adjustment for the outcome 50% seizure reduction, assuming a fixed within-study correlation of -0.3 (see [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of estimates for θ1 = θ2 = 0.4, ρB = ρW = 0.4, pi = 0.2 and δ1,sim = 0.8. The bias is shown for varying meta-analysis study sizes, heterogeneity levels, selection type and an increasing selection weight in the estimation. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of estimates for θ1 = θ2 = 0.4, ρB = ρW = 0.4, pi = 0.2 and δ1,sim = 0.8. The MSE is shown for varying meta-analysis study sizes, heterogeneity levels, selection type and an increasing selection weight in the estimation. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Comparison of estimates for θ1 = θ2 = 0.4, ρB = ρW = 0.4, pi = 0.2 and δ1,sim = 0.8. The CI width is shown for varying meta-analysis study sizes, heterogeneity levels, selection type and an increasing selection weight in the estimation. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Comparison of estimates for θ1 = θ2 = 0.4, ρB = ρW = 0.4, pi = 0.2 and δ1,sim = 0.8. The coverage is shown for varying meta-analysis study sizes, heterogeneity levels, selection type and an increasing selection weight in the estimation. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 71 canonical work pages · 1 internal anchor

  1. [1]

    Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors

    Chan AW and Altman DG. Identifying Outcome Reporting Bias in Randomised Trials on PubMed: Review of Publications and Survey of Authors. BMJ. 2005;330:753

    work page 2005

  2. [2]

    Outcome Reporting Bias in Clinical Trials Researching Disease-Modifying Therapy in Patients With Multiple Sclerosis

    Lemmens CM, Amerongen S van, Strijbis EM, and Killestein J. Outcome Reporting Bias in Clinical Trials Researching Disease-Modifying Therapy in Patients With Multiple Sclerosis. Neurology. 2024;102:e208032

    work page 2024

  3. [3]

    Increasing Value and Reducing Waste: Addressing Inac- cessible Research

    Chan AW, Song F, Vickers A, et al. Increasing Value and Reducing Waste: Addressing Inac- cessible Research. Lancet. 2014;383:257–266

    work page 2014

  4. [4]

    Comparison of Regis- tered and Published Outcomes in Randomized Controlled Trials: A Systematic Review

    Jones CW, Keil LG, Holland WC, Caughey MC, and Platts-Mills TF. Comparison of Regis- tered and Published Outcomes in Randomized Controlled Trials: A Systematic Review. BMC Med. 2015;13:1–12

    work page 2015

  5. [5]

    Reducing Waste from Incomplete or Unusable Reports of Biomedical Research

    Glasziou P, Altman DG, Bossuyt P, et al. Reducing Waste from Incomplete or Unusable Reports of Biomedical Research. Lancet. 2014;383:267–276

    work page 2014

  6. [6]

    Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias—An Updated Review

    Dwan K, Gamble C, Williamson PR, Kirkham JJ, and Group RB. Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias—An Updated Review. PLoS One. 2013;8:e66844

    work page 2013

  7. [7]

    Dissemination and Publication of Research Findings: An updated Review of Related Biases

    Song F, Parekh S, Hooper L, et al. Dissemination and Publication of Research Findings: An updated Review of Related Biases. Health Technol Assess. 2010;14:1–193

    work page 2010

  8. [8]

    Outcome Selection Bias in Meta- analysis

    Williamson PR, Gamble C, Altman DG, and Hutton J. Outcome Selection Bias in Meta- analysis. Stat Methods Med Res. 2005;14:515–524

    work page 2005

  9. [9]

    Outcome Reporting Bias in Randomized Trials funded by the Canadian Institutes of Health Research

    Chan AW, Krleˇ za-Jeri´ c K, Schmid I, and Altman DG. Outcome Reporting Bias in Randomized Trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171:735–740. 23

    work page 2004

  10. [10]

    Selective Reporting of Outcomes in Tinnitus Trials: Comparison of Trial Registries With Corresponding Publications

    Beurden Iv, Beek MJvd, Heteren JAv, Smit AL, and Stegeman I. Selective Reporting of Outcomes in Tinnitus Trials: Comparison of Trial Registries With Corresponding Publications. Front Neurol. 2021;12:669501

    work page 2021

  11. [11]

    Outcome Reporting Bias in Nephrology Randomized Clinical Trials: Examining Outcomes represented by Graphical Illustrations

    Ward F and Shiely F. Outcome Reporting Bias in Nephrology Randomized Clinical Trials: Examining Outcomes represented by Graphical Illustrations. Contemp Clin Trials Commun. 2022;28:100924

    work page 2022

  12. [12]

    Many Scenarios exist for Selective Inclusion and Report- ing of Results in Randomized Trials and Systematic Reviews

    Page MJ, McKenzie JE, and Forbes A. Many Scenarios exist for Selective Inclusion and Report- ing of Results in Randomized Trials and Systematic Reviews. J Clin Epidemiol. 2013;66:524– 537

    work page 2013

  13. [13]

    The Impact of Outcome Reporting Bias in Ran- domised Controlled Trials on a Cohort of Systematic Reviews

    Kirkham JJ, Dwan KM, Altman DG, et al. The Impact of Outcome Reporting Bias in Ran- domised Controlled Trials on a Cohort of Systematic Reviews. BMJ. 2010;340

    work page 2010

  14. [14]

    Selective Report- ing Bias of Harm Outcomes within Studies: Findings from a Cohort of Systematic Reviews

    Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, and Kirkham JJ. Selective Report- ing Bias of Harm Outcomes within Studies: Findings from a Cohort of Systematic Reviews. BMJ. 2014;349

    work page 2014

  15. [15]

    Catalogue of Bias: Selective Outcome Reporting Bias

    Thomas ET and Heneghan C. Catalogue of Bias: Selective Outcome Reporting Bias. BMJ Evid Based Med. 2022;27:370–372

    work page 2022

  16. [16]

    Outcome Report- ing Recommendations for Clinical Trial Protocols and Reports: A Scoping Review

    Butcher NJ, Mew EJ, Monsour A, Chan AW, Moher D, and Offringa M. Outcome Report- ing Recommendations for Clinical Trial Protocols and Reports: A Scoping Review. Trials. 2020;21:1–17

    work page 2020

  17. [17]

    Outcome Reporting Bias in Randomized-controlled Trials Investigating Antipsychotic Drugs

    Lancee M, Lemmens C, Kahn R, Vinkers C, and Luykx J. Outcome Reporting Bias in Randomized-controlled Trials Investigating Antipsychotic Drugs. Transl Psychiatry. 2017;7:e1232–e1232

    work page 2017

  18. [18]

    Protocol Reg- istration and Selective Outcome Reporting in recent Psychiatry Trials: New Antidepressants and Cognitive Behavioural Therapies

    Shinohara K, Tajika A, Imai H, Takeshima N, Hayasaka Y, and Furukawa TA. Protocol Reg- istration and Selective Outcome Reporting in recent Psychiatry Trials: New Antidepressants and Cognitive Behavioural Therapies. Acta Psychiatr Scand. 2015;132:489–498

    work page 2015

  19. [19]

    Has the Degree of Outcome Reporting Bias in Surgical Ran- domized Trials changed? A Meta-regression Analysis

    Wang A, Menon R, Li T, et al. Has the Degree of Outcome Reporting Bias in Surgical Ran- domized Trials changed? A Meta-regression Analysis. ANZ J Surg. 2023;93:76–82

    work page 2023

  20. [20]

    Bias due to Selective Inclusion and Reporting of Outcomes and Analyses in Systematic Reviews of Randomised Trials of Healthcare Interven- tions

    Page MJ, McKenzie JE, Kirkham J, et al. Bias due to Selective Inclusion and Reporting of Outcomes and Analyses in Systematic Reviews of Randomised Trials of Healthcare Interven- tions. Cochrane Database Syst Rev. 2014

    work page 2014

  21. [21]

    Selective Outcome Reporting in Trials of Behavioural Health Interventions in Health Psychology and Behavioural Medicine Journals: A Review

    Matvienko-Sikar K, O’Shea J, Kennedy S, et al. Selective Outcome Reporting in Trials of Behavioural Health Interventions in Health Psychology and Behavioural Medicine Journals: A Review. Health Psychol Rev. 2024:1–15

    work page 2024

  22. [22]

    Selective Outcome Reporting Across Psychopharmacotherapy Randomized Controlled Trials

    Lancee M, Schuring M, Tijdink JK, Chan AW, Vinkers CH, and Luykx JJ. Selective Outcome Reporting Across Psychopharmacotherapy Randomized Controlled Trials. Int J Methods Psy- chiatr Res. 2022;31:e1900

    work page 2022

  23. [23]

    Transparency of Outcome Reporting and Trial Registration of Randomized Controlled Trials in Top Psychosomatic and Behavioral Health Journals: A Systematic Review

    Milette K, Roseman M, and Thombs BD. Transparency of Outcome Reporting and Trial Registration of Randomized Controlled Trials in Top Psychosomatic and Behavioral Health Journals: A Systematic Review. J Psychosom Res. 2011;70:205–217

    work page 2011

  24. [24]

    Publication Bias and Selective Outcome Reporting in Randomized Controlled Trials Related to Rehabilitation: A Literature Review

    Komukai K, Sugita S, and Fujimoto S. Publication Bias and Selective Outcome Reporting in Randomized Controlled Trials Related to Rehabilitation: A Literature Review. J Phys Med Rehabil. 2024;105:150–156

    work page 2024

  25. [25]

    Clinical Trials: What A Waste

    Ioannidis JP. Clinical Trials: What A Waste. BMJ. 2014;349

    work page 2014

  26. [26]

    Multivariate Meta-analysis helps examine the Impact of Outcome Reporting Bias in Cochrane Rheumatoid Arthritis Reviews

    Frosi G, Riley RD, Williamson PR, and Kirkham JJ. Multivariate Meta-analysis helps examine the Impact of Outcome Reporting Bias in Cochrane Rheumatoid Arthritis Reviews. J Clin Epidemiol. 2015;68:542–550. 24

    work page 2015

  27. [27]

    Protocol: Assessment of Outcome Reporting Bias in Studies included in Campbell Syst Rev

    Littell JH, Gorman DM, Valentine JC, and Pigott TD. Protocol: Assessment of Outcome Reporting Bias in Studies included in Campbell Syst Rev. Campbell Syst Rev. 2023;19:e1332

    work page 2023

  28. [28]

    Selective Outcome Reporting Bias is Highly Prevalent in Randomized Clinical Trials of Nonsurgical Periodontal Therapy

    Souza NV, Nicolini AC, Dos Reis INR, Sendyk DI, Cavagni J, and Pannuti CM. Selective Outcome Reporting Bias is Highly Prevalent in Randomized Clinical Trials of Nonsurgical Periodontal Therapy. J Periodontal Res. 2023;58:1–11

    work page 2023

  29. [29]

    Many Randomized Trials in a Large Systematic Review were not registered and had Evidence of Selective Outcome Reporting: A Meta-epidemiological Study

    Silva S, Singh S, Kashif S, Ogilvie R, Pinto RZ, and Hayden JA. Many Randomized Trials in a Large Systematic Review were not registered and had Evidence of Selective Outcome Reporting: A Meta-epidemiological Study. J Clin Epidemiol. 2024:111568

    work page 2024

  30. [30]

    Association of Trial Registration with Reporting of Primary Outcomes in Protocols and Publications

    Chan AW, Pello A, Kitchen J, et al. Association of Trial Registration with Reporting of Primary Outcomes in Protocols and Publications. JAMA. 2017;318:1709–1711

    work page 2017

  31. [31]

    The Threat of Serious Outcome Reporting Bias in Random- ized Controlled Trials on Acute Ischemic Stroke to Evidence Synthesis: A Meta-epidemiological Study

    Zhang N, Long Y, Wang X, et al. The Threat of Serious Outcome Reporting Bias in Random- ized Controlled Trials on Acute Ischemic Stroke to Evidence Synthesis: A Meta-epidemiological Study. Cardiovasc Diagn Ther. 2025;15:1182–1193

    work page 2025

  32. [32]

    Systematic Review: Outcome Reporting Bias is a Problem in High Impact Factor Neurology Journals

    Howard B, Scott JT, Blubaugh M, Roepke B, Scheckel C, and Vassar M. Systematic Review: Outcome Reporting Bias is a Problem in High Impact Factor Neurology Journals. PLoS One. 2017;12:e0180986

    work page 2017

  33. [33]

    Outcome Discrepancies and Selective Report- ing: Impacting the Leading Journals? PLoS One

    Fleming PS, Koletsi D, Dwan K, and Pandis N. Outcome Discrepancies and Selective Report- ing: Impacting the Leading Journals? PLoS One. 2015;10:e0127495

    work page 2015

  34. [34]

    Assessing the Potential for Outcome Reporting Bias in a Review: A Tutorial

    Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, and Williamson PR. Assessing the Potential for Outcome Reporting Bias in a Review: A Tutorial. Trials. 2010;11:1– 10

    work page 2010

  35. [35]

    ROB-ME: A Tool for Assessing Risk of Bias due to Missing Evidence in Systematic Reviews with Meta-analysis

    Page MJ, Sterne JA, Boutron I, et al. ROB-ME: A Tool for Assessing Risk of Bias due to Missing Evidence in Systematic Reviews with Meta-analysis. BMJ. 2023;383

    work page 2023

  36. [36]

    An Evaluation of Bivariate Random-effects Meta-analysis for the Joint Synthesis of Two Correlated Outcomes

    Riley RD, Abrams K, Lambert P, Sutton A, and Thompson J. An Evaluation of Bivariate Random-effects Meta-analysis for the Joint Synthesis of Two Correlated Outcomes. Stat Med. 2007;26:78–97

    work page 2007

  37. [37]

    Multivariate Meta-analysis: Potential and Promise

    Jackson D, Riley R, and White IR. Multivariate Meta-analysis: Potential and Promise. Stat Med. 2011;30:2481–2498

    work page 2011

  38. [38]

    A Multivariate Meta-Analysis Approach for Re- ducing the Impact of Outcome Reporting Bias in Systematic Reviews

    Kirkham JJ, Riley RD, and Williamson PR. A Multivariate Meta-Analysis Approach for Re- ducing the Impact of Outcome Reporting Bias in Systematic Reviews. Stat Med. 2012;31:2179– 2195

    work page 2012

  39. [39]

    Multivariate Meta-Analysis: The Effect of Ignoring Within-Study Correlation

    Riley RD. Multivariate Meta-Analysis: The Effect of Ignoring Within-Study Correlation. J R Stat Soc Ser A Stat Soc. 2009;172:789–811

    work page 2009

  40. [40]

    Application and Investigation of a Bound for Outcome Re- porting Bias

    Williamson PR and Gamble C. Application and Investigation of a Bound for Outcome Re- porting Bias. Trials. 2007;8:1–12

    work page 2007

  41. [41]

    A Bayesian Selection Model for Correcting Outcome Reporting Bias With Application to a Meta-analysis on Heart Failure Interventions

    Bai R, Liu X, Lin L, et al. A Bayesian Selection Model for Correcting Outcome Report- ing Bias With Application to a Meta-analysis on Heart Failure Interventions. arXiv preprint arXiv:2110.08849. 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  42. [42]

    Multivariate Network Meta-analysis to Mitigate the Effects of Outcome Reporting Bias

    Hwang H and DeSantis SM. Multivariate Network Meta-analysis to Mitigate the Effects of Outcome Reporting Bias. Stat Med. 2018;37:3254–3266

    work page 2018

  43. [43]

    Bayesian Mixed Treatment Comparisons Meta-Analysis for Correlated Outcomes Subject to Reporting Bias

    Liu Y, DeSantis SM, and Chen Y. Bayesian Mixed Treatment Comparisons Meta-Analysis for Correlated Outcomes Subject to Reporting Bias. J R Stat Soc Ser C Appl Stat. 2018;67:127– 144

    work page 2018

  44. [44]

    Correcting for Outcome Reporting Bias in a Meta-analysis: A Meta-regression Approach

    Aert RC van and Wicherts JM. Correcting for Outcome Reporting Bias in a Meta-analysis: A Meta-regression Approach. Behav Res Methods. 2024;56:1994–2012. 25

    work page 2024

  45. [45]

    Addressing Outcome Reporting Bias in Meta-Analysis: A Selection Model Perspective

    Saracini AG and Held L. Addressing Outcome Reporting Bias in Meta-Analysis: A Selection Model Perspective. Stat Med. 2025;44:e70238

    work page 2025

  46. [46]

    Model-based Sensitivity Analysis for Outcome Reporting Bias in the Meta-analysis of Benefit and Harm Outcomes

    Copas J, Marson A, Williamson P, and Kirkham J. Model-based Sensitivity Analysis for Outcome Reporting Bias in the Meta-analysis of Benefit and Harm Outcomes. Stat Methods Med Res. 2019;28:889–903

    work page 2019

  47. [47]

    Assessing the Sensitivity of Meta-analysis to Selec- tion Bias: A Multiple Imputation Approach

    Carpenter J, R¨ ucker G, and Schwarzer G. Assessing the Sensitivity of Meta-analysis to Selec- tion Bias: A Multiple Imputation Approach. Biometrics. 2011;67:1066–1072

    work page 2011

  48. [48]

    Identification and Impact of Outcome Selection Bias in Meta- analysis

    Williamson PR and Gamble C. Identification and Impact of Outcome Selection Bias in Meta- analysis. Stat Med. 2005;24:1547–1561

    work page 2005

  49. [49]

    John Wiley & Sons, 2002

    Agresti A.Categorical Data Analysis. John Wiley & Sons, 2002

    work page 2002

  50. [50]

    CRC Press, 2020

    Schmid CH, Stijnen T, and White I.Handbook of Meta-Analysis. CRC Press, 2020

    work page 2020

  51. [51]

    A Practical Introduction to Multivariate Meta-analysis

    Mavridis D and Salanti G. A Practical Introduction to Multivariate Meta-analysis. Stat Meth- ods Med Res. 2013;22:133–158

    work page 2013

  52. [52]

    A Model-based Correction for Outcome Reporting Bias in Meta-analysis

    Copas J, Dwan K, Kirkham J, and Williamson P. A Model-based Correction for Outcome Reporting Bias in Meta-analysis. Biostatistics. 2014;15:370–383

    work page 2014

  53. [53]

    Springer, 2020

    Held L and Bov´ e DS.Likelihood and Bayesian Inference. Springer, 2020

    work page 2020

  54. [54]

    A Review and Assessment of Importance Sampling Meth- ods for Reliability Analysis

    Tabandeh A, Jia G, and Gardoni P. A Review and Assessment of Importance Sampling Meth- ods for Reliability Analysis. Structural Safety. 2022;97:102216

    work page 2022

  55. [55]

    Russell Sage Foundation, 2019

    Cooper H, Hedges LV, and Valentine JC.The Handbook of Research Synthesis and Meta- Analysis. Russell Sage Foundation, 2019

    work page 2019

  56. [56]

    Modeling Publication Selection Effects in Meta-analysis

    Hedges LV. Modeling Publication Selection Effects in Meta-analysis. Statist. Sci. 1992;7:246– 255

    work page 1992

  57. [57]

    Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies yielding Statistically Insignificant Mean Differences

    Hedges LV. Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies yielding Statistically Insignificant Mean Differences. J Educ Behav Stat. 1984;9:61–85

    work page 1984

  58. [58]

    Modelling Publication Bias in Meta- analysis: A Review

    Sutton AJ, Song F, Gilbody SM, and Abrams KR. Modelling Publication Bias in Meta- analysis: A Review. Stat Methods Med Res. 2000;9:421–445

    work page 2000

  59. [59]

    Egger M, Higgins JP, and Smith GD.Systematic Reviews in Health Care: Meta-Analysis in Context. 3rd. John Wiley & Sons, 2022

    work page 2022

  60. [60]

    Topiramate Add-on for Drug-resistant Partial Epilepsy

    Pulman J, Jette N, Dykeman J, Hemming K, Hutton JL, and Marson AG. Topiramate Add-on for Drug-resistant Partial Epilepsy. Cochrane Database Syst Rev. 2014

    work page 2014

  61. [61]

    Topiramate Add-on Therapy for Drug-resistant Focal Epilepsy

    Bresnahan R, Hounsome J, Jette N, Hutton JL, and Marson AG. Topiramate Add-on Therapy for Drug-resistant Focal Epilepsy. Cochrane Database Syst Rev. 2019

    work page 2019

  62. [62]

    Quantifying Heterogeneity in a Meta-analysis

    Higgins JP and Thompson SG. Quantifying Heterogeneity in a Meta-analysis. Stat Med. 2002;21:1539–1558

    work page 2002

  63. [63]

    A Re-evaluation of Random-effects Meta- analysis

    Higgins JP, Thompson SG, and Spiegelhalter DJ. A Re-evaluation of Random-effects Meta- analysis. J R Stat Soc Ser A Stat Soc. 2009;172:137–159

    work page 2009

  64. [64]

    The Hartung-Knapp-Sidik-Jonkman Method for Random Effects Meta-analysis is straightforward and considerably outperforms the Standard DerSimonian-Laird Method

    IntHout J, Ioannidis JP, and Borm GF. The Hartung-Knapp-Sidik-Jonkman Method for Random Effects Meta-analysis is straightforward and considerably outperforms the Standard DerSimonian-Laird Method. BMC Med Res Methodol. 2014;14:25

    work page 2014

  65. [65]

    A Comparison of Heterogeneity Variance Estimators in Simulated Random-effects Meta-analyses

    Langan D, Higgins JP, Jackson D, et al. A Comparison of Heterogeneity Variance Estimators in Simulated Random-effects Meta-analyses. Res Synth Methods. 2019;10:83–98

    work page 2019

  66. [66]

    Effect of Reporting Bias on Meta-analyses of Drug Trials: Reanalysis of Meta-analyses

    Hart B, Lundh A, and Bero L. Effect of Reporting Bias on Meta-analyses of Drug Trials: Reanalysis of Meta-analyses. BMJ. 2012;344:d7202

    work page 2012

  67. [67]

    Sensitivity Analysis after Multiple Imputation Under Missing At Random: A Weighting Approach

    Carpenter JR, Kenward MG, and White IR. Sensitivity Analysis after Multiple Imputation Under Missing At Random: A Weighting Approach. Stat Methods Med Res. 2007;16:259–275. 26

    work page 2007

  68. [68]

    Evaluation of a Weight- ing Approach for Performing Sensitivity Analysis after Multiple Imputation

    Hayati Rezvan P, White IR, Lee KJ, Carlin JB, and Simpson JA. Evaluation of a Weight- ing Approach for Performing Sensitivity Analysis after Multiple Imputation. BMC Med Res Methodol. 2015;15:83

    work page 2015

  69. [69]

    A Selection Model for Accounting for Publication Bias in a Full Network Meta-analysis

    Mavridis D, Welton NJ, Sutton A, and Salanti G. A Selection Model for Accounting for Publication Bias in a Full Network Meta-analysis. Stat Med. 2014;33:5399–5412

    work page 2014

  70. [70]

    Demystifying Trial Networks and Network Meta- analysis

    Mills EJ, Thorlund K, and Ioannidis JP. Demystifying Trial Networks and Network Meta- analysis. BMJ. 2013;346

    work page 2013

  71. [71]

    Meta-analysis of Multitreatment Studies

    Hasselblad V. Meta-analysis of Multitreatment Studies. Med Decis Making. 1998;18:37–43. 27 Supplementary Material Application In the manuscript, we focused primarily on the outcomeseizure freedomas it has more unreported study outcomes and therefore provides a clearer visualization of our ORB adjustment method. For completeness, this section presents the ...

    work page 1998