REVIEW 3 major objections 3 minor
Copas-Jackson-type bounds for publication bias over a general class of selection models
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that worst-case bounds for publication bias in meta-analysis can be computed over a wider nonparametric class of selection models than the Copas-Jackson bound allows, using tractable nonlinear programming with linear constr
desk verdict A legitimate generalization of the Copas-Jackson bound, but the abstract never says whether the NLP approximation is conservative, and that is the make-or-break question. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the selection model, which describes the probability that a study is published as a function of its outcome and characteristics. The Copas-Jackson bound is an analytical worst-case bias bound over selection models that are monotone in study standard error. The paper's machinery replaces that restrictive class with a broader nonparametric class and reformulates the worst-case bias as an optimization problem. The approximate worst-case bound is then obtained by tractable nonlinear programming with linear constraints, which is the mechanism that makes the generalization computable.
What would settle it
Construct a concrete non-monotone selection model in the proposed general class, compute the true worst-case bias over that model by exhaustive search or an analytic formula, and then run the proposed nonlinear program. If the program's approximate worst-case bound is smaller than the true worst case, the method understates publication bias and the central claim fails.
Extended reading notes
Core claim
The paper's central claim is that the Copas-Jackson bound, which holds over selection models where publication probability is monotone in study standard error, can be extended to a general class of selection models. For that general class the paper constructs an approximate worst-case bound by solving a nonlinear program whose constraints are linear. The practical question is: when a meta-analyst cannot credibly assume monotonicity in standard error, can they still put a worst-case number on publication bias? The paper's answer is yes, with a bound that generalizes the Copas-Jackson construction and remains computable in practice.
Load-bearing premise
The load-bearing premise is that the nonlinear programming relaxation faithfully represents the full general class of selection models, so that the computed approximate worst-case bound never falls below the true worst-case publication bias.
Editorial extensions
If this is right
- Meta-analysts can perform publication-bias sensitivity analysis without requiring publication probability to be monotone in standard error.
- The proposed worst-case bound can be computed numerically even though it covers a more flexible class of selection models.
- The bound provides a more objective sensitivity check than simple graphical methods like trim-and-fill, while weakening the assumptions of the Copas-Jackson bound.
- If the bound is used in practice, conclusions from meta-analyses can be reported with a worst-case bias interval under a broader class of publication mechanisms.
Reading between the lines
- The same computational strategy could be applied to other sensitivity quantities, such as worst-case shifts in effect size, by keeping the selection-model constraints and changing the objective.
- If the approximate bound is conservative, it could be inverted to identify which selection models would overturn a meta-analytic conclusion, giving a threshold for criticism.
- The method might extend to meta-regression or multivariate outcomes, where monotonicity in standard error is even harder to defend; this extension is not claimed in the paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a method for constructing worst-case publication-bias bounds over a broader class of selection models than those allowed by the Copas-Jackson (C-J) bound. The abstract states that the C-J bound covers only selection models monotone in study standard error, while the proposed method weakens this assumption and computes an approximate worst-case bound via tractable nonlinear programming with linear constraints. The effectiveness is claimed to be substantiated by extensive simulations and two real-world meta-analyses. The central contribution, if valid, is a sensitivity-analysis tool that is less assumption-dependent than the existing C-J bound.
Significance. Worst-case bounds for publication bias are valuable because they avoid strong parametric assumptions in sensitivity analysis. The C-J bound is a nonparametric benchmark, but its monotonicity restriction is limiting. If the proposed generalization is rigorous and the NLP approximation is provably conservative, the paper would make a useful methodological contribution. The significance is therefore potentially high. However, because the full text is not available for review and the abstract omits several load-bearing technical details, the significance cannot be confirmed from the submitted material alone.
major comments (3)
- [Abstract, sentence 4] The phrase 'approximate worst-case bound via tractable nonlinear programming with linear constraints' does not state whether the approximation is an upper bound (conservative) or a lower bound (optimistic). In publication-bias sensitivity analysis, a non-conservative approximation is problematic: if the NLP optimum undershoots the true supremum of bias over the allowed selection models, the resulting interval is too narrow and gives false reassurance. The manuscript must either prove that the NLP formulation yields an upper bound over the entire model class, or specify the sense in which the approximation is valid (e.g., asymptotically conservative under discretization refinement). Without such a statement the central claim is not established.
- [Abstract, sentence 3] The 'general class' of selection models is not defined. The contribution rests on the claim that this class strictly generalizes the C-J monotonicity class, but the abstract gives no membership conditions. The paper must formally define the class, state the assumptions imposed on the selection function (e.g., monotonicity in a different variable, bounds on selection probabilities, smoothness), and show that the supremum of bias over this class is finite and representable by the proposed finite-dimensional NLP. Without this, the bound could be vacuous, infinite, or not actually weaker than the C-J assumption.
- [Abstract, sentence 5] The validation claim 'extensive simulation studies' and 'two real-world meta-analyses' is asserted without any summary of the results. The paper should report, at minimum, the simulation design (data-generating models, selection mechanisms, sample sizes), the outcome measure (e.g., coverage rate of the true bias or true effect), and a comparison with the C-J bound. Crucially, the simulations must verify conservatism: the approximate worst-case bound should be at or above the true worst-case publication bias in the scenarios studied. The abstract alone gives no evidence for this property, which is the central risk identified above.
minor comments (3)
- [Abstract, general] The abstract would benefit from a precise statement of the optimization problem and the nature of the approximation, even in qualitative terms. For example, is the NLP an inner or outer approximation of the feasible set, and does the 'linear constraints' formulation follow from discretization or from convexification?
- [Abstract, sentence 1] The acronym PB is introduced but not used again; consider removing it or using it consistently. Also, the phrase 'provide a more objective evaluation' is vague; the comparison to trim-and-fill should be stated in terms of assumptions or bias, not 'objectivity'.
- [Not applicable] No software or code availability is mentioned. For a numerical method, sharing code would improve reproducibility; this is a minor point for the revision.
Circularity Check
No circularity detectable from abstract; derivation chain is not self-referential.
full rationale
The abstract describes a method that generalizes the external Copas-Jackson bound by constructing worst-case bounds over a broader class of selection models via nonlinear programming with linear constraints. There is no visible step in which a fitted parameter or data quantity is renamed as a prediction, no definition of the target quantity in terms of the method's own outputs, and no load-bearing self-citation. The stated goal is to weaken an existing assumption, and the evaluation is against simulations and real meta-analyses, which are external checks. Concerns that the approximate bound may or may not be conservative are correctness or validation risks, not circularity, and the full text is not available to check for hidden reductions. Therefore, under the rule that honest non-finding is expected when no specific circular step can be quoted, the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Numerical approximation/discretization parameters of the NLP bound
assumptions (3)
- domain assumption Standard meta-analysis measurement model: observed effect estimates are normally distributed around study-specific true effects with known standard errors, so selective publication acts only through study-level selection probabilities.
- ad hoc to paper The 'general class' of selection models is characterized by a finite set of conditions that admit an exact or conservative nonlinear programming reformulation with linear constraints.
- ad hoc to paper The numerical solution of the nonlinear program is a conservative bound: it does not fall below the true worst-case publication bias over the general class.
Cite this review
Pith. "Pith review of Copas-Jackson-type bounds for publication bias over a general class of selection models." pith.science (2026). https://pith.science/paper/E353UVLJ
@misc{pith2026250817716,
author = {Pith},
title = {Pith review of: Copas-Jackson-type bounds for publication bias over a general class of selection models},
year = {2026},
howpublished = {\url{https://pith.science/paper/E353UVLJ}},
note = {Machine review of arXiv:2508.17716}
}
read the original abstract
Publication bias (PB) is one of the most vital threats to the accuracy of meta-analysis. Adjustment or sensitivity analysis based on selection models, which describe the probability of a study being published, provide a more objective evaluation of PB than widely-used simple graphical methods such as the trim-and-fill method. Most existing methods rely on parametric selection models. The Copas-Jackson bound (C-J bound) provides a worst-case bound of an analytical form over a nonparametric class of selection models, which would provide more robust conclusions than parametric sensitivity analysis. The nonparametric class of the selection models in the C-J bound is restrictive and only covers parametric selection models monotonic to the standard errors of outcomes. The novelty of this paper is to develop a method that constructs worst-case bounds over a general class of selection models weakening the assumption in the C-J bound. We propose an efficient numerical method to obtain an approximate worst-case bound via tractable nonlinear programming with linear constraints. We substantiate the effectiveness of the proposed bound with extensive simulation studies and show its applicability with two real-world meta-analyses.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.