Pith. sign in

REVIEW 3 major objections 5 minor 30 references

Analyzing Basket Trials under Multisource Exchangeability Assumptions

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces basket, the first R package for Bayesian multisource exchangeability analysis of basket trials, and claims it yields valid posterior inference for response rates, exchangeability, effective sample sizes, and cluster…

desk verdict Useful package, but the paper's equations are not the symmetric MEM and no validation supports the implementation. read the letter →

arxiv 1908.00618 v1 pith:PKTP2HTU submitted 2019-08-01 stat.CO

Michael J. Kane , Nan Chen , Alexander M. Kaizer , Xun Jiang , H. Amy Xia , Brian P. Hobbs This is my paper · ORCID
classification stat.CO MSC 62F1562P1062-04
keywords BayesiananalysisbasketdesignhierarchicalmodelmasterprotocoloncologypatientheterogeneitymultisourceexchangeabilityRpackage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Basket trials enroll several subpopulations that share a molecular target, but the treatment may work in some groups and fail in others, so pooling everyone is biased while analyzing each group alone is underpowered. This paper presents basket, an R package that fits the Bayesian multisource exchangeability model (MEM), which gives every pair of subpopulations a probability of being exchangeable and borrows strength only among exchangeable pairs. The package reports posterior response rates, a pairwise posterior exchangeability probability matrix, effective sample sizes, and cluster assignments, and it can designate “poolable” meta-baskets. The authors demonstrate the workflow on the vemurafenib BRAF V600 trial, where the analysis separates low-responding colorectal and bile-duct baskets from high-responding lung, histiocytosis, and thyroid baskets.

What carries the argument

The central machinery is the exchangeability configuration matrix Ω, a symmetric J×J binary matrix whose diagonal entries are fixed at 1 and whose off-diagonal entry Ωij indicates whether baskets i and j are statistically exchangeable. The model evaluates all configurations of Ω; Equation 1 supplies the marginal density of a basket's response count conditional on a configuration, and Equation 3 converts those marginal densities into posterior configuration probabilities. The PEP matrix is obtained by summing posterior weights of configurations with Ωij=1, and the maximum a posteriori MEM is the single most probable configuration. Effective sample sizes are computed by matching each posterior HPD interval to the nearest beta distribution through a simulated-annealing search, yielding conservative estimates of borrowing. This machinery replaces the single shrinkage parameter of conventional hierarchical models with source-specific exchangeability relationships.

What would settle it

Simulate basket trials with known response rates for J = 3,...,12 under configurations where some baskets are exchangeable and others are not, then run mem_mcmc() with default settings and compare the resulting posterior exchangeability probabilities, response-rate means, and 95% HPD coverage to mem_exact() (for small J) or to long-run exact calculations; if the MCMC results deviate by more than Monte Carlo error, or if HPD coverage falls systematically below 95%, the paper's claim of valid posterior inference with default settings fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that basket is the first available software package implementing Bayesian analysis with the multisource exchangeability model (MEM), and that it provides valid posterior inference for basket-specific response rates, pairwise exchangeability probabilities, effective sample sizes, and cluster assignments. Whereas a single-source hierarchical model uses one shrinkage parameter for all subpopulations, the MEM enumerates all possible pairwise exchangeability configurations and weights each configuration by its marginal likelihood. The package computes the posterior exchangeability probability (PEP) matrix by summing posterior weights over configurations in which a given pair is exchangeable, and it identifies the maximum a posteriori configuration as the most probable partition. It implements both exact enumeration, recommended for fewer than seven baskets, and a Metropolis MCMC sampler that extends the analysis to more than twenty baskets. In the vemurafenib case study, the analysis yields two clusters, with posterior probabilities near zero that the low-responding colorectal baskets are exchangeable with the higher-responding lung, histiocytosis, and thyroid baskets.

Load-bearing premise

The load-bearing premise is that the MCMC sampler with the default 50,000-iteration burn-in and 200,000 iterations converges reliably to the MEM posterior for every basket configuration, and that the code implementing Equation 1 exactly matches the model—an assumption the paper does not support with simulation studies or convergence diagnostics.

Editorial extensions

If this is right

  • Analysts can report a posterior exchangeability probability for every pair of baskets, making the amount of borrowed information explicit rather than hidden in a global shrinkage parameter.
  • The MAP configuration and cluster assignments designate meta-baskets, so trial reports can state which subpopulations form a common pool and which are treated separately.
  • Effective sample sizes quantify how much each basket's posterior precision owes to other baskets, which is directly relevant for interpreting sparsely enrolled subpopulations.
  • The MCMC implementation extends MEM analyses beyond 20 baskets, so the method is not confined to small trials with a handful of subpopulations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the PEP matrix could become a routine reporting element in early-phase oncology trials, giving readers a direct numerical measure of heterogeneity rather than a single pooled estimate.
  • The same pairwise exchangeability machinery is not tied to tumor histology and could be applied to any multi-cohort binary-endpoint trial, such as rare-disease platform studies or trials testing one agent across several molecular subtypes.
  • A calibrating study that runs mem_exact() and mem_mcmc() on simulated trials across a grid of J and response-rate configurations would give users principled guidance for setting burn-in and iteration counts, a step the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This manuscript presents the R package basket for Bayesian analysis of basket trials under the multisource exchangeability model (MEM) of Hobbs and Landin (2018). It describes exact and MCMC implementations, accessor functions for posterior exchangeability probabilities, MAP configurations, effective sample sizes, posterior probabilities of response, and visualization tools, and demonstrates the package on the vemurafenib basket trial. The central claim is that basket is the first available software package implementing Bayesian analysis with the MEM and that it provides valid posterior inference for response rates, exchangeability probabilities, effective sample sizes, and cluster assignments in basket trials.

Significance. If the implementation is correct, the package would fill a genuine gap: no other software implements symmetric MEM posterior inference, and the case study with reproducible code is useful to practitioners. The paper provides the data, commands, and output for a real oncology trial, and the availability of both exact enumeration for small J and MCMC for larger J is appealing. The contribution is potentially significant for clinical trial methodology. However, the manuscript does not currently establish that the implemented model is the symmetric MEM, and the lack of validation against exact enumeration or simulation leaves the central statistical claim unsecured. The present version requires a corrected model specification and empirical validation before the claims can be accepted.

major comments (3)
  1. [§2.2, Eqs. (1)–(3)] Equations (1)–(3) specify a row-wise posterior, not the symmetric MEM posterior. For a symmetric exchangeability configuration Ω, the integrated likelihood is a product of beta-binomial factors over exchangeability clusters, ∏_c B(a + Σ_{i∈c} S_i, b + Σ_{i∈c}(n_i − S_i))/B(a,b). Equation (1) instead conditions on a single row Ω_j and treats every basket not exchangeable with j as a singleton, even when those baskets are exchangeable with one another; Equation (3) is therefore the posterior of Ω_j under that row-wise model. The text then states that P(Ω|S) is the product of row-wise calculations, which for J=3 with all baskets exchangeable multiplies the single-cluster likelihood three times. If mem_exact or mem_mcmc implement this product, the reported PEPs, MAP configuration, and posterior means are not MEM posteriors. Please correct the model equations to the global symmetric formulation, or clearly state and justify a row-wise approximation and confirm which one the package implements.
  2. [§3.1] No validation of the MCMC sampler is provided. The paper asserts that mem_mcmc approximates P(Ω_j = ω_g | S) and that the default 200,000 iterations with 50,000 burn-in are a practical lower bound, but it reports no convergence diagnostics, no comparison to mem_exact for configurations where exact enumeration is feasible, and no simulation study in which true response rates and exchangeability structures are known. Because the package's validity claim rests on this sampler, please add an exact-versus-MCMC comparison for J ≤ 6 and a small simulation study, and report trace plots or Gelman–Rubin statistics for the case study.
  3. [§2.2 (PEP and MAP definitions)] The definition of the PEP via P(Ω|S) as a product of row-wise probabilities is not a probability distribution over symmetric MEMs, since the factors are not normalized joint probabilities and different rows share the same cluster memberships. This makes the PEP matrix and MAP configuration ill-defined unless a joint model is specified. Please provide the joint posterior over exchangeability partitions (e.g., over set partitions of {1,...,J}) and define PEPs as sums of that joint posterior.
minor comments (5)
  1. [§2.2] The word "singelton" appears where "singleton" is intended; please correct the typo.
  2. [§3.2] The method name "print.exchageability_model()" contains a typo; it should be "print.exchangeability_model()".
  3. [References] The reference list gives the basket package version as 0.9.2 while the text states version 1.0.0; please make these consistent.
  4. [§3.1] The complexity statement says the number of configurations grows as O(2^{J^2}), but the number of symmetric binary matrices with unit diagonal is 2^{J(J−1)/2}; please correct the exponent.
  5. [§4] The text says the case study comprises "six organs" but then lists five organ sites (with colorectal cancer split into two baskets); please clarify whether six baskets from five organ sites are intended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the package implements an externally published MEM; no prediction reduces by construction to a fitted input.

full rationale

The paper's derivation chain begins with the MEM specification of Hobbs and Landin (2018) and Kaizer et al. (2017), which are external inputs rather than outputs of this manuscript. Equations (1)-(3) define the marginal likelihood and posterior exchangeability probabilities directly from the beta-binomial model and the prior over exchangeability configurations; the PEP matrix, MAP configuration, posterior means, and posterior probabilities are computed by enumerating or sampling these configurations, not by fitting a parameter and then renaming it as a prediction. The effective sample size routine uses a beta approximation to the posterior HPD interval, but this is an internal descriptive utility and does not feed back into the model or into the package's central claim of implementing MEM inference. The case study is an illustration, not a predictive validation. The absence of simulation or exact-versus-MCMC convergence checks is a correctness/robustness concern, not a circularity: the equations and algorithm are stated openly and could fail to match the intended model without making the derivation circular. Self-citations to the original MEM papers are load-bearing in the sense that the method is adopted from prior work, but that is normal scientific attribution, and the cited work is external and not constrained by this paper's outputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim of a working software implementation rests primarily on the correctness of the cited MEM model (an external published framework) and the trustworthiness of the MCMC approximation. No new entities are introduced.

assumptions (3)
  • domain assumption The marginal density formula m(Sj|Omega_j, S(-j)) in Equation 1 correctly represents the symmetric MEM model for binary outcomes.
    The paper references Hobbs and Landin (2018) for this formula; correctness of the package's posterior inference depends on this transcription.
  • domain assumption The Metropolis MCMC algorithm described in Section 3.1 converges to the posterior distribution over exchangeability configurations with the default burn-in and iteration counts.
    No convergence diagnostics or simulation validation are presented; the paper asserts default settings are adequate.
  • domain assumption The reference prior with off-diagonal exchangeability probabilities of 0.5 is an appropriate neutral default that yields unbiased posterior exchangeability estimates.
    The paper adopts this prior as default from Hobbs and Landin (2018) without empirical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing Basket Trials under Multisource Exchangeability Assumptions." pith.science (2026). https://pith.science/paper/PKTP2HTU

@misc{pith2026190800618,
  author       = {Pith},
  title        = {Pith review of: Analyzing Basket Trials under Multisource Exchangeability Assumptions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PKTP2HTU}},
  note         = {Machine review of arXiv:1908.00618}
}
read the original abstract

Basket designs are prospective clinical trials that are devised with the hypothesis that the presence of selected molecular features determine a patient's subsequent response to a particular "targeted" treatment strategy. Basket trials are designed to enroll multiple clinical subpopulations to which it is assumed that the therapy in question offers beneficial efficacy in the presence of the targeted molecular profile. The treatment, however, may not offer acceptable efficacy to all subpopulations enrolled. Moreover, for rare disease settings, such as oncology wherein these trials have become popular, marginal measures of statistical evidence are difficult to interpret for sparsely enrolled subpopulations. Consequently, basket trials pose challenges to the traditional paradigm for trial design, which assumes inter-patient exchangeability. The R-package \pkg{basket} facilitates the analysis of basket trials by implementing multi-source exchangeability models. By evaluating all possible pairwise exchangeability relationships, this hierarchical modeling framework facilitates Bayesian posterior shrinkage among a collection of discrete and pre-specified subpopulations. Analysis functions are provided to implement posterior inference of the response rates and all possible exchangeability relationships between subpopulations. In addition, the package can identify "poolable" subsets of and report their response characteristics. The functionality of the package is demonstrated using data from an oncology study with subpopulations defined by tumor histology.

Figures

Figures reproduced from arXiv: 1908.00618 by the authors.

Figure 1
Figure 1. A conventional single-source Bayesian hierarchical model with [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Two example exchangeability configurations of the MEM. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Posterior distributions of the MEM analysis. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The exchangeogram for the Vemurafenib study. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 28 canonical work pages

  1. [1]

    Bayesian hierarchical modeling of patient subpopulations: Efficient designs of Phase II oncology clinical trials

    Berry SM, Broglio KR, Groshen S, Berry DA (2013). Bayesian hierarchical modeling of patient subpopulations: Efficient designs of Phase II oncology clinical trials. Clinical Trials, 10(5), 720--734

    work page 2013

  2. [2]

    Bayesian Adaptive Methods for Clinical Trials

    Berry SM, Carlin BP, Lee JJ, M\" u ller P (2010). Bayesian Adaptive Methods for Clinical Trials. Chapman and Hall/CRC Press, Boca Raton, FL

    work page 2010

  3. [3]

    Probabilistic modeling of bifurcations in single-cell gene expression data using a Bayesian mixture of factor analyzers

    Campbell KR, Yau C (2017). Probabilistic modeling of bifurcations in single-cell gene expression data using a Bayesian mixture of factor analyzers. Wellcome open research, 2. doi:10.12688/wellcomeopenres.11087.1. ://dx.doi.org/10.12688/wellcomeopenres.11087.1

    work page doi:10.12688/wellcomeopenres.11087.1 2017

  4. [4]

    basket: Basket Trial Analysis

    Chen N, Hobbs B, Kaizer A, Kane MJ (2019). basket: Basket Trial Analysis. R package version 0.9.2

    work page 2019

  5. [5]

    Basket Trials in Oncology: A Trade-Off Between Complexity and Efficiency

    Cunanan KM, Gonen M, Shen R, Hyman DM, Riely GJ, Begg CB, Iasonos A (2017 a ). Basket Trials in Oncology: A Trade-Off Between Complexity and Efficiency. Journal of Clinical Oncology, 35(3), 271--273. doi:10.1200/JCO.2016.69.9751. PMID: 27893325, https://doi.org/10.1200/JCO.2016.69.9751 , ://doi.org/10.1200/JCO.2016.69.9751

    work page doi:10.1200/jco.2016.69.9751 2017

  6. [6]

    Specifying the True-and False-Positive Rates in Basket Trials

    Cunanan KM, Iasonos A, Shen R, Hyman DM, Riely GJ, G \"o nen M, Begg CB (2017 b ). Specifying the True-and False-Positive Rates in Basket Trials. JCO Precision Oncology, 1, 1--5

    work page 2017

  7. [7]

    Durie BG, Harousseau J, Miguel J, Blade J, Barlogie B, Anderson K, Gertz M, Dimopoulos M, Westin J, Sonneveld P, et al. (2006). International uniform response criteria for multiple myeloma. Leukemia, 20(9), 1467

    work page 2006

  8. [8]

    Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, Dancey J, Arbuck S, Gwyther S, Mooney M, et al. (2009). New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). European journal of cancer, 45(2), 228--247

    work page 2009

Show all 30 references
  1. [9]

    Borrowing Information across Subgroups in Phase II Trials: Is It Useful? Clinical Cancer Research, 19, 1326--1334

    Freidlin B, Korn E (2013). Borrowing Information across Subgroups in Phase II Trials: Is It Useful? Clinical Cancer Research, 19, 1326--1334

    2013

  2. [10]

    Bayesian Data Analysis

    Gelman A, Carlin JB, Stern HS, Rubin DB (2013). Bayesian Data Analysis. 3rd edition. Chapman and Hall/CRC Press, Boca Raton, FL

    2013

  3. [11]

    Statistical challenges posed by uncontrolled master protocols: sensitivity analysis of the vemurafenib study

    Hobbs B, Kane M, Hong D, Landin R (2018). Statistical challenges posed by uncontrolled master protocols: sensitivity analysis of the vemurafenib study. Annals of Oncology, 29(12), 2296--2301

    2018

  4. [12]

    Adaptive adjustment of the randomization ratio using historical control data

    Hobbs BP, Carlin BP, Sargent DJ (2013). Adaptive adjustment of the randomization ratio using historical control data. Clinical Trials, 10, 430--440

    2013

  5. [13]

    Bayesian basket trial design with exchangeability monitoring

    Hobbs BP, Landin R (2018). Bayesian basket trial design with exchangeability monitoring. Statistics in medicine, 37(25), 3557--3572

    2018

  6. [14]

    Hyman DM, Puzanov I, Subbiah V, Faris JE, Chau I, Blay JY, Wolf J, Raje NS, Diamond EL, Hollebecque A, et al. (2015). Vemurafenib in multiple nonmelanoma cancers with BRAF V600 mutations. New England Journal of Medicine, 373(8), 726--736

    2015

  7. [15]

    A multi-source adaptive platform design for testing sequential combinatorial therapeutic strategies

    Kaizer AM, Hobbs BP, Koopmeiners JS (2018). A multi-source adaptive platform design for testing sequential combinatorial therapeutic strategies. Biometrics, 74(3), 1082--1094

    2018

  8. [16]

    Bayesian hierarchical modeling based on multisource exchangeability

    Kaizer AM, Koopmeiners JS, Hobbs BP (2017). Bayesian hierarchical modeling based on multisource exchangeability. Biostatistics, 19(2), 169--184

    2017

  9. [17]

    iSeq: Bayesian Hierarchical Modeling of ChIP-seq Data Through Hidden Ising Models

    Mo Q (2018). iSeq: Bayesian Hierarchical Modeling of ChIP-seq Data Through Hidden Ising Models. R package version 1.34.0

    2018

  10. [18]

    Combining nonexchangeable functional or survival data sources in oncology using generalized mixture commensurate priors

    Murray TA, Hobbs BP, Carlin BP (2015). Combining nonexchangeable functional or survival data sources in oncology using generalized mixture commensurate priors. Annals of Applied Statistics, 9(3), 1549--1570

    2015

  11. [19]

    High-Dimensional Bayesian Clustering with Variable Selection: The R Package bclust

    Nia VP, Davison AC (2012). High-Dimensional Bayesian Clustering with Variable Selection: The R Package bclust . Journal of Statistical Software, 47(5), 1--22. ://www.jstatsoft.org/v47/i05/

    2012

  12. [20]

    R: A Language and Environment for Statistical Computing

    R Core Team (2019). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ://www.R-project.org/

    2019

  13. [21]

    Using informative Multinomial-Dirichlet prior in a t-mixture with reversible jump estimation of nucleosome positions for genome-wide profiling

    Samb R, Khadraoui K, Belleau P, Desch\^enes A, Lakhal-Chaieb L, Droit A (2015). Using informative Multinomial-Dirichlet prior in a t-mixture with reversible jump estimation of nucleosome positions for genome-wide profiling. Statistical Applications in Genetics and Molecular Bi...

    2015 doi

  14. [22]

    BHC: Bayesian Hierarchical Clustering

    Savage R, Cooke E, Darkins R, Xu Y (2018). BHC: Bayesian Hierarchical Clustering. R package version 1.34.0

    2018

  15. [23]

    A Bayesian model for cross-study differential gene expression

    Scharpf RB, Tjelmeland H, Parmigiani G, Nobel A (2009). A Bayesian model for cross-study differential gene expression. JASA. 10.1198/jasa.2009.ap07611

    2009 doi

  16. [24]

    Malaria parasite clearance rate regression: an R software package for a Bayesian hierarchical regression model

    Sharifi-Malvajerdi S, Zhu F, Fogarty CB, Fay MP, Fairhurst RM, Flegg JA, Stepniewska K, Small DS (2019). Malaria parasite clearance rate regression: an R software package for a Bayesian hierarchical regression model. Malaria Journal, 18(1), 4. ISSN 1475-2875. doi:10.1186/s1293...

    2019 doi

  17. [25]

    Coordinate-Based Meta-Analysis of fMRI Studies with R

    Stocco A (2014). Coordinate-Based Meta-Analysis of fMRI Studies with R. R Journal, 6(2)

    2014

  18. [26]

    Hierarchical Bayesian approaches to phase II trials in diseases with multiple subtypes

    Thall P, Wathen J, Bekele B, Champlin R, Baker L, Benjamin R (2003). Hierarchical Bayesian approaches to phase II trials in diseases with multiple subtypes. Statistics in Medicine, 22, 763--780

    2003

  19. [27]

    Generalized Simulated Annealing for Efficient Global Optimization: the GenSA Package for R

    Yang Xiang , Gubian S, Suomela B, Hoeng J (2013). Generalized Simulated Annealing for Efficient Global Optimization: the GenSA Package for R . The R Journal Volume 5/1, June 2013. ://journal.r-project.org/archive/2013/RJ-2013-002/index.html

    2013

  20. [28]

    CRAN Task View: Clinical Trial Design, Monitoring, and Analysis

    Zhang E, Zhang HG (2018). CRAN Task View: Clinical Trial Design, Monitoring, and Analysis . https://CRAN.R-project.org/view=ClinicalTrials . Version 2018-06-18

    2018

  21. [29]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter collaboration doi edition editor eid eprint howpublished institution isbn issn journal key month note number numpages organization pages publisher school series title type url volume year label extra.label sort.label short.list IN...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.