{"id":"561d2a19-2cd3-4626-b454-5a4082f844eb","arxiv_id":"1908.00618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The basket R package implements Bayesian multisource exchangeability models, providing posterior inference, exchangeability probabilities, and clustering for basket trial data.","lead":"This paper presents basket, an R package for analyzing basket trials with Bayesian multisource exchangeability models that borrow information across patient subgroups. It is a software introduction with a worked example from a cancer trial.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The package's likelihood specification is not shown to match the symmetric MEM: Eq. 1–3 are row-wise and may double-count clusters, and no exact-vs-MCMC validation is reported.","rationale":"The reader's verdict is CONDITIONAL and this stress-test agrees with that judgment. The package is real, available on CRAN, and the case study demonstrates a plausible workflow, but the central claim of correct MEM inference depends on two unverified conditions: the likelihood implemented in the package must match the published symmetric MEM model, and the default MCMC settings must produce accurate posterior summaries. The manuscript's equations for the MEM likelihood are ambiguous or incorrect as written: Eqs. (1)–(3) are row-wise and the stated product-of-rows construction violates the symmetry of Ω and double-counts shared clusters. Because the package is available, the decisive check is inexpensive: compare mem_exact() and mem_mcmc() against an independent exact implementation for a small number of baskets. That single comparison would settle both the model-matching concern and the MCMC-convergence concern. Until that check is run, CONDITIONAL is the appropriate verdict; if the row-wise likelihood is in fact what the package computes, the verdict would need to move toward REJECT. The reader's weakest assumption identified MCMC convergence and Equation 1 matching; this stress-test sharpens the Equation 1 issue into a concrete model-specification risk, hence partial agreement.","tokens_in":12170,"tokens_out":7504,"duration_ms":80538,"concrete_test":"Compute the exact symmetric MEM posterior for the J=6 vemurafenib data by enumerating all symmetric binary exchangeability matrices (or all partitions, if exchangeability is intended as an equivalence relation), using the package's default Beta(0.5, 0.5) prior and uniform prior over configurations, with the clique-product integrated likelihood. Compare the resulting PEP matrix, MAP configuration, and posterior means with mem_exact() and mem_mcmc() output. Additionally, run mem_exact() on a small simulated example (J=3 or 4) and compare with an independent implementation of the clique-product likelihood. If the package output differs from the clique-product posterior, the package does not implement the symmetric MEM as defined; if mem_mcmc differs from mem_exact beyond Monte Carlo error, the default sampler settings are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that basket's posterior computations correspond to the symmetric MEM posterior of Hobbs and Landin (2018), and that mem_mcmc approximates it accurately. The manuscript provides no simulation or exact-vs-MCMC comparison to support this. More specifically, Eqs. (1)–(3) as written are not a correct specification of the symmetric MEM posterior. For a full symmetric exchangeability configuration Ω, the integrated likelihood is a product of beta-binomial factors over exchangeability clusters: ∏_c B(a + Σ_{i∈c} S_i, b + Σ_{i∈c}(n_i − S_i))/B(a,b). Eq. (1) instead conditions on one row Ω_j and treats every basket not exchangeable with j as a singleton, even when those other baskets are exchangeable with one another. Eq. (3) is therefore a row-wise posterior, not the marginal posterior of Ω_j under the global symmetric model. The paper then states that P(Ω|S) is the product of row-wise calculations; for J=3 with all baskets exchangeable, that product would multiply the single-cluster likelihood three times. If mem_exact or mem_mcmc implement this row-wise product, the reported PEPs, MAP configuration, and posterior means are not MEM posteriors. No evidence in the manuscript rules this out, so the central claim of valid MEM inference is not secured.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents the R package basket for Bayesian analysis of basket trials under the multisource exchangeability model (MEM) of Hobbs and Landin (2018). It describes exact and MCMC implementations, accessor functions for posterior exchangeability probabilities, MAP configurations, effective sample sizes, posterior probabilities of response, and visualization tools, and demonstrates the package on the vemurafenib basket trial. The central claim is that basket is the first available software package implementing Bayesian analysis with the MEM and that it provides valid posterior inference for response rates, exchangeability probabilities, effective sample sizes, and cluster assignments in basket trials.","tokens_in":12411,"tokens_out":3966,"duration_ms":39405,"significance":"If the implementation is correct, the package would fill a genuine gap: no other software implements symmetric MEM posterior inference, and the case study with reproducible code is useful to practitioners. The paper provides the data, commands, and output for a real oncology trial, and the availability of both exact enumeration for small J and MCMC for larger J is appealing. The contribution is potentially significant for clinical trial methodology. However, the manuscript does not currently establish that the implemented model is the symmetric MEM, and the lack of validation against exact enumeration or simulation leaves the central statistical claim unsecured. The present version requires a corrected model specification and empirical validation before the claims can be accepted.","major_comments":[{"comment":"Equations (1)–(3) specify a row-wise posterior, not the symmetric MEM posterior. For a symmetric exchangeability configuration Ω, the integrated likelihood is a product of beta-binomial factors over exchangeability clusters, ∏_c B(a + Σ_{i∈c} S_i, b + Σ_{i∈c}(n_i − S_i))/B(a,b). Equation (1) instead conditions on a single row Ω_j and treats every basket not exchangeable with j as a singleton, even when those baskets are exchangeable with one another; Equation (3) is therefore the posterior of Ω_j under that row-wise model. The text then states that P(Ω|S) is the product of row-wise calculations, which for J=3 with all baskets exchangeable multiplies the single-cluster likelihood three times. If mem_exact or mem_mcmc implement this product, the reported PEPs, MAP configuration, and posterior means are not MEM posteriors. Please correct the model equations to the global symmetric formulation, or clearly state and justify a row-wise approximation and confirm which one the package implements.","section":"§2.2, Eqs. (1)–(3)"},{"comment":"No validation of the MCMC sampler is provided. The paper asserts that mem_mcmc approximates P(Ω_j = ω_g | S) and that the default 200,000 iterations with 50,000 burn-in are a practical lower bound, but it reports no convergence diagnostics, no comparison to mem_exact for configurations where exact enumeration is feasible, and no simulation study in which true response rates and exchangeability structures are known. Because the package's validity claim rests on this sampler, please add an exact-versus-MCMC comparison for J ≤ 6 and a small simulation study, and report trace plots or Gelman–Rubin statistics for the case study.","section":"§3.1"},{"comment":"The definition of the PEP via P(Ω|S) as a product of row-wise probabilities is not a probability distribution over symmetric MEMs, since the factors are not normalized joint probabilities and different rows share the same cluster memberships. This makes the PEP matrix and MAP configuration ill-defined unless a joint model is specified. Please provide the joint posterior over exchangeability partitions (e.g., over set partitions of {1,...,J}) and define PEPs as sums of that joint posterior.","section":"§2.2 (PEP and MAP definitions)"}],"minor_comments":[{"comment":"The word \"singelton\" appears where \"singleton\" is intended; please correct the typo.","section":"§2.2"},{"comment":"The method name \"print.exchageability_model()\" contains a typo; it should be \"print.exchangeability_model()\".","section":"§3.2"},{"comment":"The reference list gives the basket package version as 0.9.2 while the text states version 1.0.0; please make these consistent.","section":"References"},{"comment":"The complexity statement says the number of configurations grows as O(2^{J^2}), but the number of symmetric binary matrices with unit diagonal is 2^{J(J−1)/2}; please correct the exponent.","section":"§3.1"},{"comment":"The text says the case study comprises \"six organs\" but then lists five organ sites (with colorectal cancer split into two baskets); please clarify whether six baskets from five organ sites are intended.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe short version: this is a software paper for an R package that implements MEM for basket trials. The package is real, on CRAN, and the case study is clean. But the paper's central claim—that it computes valid MEM posterior inference—is not supported. Equations 1–3 as written are not the symmetric MEM, and there is no simulation or exact-versus-MCMC check that would catch the problem.\n\nWhat's new: the basket package is the first accessible implementation of MEM for basket trials, and the MCMC sampler claims to handle 20+ baskets, which the exact enumeration can't. That's a practical step forward. The writing is clear, the functions are well documented, and the vemurafenib case study is a useful demonstration.\n\nThe problem is the model specification. Equation (1) defines a row-wise marginal density for basket j given row j of Ω. It pools j with baskets exchangeable to j, but treats every other basket as a singleton, as if they had no exchangeability relations among themselves. Equation (3) then defines P(Ω_j = ω_g | S) per row, and the paper says the joint P(Ω|S) is the product of these row-wise calculations. For J=3 with all baskets exchangeable, that product multiplies the same single-cluster likelihood three times. That is not the symmetric MEM posterior from Hobbs and Landin. If mem_exact or mem_mcmc implement this product, the reported PEPs, MAP configuration, and posterior means are not MEM quantities. The manuscript gives no reason to think the implementation deviates from the equations.\n\nThere are smaller issues too. The paper claims K = ∏_{j=1}^{J-1} 2^j configurations, which counts all undirected graphs, not the set partitions that MEM actually uses. It also asserts the default 200,000 MCMC iterations is a \"practical lower bound\" without showing convergence diagnostics or a justification. The ESS calculation is described as a modification of Hobbs and Landin, but no simulation shows it recovers sensible values.\n\nWho gets value from this paper? Someone who wants to try a MEM-style analysis quickly and is willing to treat the results as heuristic until the model issue is resolved. It deserves peer review—a serious referee should catch this—but as it stands the central claim is not established. If the implementation is actually correct, a revised paper with corrected equations and a small exact-vs-MCMC simulation would make it a solid contribution.\n\nRecommendation: send to peer review, but expect major revision.","headline":"Useful package, but the paper's equations are not the symmetric MEM and no validation supports the implementation.","tokens_in":12917,"tokens_out":5476,"would_cite":false,"duration_ms":50013,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62P10","62-04"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces basket, the first R package for Bayesian multisource exchangeability analysis of basket trials, and claims it yields valid posterior inference for response rates, exchangeability, effective sample sizes, and cluster…","keywords":["Bayesian analysis","basket design","hierarchical model","master protocol","oncology","patient heterogeneity","multisource exchangeability model","R package"],"falsifier":"Simulate basket trials with known response rates for J = 3,...,12 under configurations where some baskets are exchangeable and others are not, then run mem_mcmc() with default settings and compare the resulting posterior exchangeability probabilities, response-rate means, and 95% HPD coverage to mem_exact() (for small J) or to long-run exact calculations; if the MCMC results deviate by more than Monte Carlo error, or if HPD coverage falls systematically below 95%, the paper's claim of valid posterior inference with default settings fails.","tokens_in":11987,"feed_emoji":"🧺","tokens_out":9960,"duration_ms":87932,"temperature":0.7,"pith_summary":"Basket trials enroll several subpopulations that share a molecular target, but the treatment may work in some groups and fail in others, so pooling everyone is biased while analyzing each group alone is underpowered. This paper presents basket, an R package that fits the Bayesian multisource exchangeability model (MEM), which gives every pair of subpopulations a probability of being exchangeable and borrows strength only among exchangeable pairs. The package reports posterior response rates, a pairwise posterior exchangeability probability matrix, effective sample sizes, and cluster assignments, and it can designate “poolable” meta-baskets. The authors demonstrate the workflow on the vemurafenib BRAF V600 trial, where the analysis separates low-responding colorectal and bile-duct baskets from high-responding lung, histiocytosis, and thyroid baskets.","feed_headline":"R package 'basket' is first to pool only exchangeable trial subgroups","feed_subtitle":"This Bayesian tool reports exchangeability probabilities and effective sample sizes for each subgroup and cluster.","key_machinery":"The central machinery is the exchangeability configuration matrix Ω, a symmetric J×J binary matrix whose diagonal entries are fixed at 1 and whose off-diagonal entry Ωij indicates whether baskets i and j are statistically exchangeable. The model evaluates all configurations of Ω; Equation 1 supplies the marginal density of a basket's response count conditional on a configuration, and Equation 3 converts those marginal densities into posterior configuration probabilities. The PEP matrix is obtained by summing posterior weights of configurations with Ωij=1, and the maximum a posteriori MEM is the single most probable configuration. Effective sample sizes are computed by matching each posterior HPD interval to the nearest beta distribution through a simulated-annealing search, yielding conservative estimates of borrowing. This machinery replaces the single shrinkage parameter of conventional hierarchical models with source-specific exchangeability relationships.","core_discovery":"The paper's central claim is that basket is the first available software package implementing Bayesian analysis with the multisource exchangeability model (MEM), and that it provides valid posterior inference for basket-specific response rates, pairwise exchangeability probabilities, effective sample sizes, and cluster assignments. Whereas a single-source hierarchical model uses one shrinkage parameter for all subpopulations, the MEM enumerates all possible pairwise exchangeability configurations and weights each configuration by its marginal likelihood. The package computes the posterior exchangeability probability (PEP) matrix by summing posterior weights over configurations in which a given pair is exchangeable, and it identifies the maximum a posteriori configuration as the most probable partition. It implements both exact enumeration, recommended for fewer than seven baskets, and a Metropolis MCMC sampler that extends the analysis to more than twenty baskets. In the vemurafenib case study, the analysis yields two clusters, with posterior probabilities near zero that the low-responding colorectal baskets are exchangeable with the higher-responding lung, histiocytosis, and thyroid baskets.","pith_inferences":["Beyond the paper, the PEP matrix could become a routine reporting element in early-phase oncology trials, giving readers a direct numerical measure of heterogeneity rather than a single pooled estimate.","The same pairwise exchangeability machinery is not tied to tumor histology and could be applied to any multi-cohort binary-endpoint trial, such as rare-disease platform studies or trials testing one agent across several molecular subtypes.","A calibrating study that runs mem_exact() and mem_mcmc() on simulated trials across a grid of J and response-rate configurations would give users principled guidance for setting burn-in and iteration counts, a step the paper leaves implicit."],"forward_implications":["Analysts can report a posterior exchangeability probability for every pair of baskets, making the amount of borrowed information explicit rather than hidden in a global shrinkage parameter.","The MAP configuration and cluster assignments designate meta-baskets, so trial reports can state which subpopulations form a common pool and which are treated separately.","Effective sample sizes quantify how much each basket's posterior precision owes to other baskets, which is directly relevant for interpreting sparsely enrolled subpopulations.","The MCMC implementation extends MEM analyses beyond 20 baskets, so the method is not confined to small trials with a handful of subpopulations."],"supporting_citations":[{"why":"Supplies the symmetric MEM formulation, the marginal density in Equation 1, and the default reference prior for exchangeability.","marker":"Hobbs and Landin (2018)"},{"why":"Introduces the MEM framework and its asymptotic consistency, the theoretical basis the package implements.","marker":"Kaizer, Koopmeiners, and Hobbs (2017)"},{"why":"Extends MEM to a sequential combinatorial platform design and demonstrates efficiency gains from information sharing, supporting the framework's broader utility.","marker":"Kaizer, Hobbs, and Koopmeiners (2018)"},{"why":"Provides the vemurafenib basket trial data used as the case study for the package demonstration.","marker":"Hyman et al. (2015)"},{"why":"Presents a permutation-based sensitivity analysis of the vemurafenib study that motivates the case-study evaluation of heterogeneity.","marker":"Hobbs, Kane, Hong, and Landin (2018)"},{"why":"Implements the simulated-annealing algorithm used to match posterior HPD intervals to beta distributions for effective sample size estimation.","marker":"Yang Xiang, Gubian, Suomela, and Hoeng (2013)"}],"fun_headline_variants":["First R package to pool only exchangeable basket-trial subgroups","Bayesian tool pools only exchangeable subgroups in basket trials","MEM-based R package shrinks only exchangeable basket subgroups","Multisource exchangeability modeling for basket trials, now in R"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the MCMC sampler with the default 50,000-iteration burn-in and 200,000 iterations converges reliably to the MEM posterior for every basket configuration, and that the code implementing Equation 1 exactly matches the model—an assumption the paper does not support with simulation studies or convergence diagnostics.","fun_headline_variants_meta":{"raw":{"variants":["First R package to pool only exchangeable basket-trial subgroups","Bayesian tool pools only exchangeable subgroups in basket trials","MEM-based R package shrinks only exchangeable basket subgroups","Multisource exchangeability modeling for basket trials, now in R"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001347,"raw_usage":{"total_tokens":5487,"prompt_tokens":974,"completion_tokens":4513,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":4443}},"tokens_in":590,"tokens_out":4513,"duration_ms":26235,"temperature":1.0,"reasoning_tokens":4443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:42:24.583026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate basket trials with known response rates for J = 3,...,12 under configurations where some baskets are exchangeable and others are not, then run mem_mcmc() with default settings and compare the resulting posterior exchangeability probabilities, response-rate means, and 95% HPD coverage to mem_exact() (for small J) or to long-run exact calculations; if the MCMC results deviate by more than Monte Carlo error, or if HPD coverage falls systematically below 95%, the paper's claim of valid posterior inference with default settings fails.","supporting_citations":[{"cited_title":"Bayesian hierarchical modeling based on multisource exchangeability","cited_arxiv_id":null,"evidence_quote":"Introduces the MEM framework and its asymptotic consistency, the theoretical basis the package implements."},{"cited_title":"A multi-source adaptive platform design for testing sequential combinatorial therapeutic strategies","cited_arxiv_id":null,"evidence_quote":"Extends MEM to a sequential combinatorial platform design and demonstrates efficiency gains from information sharing, supporting the framework's broader utility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the vemurafenib basket trial data used as the case study for the package demonstration."},{"cited_title":"Generalized Simulated Annealing for Efficient Global Optimization: the GenSA Package for R","cited_arxiv_id":null,"evidence_quote":"Implements the simulated-annealing algorithm used to match posterior HPD intervals to beta distributions for effective sample size estimation."}],"review_version":1}