{"id":"c41d530a-4fc3-4897-a955-c92ce8144e87","arxiv_id":"2411.15566","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Shapley-Owen interaction indices in bioprocess Bayesian network models can be estimated efficiently with a non-nested estimator and sequential budget allocation, with lower mean squared error than equal-allocation nested simulation.","lead":"This paper develops efficient Monte Carlo algorithms for measuring how pairs of random factors jointly affect output variance in policy-augmented Bayesian network models of biomanufacturing. It combines Shapley-Owen interaction indices with a non-nested estimator and a sequential budget allocation scheme to reduce estimation error under a fixed computational budget.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's group allocation biases the permutation distribution for each pair, making its Shapley-Owen estimator inconsistent.","rationale":"Agreement with reader: partial. The reader identified the allocation rule as an unproven heuristic and caught the plus-sign error in the sample standard deviation, but did not identify the deeper, correctness-level defect: Stage 2's reuse-driven group selection conditions the permutation distribution per pair. That conditioning follows directly from the algorithm's text: i* is chosen as the leftmost element of the argmax pair in π(n), and only pairs with first element i* are updated. For a fixed pair, the sampled permutations are exactly those satisfying a relative-order event for the current argmax pair, which is not independent of the pair's own contribution. Since the Shapley-Owen index is an average over uniform permutations, any such conditioning biases the estimator. The concrete toy example demonstrates that the bias need not vanish asymptotically, and it can be substantial. Because the paper's headline experimental claim is that Algorithm 2 estimates interaction effects more accurately, an inconsistent estimator directly invalidates the central claim. Algorithm 1 remains unbiased, but the main methodological contribution and the numerical superiority claim are tied to Algorithm 2. The variance-formula error and the lack of optimality proof are real but would only affect efficiency/allocation; the selection bias affects consistency. Therefore the appropriate verdict is REJECT for the current manuscript, rather than CONDITIONAL.","tokens_in":12956,"tokens_out":27537,"duration_ms":246012,"concrete_test":"For the 3-input set function above, run Algorithm 2 Stage 2 with N0=0 and hard-code gHL so {1,2} is always the argmax. After 10^6 iterations, compare the estimate for pair {1,3} with the analytic value -10/3; confirmation of bias is convergence to -5 rather than -10/3. A more direct check: among the iterations used for pair {1,3}, compute the empirical frequency of each permutation and compare to uniform over Π(I); any discrepancy confirms the non-uniform sampling that causes the bias.","verdict_should_be":"REJECT","load_bearing_attack":"Algorithm 2, Stage 2 (Section 4), updates a pair {i,j} only when i*, the leftmost element of the current argmax pair in the freshly sampled permutation π(n), equals i or j. The inclusion event therefore constrains the relative order of the argmax pair's two indices in π(n). Conditional on inclusion, π(n) is not uniform over Π(I), so the samples entering \\hatSh_{i,j} are drawn from a biased permutation distribution. The permutation formula for the Shapley-Owen index (Eq. 1) is an expectation over uniform permutations; averaging over a conditional distribution does not converge to Sh_{i,j}. This is not an efficiency issue: the estimator is asymptotically biased. Example with I={1,2,3} and g(∅)=0, g({1})=10, g({3})=10, g({1,3})=15, g({1,2})=50, g({2,3})=50, g({1,2,3})=100, g({2})=0 gives Sh_{1,3}=-10/3. If pair {1,2} is the permanent argmax, pair {1,3} is sampled only when 1 precedes 2; the conditional mean of Δ_{1,3} is -5, not Sh_{1,3}. The sign error in the \\hatσ formula and lack of optimality proof are secondary; this selection bias breaks the central accuracy claim of Algorithm 2.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes simulation-based estimation of Shapley-Owen interaction indices for policy-augmented Bayesian network (PABN) models used in biomanufacturing. It presents a nested Monte Carlo algorithm (Algorithm 1) that samples model parameters and permutations, and a non-nested algorithm (Algorithm 2) that combines the non-nested estimator of Cheng and Zhang (2021) with a sequential budget allocation rule driven by confidence-interval half-length gradients. Numerical experiments on a linear Gaussian PABN (with analytic ground truth) and a nonlinear cell-progenitor feedback model are used to support the claim that Algorithm 2 is more accurate than Algorithm 1 under a fixed computation budget.","tokens_in":13244,"tokens_out":14547,"duration_ms":124029,"significance":"If the methods were sound, the paper would offer a practical way to quantify interaction effects in a class of stochastic simulation models that are relevant to bioprocess risk assessment. The analytic linear-Gaussian test in Section 5.1 is a useful validation design, and the reuse of intermediate value-function calculations in Algorithm 1 is a sensible efficiency idea. However, the central contribution, Algorithm 2, suffers from a selection bias that makes the estimator inconsistent; this is a load-bearing flaw that invalidates the paper's main accuracy claim. The non-nested estimator itself is grounded in prior literature, but its unbiasedness does not carry over to the sequentially allocated samples.","major_comments":[{"comment":"The sequential allocation scheme conditions the set of pairs updated on the relative order of the elements of the argmax pair. The instruction \"Find {i*,j*} = argmax_{i,j} gHL_{i,j} and i* that is the leftmost element of {i*,j*} in pi(n); Construct the group G_m(i*, pi(n))\" means that a pair {i,j} is updated only when the element of that pair that appears earlier in the permutation is the leftmost element of the currently selected argmax pair. Consequently, the permutation distribution used for each pair is not uniform over Pi(I), but is conditioned on a relative-order event. The Shapley-Owen index in Eq. (1) is an expectation over uniform permutations, so averaging the marginal contributions over this conditional distribution does not converge to Sh_{i,j} in general. For a concrete counterexample, let I={1,2,3} with g(empty)=0, g({1})=10, g({3})=10, g({1,3})=15, g({1,2})=50, g({2,3})=50, g({1,2,3})=100, g({2})=0. Then Sh_{1,3}=-5/2. If pair {1,2} is the argmax at every iteration and m=2, pair {1,3} is updated only in permutations where 1 precedes both 2 and 3; among those permutations, Delta_{1,3}(pi)=-5 for every such pi, so the conditional mean is -5, not -5/2. The estimator is therefore asymptotically biased, and the bias does not vanish as the budget grows.","section":"Section 4, Algorithm 2, Stage 2"},{"comment":"The sample standard deviation formula in Stage 1 is incorrect. The algorithm sets dSh2_{i,j} = dSh2_{i,j} + (Delta)^2 and Sh_{i,j} = Sh_{i,j} + Delta, then computes sigma_hat_{i,j} = sqrt( (dSh2_{i,j} + (Sh_{i,j})^2/N0) / (N0-1) ). Since dSh2 is the sum of squares and Sh is the sum, the correct sample variance is (sum Delta^2 - (sum Delta)^2/N0)/(N0-1), which requires a minus sign, not a plus sign. The printed formula overestimates the variance and distorts the gHL scores that drive the sequential allocation, as well as the confidence interval in Proposition 1.","section":"Section 4, Algorithm 2, Stage 1"},{"comment":"The variance estimates sigma_hat computed in Stage 1 are never updated in Stage 2. The pseudo-code updates N_{i,j} and Sh_{i,j} for the selected group, but not dSh2 or sigma_hat. Thus the allocation scores gHL_{i,j} remain based on the initial pilot variance estimates, which become increasingly stale as the sample sizes grow. This contradicts the stated goal of dynamically allocating budget to interactions with high uncertainty and means the algorithm does not actually track the evolution of estimation uncertainty.","section":"Section 4, Algorithm 2, Stage 2"},{"comment":"The greedy allocation rule is asserted to improve accuracy, but no optimality or finite-sample guarantee is provided for the choice of gHL_{i,j} = z_{alpha/2} sigma_hat_{i,j}/(2 N_{i,j}^{3/2}) as the metric for allocating one additional sample. The derivative of the confidence-interval half-length with respect to N treats sigma_hat as constant and ignores the fact that samples are shared among pairs in the group G_m. The numerical claim in Section 5.1 that \"the performance of Algorithm 2 is superior over Algorithm 1\" is therefore not supported by theory, and the MSE comparison cannot separate the bias identified in the first comment from variance reduction.","section":"Section 4, leading to Section 5.1"}],"minor_comments":[{"comment":"The summation set is written as \"U in I / {i,j})\"; it should be \"U subset of I \\ {i,j}\".","section":"Equation (1)"},{"comment":"The loop header \"for all set {i,j} subseteq I\" should be \"for all {i,j} subseteq I\".","section":"Algorithm 1"},{"comment":"The caption says \"with 90% confidence interval\" but the numbers in parentheses are not labeled as a half-width or standard error; the reporting format is ambiguous.","section":"Table 2"},{"comment":"The ground-truth Shapley-Owen values are said to be computed from Eq. (2) with \"a large number of model parameter samples,\" but the actual number of samples used for the ground truth is not reported.","section":"Section 5.1"},{"comment":"The nonlinear PABN experiment reports only point estimates of the Shapley-Owen indices, without standard errors or replications, so qualitative statements such as \"the interaction effects diminish as time proceeds\" are not statistically supported.","section":"Section 5.2"},{"comment":"Some reference entries contain formatting errors, such as double commas in the author lists of Aijaz et al. (2018) and Liu et al. (2015).","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central Algorithm 2 is not a consistent estimator of the Shapley-Owen index because the sequential allocation conditions on the relative order of the argmax pair in the sampled permutation. This is a fundamental design flaw, not a local typographical issue. The non-nested estimator and the analytic linear-Gaussian validation are sound components, but they do not rescue the main contribution. The authors would need to redesign the allocation mechanism—for example, by sampling permutations uniformly and only then choosing which pairs to update—before the paper could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: this paper's central claim about Algorithm 2 does not hold up. The sequential allocation scheme in Stage 2 conditions on the sampled permutation in a way that biases the Shapley-Owen estimator, so the estimator is not consistent. A simple three-input example shows the bias: with g(∅)=0, g({1})=10, g({3})=10, g({1,3})=15, g({1,2})=50, g({2,3})=50, g({1,2,3})=100, g({2})=0, the true Sh_{1,3} is -2.5, but if {1,2} is the permanent argmax, Algorithm 2 samples pair {1,3} only for permutations where 1 precedes both 2 and 3, and the conditional mean of the increment is -5. That is not a small-sample artifact; it persists as N grows.\n\nWhat is actually new: the application of Shapley-Owen interaction indices to policy-augmented Bayesian networks, and the attempt to combine a non-nested estimator with adaptive budget allocation. The linear Gaussian test provides analytic ground truth, which is a good check and gives the paper some empirical credibility. The reuse of common terms across pairs is also sensible.\n\nThe soft spots are not just the sign error in the sample variance formula (plus instead of minus in Algorithm 2) and the unproven greedy allocation heuristic; the selection bias is the load-bearing problem. The non-nested estimator itself is unbiased when samples are i.i.d., as Cheng and Zhang showed, but Algorithm 2 does not feed it i.i.d. samples from the target permutation distribution. Proposition 1 is stated for a fixed sample size and does not apply to the adaptive scheme. The numerical experiments are too narrow to expose the bias: one linear model with six factors and one nonlinear case without ground truth. The MSE comparisons may still favor Algorithm 2 because variance reduction can outweigh bias in small samples, but the estimator is not consistent.\n\nWho gets value: readers working on Shapley-Owen estimation might learn from the setup and the pitfalls, but the paper does not deliver a reliable method as written. It deserves a serious referee to catch this subtle issue rather than a desk reject, because fixing the allocation (e.g., reweighting selected samples or sampling without conditioning on the permutation) might salvage the idea. As is, it needs major revision.\n\nRecommendation: send to peer review, expect heavy revision; the bias and the variance formula error must be addressed. I would not cite it in its current form.","headline":"Algorithm 2's adaptive allocation biases the permutation distribution, making the main estimator inconsistent; the paper's central claim does not survive a close look.","tokens_in":13773,"tokens_out":12221,"would_cite":false,"duration_ms":101186,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that Shapley-Owen interaction effects in policy-augmented Bayesian networks can be estimated accurately under a fixed simulation budget by replacing nested simulation with a non-nested estimator and spending samples…","keywords":["Shapley-Owen index","interaction effects","global sensitivity analysis","policy-augmented Bayesian network","nested Monte Carlo","sequential budget allocation","quasi-Monte Carlo","biomanufacturing"],"falsifier":"Re-run the linear Gaussian experiment of Section 5.1 with Algorithm 2's sample standard deviation replaced by the correct unbiased form $dSh^2_{i,j}-(\\hat{Sh}_{i,j})^2/N_0$ divided by $N_0-1$, and compare the mean squared error curves with Algorithm 1; if the corrected allocation no longer beats equal allocation except at very small budgets, the reported advantage is not robust. A second check would compare the greedy allocation against an oracle that allocates the same total budget in proportion to the true per-pair variances.","tokens_in":12740,"feed_emoji":"🧫","tokens_out":14772,"duration_ms":108698,"temperature":0.7,"pith_summary":"This paper addresses the cost of measuring interaction effects among random factors in policy-augmented Bayesian networks (PABN), a hybrid model used to describe biomanufacturing processes. The authors adopt the Shapley-Owen index, whose exact evaluation is exponentially expensive in the number of inputs, and propose replacing exact evaluation with sampling: a nested Monte Carlo estimator in Algorithm 1 and a non-nested estimator in Algorithm 2. Algorithm 2 also allocates a fixed simulation budget sequentially, giving each next replication to the interaction pair whose confidence interval is currently widest, and reusing shared value-function terms. In a linear Gaussian PABN where the true indices can be computed exactly, they report that Algorithm 2 has lower mean squared error than Algorithm 1 unless the budget is very small, and that quasi-Monte Carlo sampling further lowers error. If this holds, interaction quantification for PABN becomes practical under a fixed computation budget without manually choosing nested sample sizes, and the sign and magnitude of the indices can guide process understanding and control.","feed_headline":"Sequential sampling estimates interaction effects in bioprocess models","feed_subtitle":"Non-nested sampling plus sequential budget allocation measures factor interactions under a fixed simulation budget.","key_machinery":"Three pieces carry the argument. First, the Shapley-Owen index (1), a weighted average over subsets of the second-order finite difference $g(U\\cup\\{i,j\\})-g(U\\cup\\{i\\})-g(U\\cup\\{j\\})+g(U)$, measures how much the variance explained by the pair exceeds the sum of the individual effects; its sign indicates synergy or antagonism. Second, the non-nested value-function estimator $\\hat{g}_{nn}(U|\\mathbf{w})=y^{(1)}(y^{(2)}-y^{(3)})$ rests on the identity $\\mathrm{Var}[E(Y|X)]=E[Y^{(1)}(Y^{(2)}-Y^{(3)})]$, so the value function is estimated without the inner simulation layer of the nested estimator. Third, the sequential allocation rule computes the gradient of the confidence-interval half-width, $gHL_{i,j}=z_{\\alpha/2}\\hat{\\sigma}_{i,j}/(2N_{i,j}^{3/2})$, treats this as the benefit of one additional sample for pair $\\{i,j\\}$, and allocates the next replication to the largest such gradient among the group of pairs sharing a reusable quantity; recorded value-function evaluations are reused to cut computation.","core_discovery":"The paper's central claim is that for a PABN the Shapley-Owen interaction index $Sh_{i,j}$ can be estimated accurately and efficiently by sampling random model parameters and random permutations, evaluating the alternating value-function sum $\\hat{g}(P_{i,j}(\\pi)\\cup\\{i,j\\})-\\hat{g}(P_{i,j}(\\pi)\\cup\\{i\\})-\\hat{g}(P_{i,j}(\\pi)\\cup\\{j\\})+\\hat{g}(P_{i,j}(\\pi))$, and using the identity $\\mathrm{Var}[E(Y|X)]=E[Y^{(1)}(Y^{(2)}-Y^{(3)})]$ to avoid an inner simulation loop. The paper states that the resulting non-nested estimator enables error analysis without choosing the multilevel sample sizes, and that the sequential budget allocation of Algorithm 2, which at each step sends a replication to the pair with the largest $gHL_{i,j}=z_{\\alpha/2}\\hat{\\sigma}_{i,j}/(2N_{i,j}^{3/2})$, estimates the indices more accurately than the equal-allocation nested Algorithm 1 under the same budget, except when the budget is very limited. Numerical evidence comes from a six-factor linear Gaussian PABN with analytic value function, where true Shapley-Owen indices are available, and from a nonlinear mature cell-progenitor negative-feedback PABN, where the indices track known interaction structure and correlation dependence.","pith_inferences":["Beyond the paper, the greedy allocation rule is a heuristic with no optimality proof; a direct test against an oracle allocation that knows the true per-pair variances would show whether the sequential rule is actually near-optimal.","The sign error in the printed standard-deviation update could change which pair receives the next sample, so the reported MSE advantage of Algorithm 2 should be re-verified under the corrected estimator.","The same non-nested identity and sequential allocation idea could transfer to main-effect Shapley values or higher-order interactions in other expensive simulation models, though the shared-calculation reuse pattern would need to be reconstructed.","Because the real PABN experiments have no ground-truth interaction indices, a practical rollout would need calibrated confidence intervals alongside the point estimates; the paper constructs intervals but does not formally validate their coverage."],"forward_implications":["Practitioners can quantify pair interactions among the random factors of a PABN under a fixed computational budget without predetermining four nested sample sizes, because the non-nested estimator removes the inner simulation loop.","The sign and magnitude of estimated Shapley-Owen indices can classify interactions as synergistic or antagonistic, informing whether a pair deserves a full design-of-experiments study, a univariate study, or none.","Quasi-Monte Carlo sampling of parameters and permutations reduces estimation error for the same algorithm, so low-discrepancy sequences provide a cheap additional variance reduction.","Because the method samples permutations instead of enumerating subsets, the cost no longer grows exponentially in the number of inputs, making interaction measurement feasible for larger factor sets.","The nonlinear cell-progenitor experiment indicates that early-stage interactions can amplify and accumulate through the process, so estimated indices can suggest when intervention or control is most valuable."],"supporting_citations":[{"why":"Defines the Shapley-Owen index used throughout as the interaction measure and supplies formula (1).","marker":"Rabitti and Borgonovo (2019)"},{"why":"Provides the non-nested estimator identity Var[E(Y|X)] = E[Y^(1)(Y^(2)-Y^(3))] that Algorithm 2 uses to eliminate the inner simulation loop.","marker":"Cheng and Zhang (2021)"},{"why":"Gives the efficient Monte Carlo approach to Shapley effects that Algorithm 1 extends to interaction estimation.","marker":"Song et al. (2016)"},{"why":"Introduces the unbiased nested Monte Carlo estimator for the variance of a conditional expectation used to estimate the value function in Algorithm 1.","marker":"Sun et al. (2011)"},{"why":"Defines the policy-augmented Bayesian network model, the target model whose random factors and posterior parameter distribution are used throughout.","marker":"Zheng et al. (2023)"},{"why":"Supplies the nonlinear mature cell-progenitor negative-feedback dynamics used in the real-world PABN experiment.","marker":"Stacey et al. (2018)"},{"why":"Provides the transformation method for sampling permutations used in the quasi-Monte Carlo version of Algorithm 2.","marker":"Mitchell et al. (2022)"},{"why":"Supplies the low-discrepancy point-set construction and mapping used to implement quasi-Monte Carlo sampling.","marker":"Lemieux (2009)"}],"fun_headline_variants":["Adaptive sampling sharpens interaction effect estimates in bioprocess models","Sequential sampling boosts interaction effect accuracy under fixed budgets","Smarter sampling allocates budget to high-uncertainty factor pairs","Non-nested simulation plus adaptive sampling reveals interaction effects","Budget-conscious sampling yields precise interaction indices"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one extra simulation should always go to the interaction pair with the largest value of a simple uncertainty-gradient formula, and that this greedy choice improves overall accuracy; the paper asserts the rule without a proof, and the formula as printed in Algorithm 2 uses a plus sign where an unbiased variance estimate needs a minus sign.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive sampling sharpens interaction effect estimates in bioprocess models","Sequential sampling boosts interaction effect accuracy under fixed budgets","Smarter sampling allocates budget to high-uncertainty factor pairs","Non-nested simulation plus adaptive sampling reveals interaction effects","Budget-conscious sampling yields precise interaction indices"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000886,"raw_usage":{"total_tokens":3839,"prompt_tokens":975,"completion_tokens":2864,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2784}},"tokens_in":591,"tokens_out":2864,"duration_ms":20484,"temperature":1.0,"reasoning_tokens":2784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:09:28.868174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the linear Gaussian experiment of Section 5.1 with Algorithm 2's sample standard deviation replaced by the correct unbiased form $dSh^2_{i,j}-(\\hat{Sh}_{i,j})^2/N_0$ divided by $N_0-1$, and compare the mean squared error curves with Algorithm 1; if the corrected allocation no longer beats equal allocation except at very small budgets, the reported advantage is not robust. A second check would compare the greedy allocation against an oracle that allocates the same total budget in proportion to the true per-pair variances.","supporting_citations":[{"cited_title":"A Shapley–Owen index for interaction quantification","cited_arxiv_id":null,"evidence_quote":"Defines the Shapley-Owen index used throughout as the interaction measure and supplies formula (1)."},{"cited_title":"Non-nested estimators for the central moments of a conditional expectation and their convergence properties","cited_arxiv_id":null,"evidence_quote":"Provides the non-nested estimator identity Var[E(Y|X)] = E[Y^(1)(Y^(2)-Y^(3))] that Algorithm 2 uses to eliminate the inner simulation loop."},{"cited_title":"Shapley effects for global sensitivity analysis: Theory and computation","cited_arxiv_id":null,"evidence_quote":"Gives the efficient Monte Carlo approach to Shapley effects that Algorithm 1 extends to interaction estimation."},{"cited_title":"Efficient nested simulation for estimating the variance of a conditional expectation","cited_arxiv_id":null,"evidence_quote":"Introduces the unbiased nested Monte Carlo estimator for the variance of a conditional expectation used to estimate the value function in Algorithm 1."},{"cited_title":"Policy Optimization in Dynamic Bayesian Network Hybrid Models of Biomanufacturing Processes","cited_arxiv_id":null,"evidence_quote":"Defines the policy-augmented Bayesian network model, the target model whose random factors and posterior parameter distribution are used throughout."},{"cited_title":"Experimentally integrated dynamic modelling for intuitive optimisation of cell based processes and manufacture","cited_arxiv_id":null,"evidence_quote":"Supplies the nonlinear mature cell-progenitor negative-feedback dynamics used in the real-world PABN experiment."},{"cited_title":"Sampling permutations for shapley value estimation","cited_arxiv_id":null,"evidence_quote":"Provides the transformation method for sampling permutations used in the quasi-Monte Carlo version of Algorithm 2."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the low-discrepancy point-set construction and mapping used to implement quasi-Monte Carlo sampling."}],"review_version":1}