{"id":"27ce4a47-07b6-4291-a4d3-f39053a1224d","arxiv_id":"2502.08004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SBI-BOED trains a normalizing-flow likelihood surrogate while optimizing experimental designs via InfoNCE-style mutual information bounds, and works when the simulator is non-differentiable.","lead":"The authors show that simulation-based inference and Bayesian optimal experimental design can be optimized together through a shared mutual-information bound, using a normalizing flow as an amortized likelihood surrogate so that non-differentiable simulators can be used. The resulting method, SBI-BOED, improves calibration and predictive accuracy over benchmark design methods on SIR epidemiology and BMP biology simulators.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's claimed equivalence has a sign error: the InfoNCE objective's population limit minimizes KL_like − KL_marginal, not KL_like + KL_marginal, so the central theoretical bridge is not established.","rationale":"I read the paper in good faith. The empirical method SBI-BOED is plausible and the reported results on SIR and BMP may be useful, but the paper's central novelty is the theoretical bridge in Theorem 3.1. The reader's weakest assumption correctly identified the unproven finite-sample approximation in Appendix A.1, but my reading finds a more fundamental problem: the claimed equivalence has a sign error in the marginal KL term. In the population limit of the InfoNCE objective used in Eq. (6), the objective equals I − KL_like + KL_marginal, so maximizing it is equivalent to minimizing KL_like − KL_marginal. The theorem claims minimization of KL_like + KL_marginal. The proof's Eq. (22) reverses the direction of the marginal KL and changes the expectation variable, and Eq. (23) treats a φ-dependent term as constant. These are not cosmetic issues; they produce the wrong sign in the central claim. The proposed concrete test would settle the matter analytically in a toy model, but the derivation already shows the theorem cannot hold as stated. Because the paper's headline theoretical contribution is falsified, conditional acceptance is no longer appropriate; the authors would need to either correct the theorem, explicitly relabel it as a heuristic motivated by the bound, or substantially reframe the paper's claims. I therefore recommend REJECT, while acknowledging that the empirical design procedure may still have value if resubmitted with a corrected or weakened theoretical statement.","tokens_in":20585,"tokens_out":22540,"duration_ms":168709,"concrete_test":"Test the claimed equivalence in a toy model where the flow family cannot represent the true likelihood. Let p(θ)=N(0,1), p(y|θ)=N(y;θ,1), and pφ(y|θ)=N(y;aθ,2) with scalar a and variance fixed at 2. Compute the exact population objectives analytically: KL_like = 0.25(a−1)^2 + 0.5(0.5−1+log2) and KL_marginal = 0.5[log((a^2+2)/2) + 2/(a^2+2) − 1]. Evaluate J_NCE(a) = 0.5 log2 − KL_like + KL_marginal and J_Theorem(a) = 0.5 log2 − KL_like − KL_marginal over a ∈ [0,2]. The theorem predicts the same maximizer for both; numerically, J_NCE is maximized near a ≈ 1.3, while J_Theorem is maximized near a ≈ 0.85. If this discrepancy is confirmed, Theorem 3.1's claimed equivalence is falsified even before considering the finite-L approximation.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central Theorem 3.1 is contradicted by the InfoNCE objective it is meant to characterize. For the objective in Eq. (6), L_NCE = E log[ pφ(y|θ0) / ((L+1)^{-1} Σ_{ℓ=0}^L pφ(y|θℓ)) ], the L→∞ limit is E_{p(θ,y)} log pφ(y|θ) − E_{p(y)} log pφ(y), where pφ(y) = E_{p(θ)} pφ(y|θ). Adding and subtracting the true terms gives I(θ;y) − KL_like + KL_marginal, with KL_like = E_{p(θ)}KL(p(y|θ)∥pφ(y|θ)) and KL_marginal = KL(p(y)∥pφ(y)). Since I(θ;y) is constant in φ, maximizing this objective is equivalent to minimizing KL_like − KL_marginal, not KL_like + KL_marginal as Eq. (7) claims. The Appendix A.1 proof obscures this: Eq. (21)'s second term is E_{p(y)} log[ˆp(y)/p(y)] = −KL(p(y)∥ˆp(y)), but Eq. (22) replaces it with −KL(ˆp(y)∥p(y)), reversing both the KL direction and the expectation distribution; Eq. (23) then treats E_{ˆp} log p(y) as constant even though ˆp depends on φ. The sign of the marginal term is therefore an artifact of the derivation. This is not a finite-L approximation issue: the discrepancy survives at L=∞. By data processing, KL_like ≥ KL_marginal, so the InfoNCE objective remains a valid lower bound, but its optimum differs from the theorem's objective whenever the flow family cannot represent the true likelihood exactly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SBI-BOED, a method for jointly training an amortized normalizing-flow likelihood surrogate and optimizing experimental designs in simulation-based inference without differentiating through the simulator. The method maximizes an InfoNCE-style lower bound on mutual information (Eq. 6), adds a likelihood regularizer INCE-λ (Eq. 8), and uses a truncated-normal design distribution with checkpoints to stabilize design optimization. Experiments on a noisy linear model, an SIR epidemic model, and a BMP signaling model compare EIG, L-C2ST calibration, and median posterior prediction distance against MINEBED and iDAD. The central theoretical claim is Theorem 3.1, which asserts equivalence between maximizing the InfoNCE bound and minimizing the sum of the likelihood KL and marginal KL.","tokens_in":20944,"tokens_out":7328,"duration_ms":60927,"significance":"The proposed pipeline is practically appealing: it handles non-differentiable simulators, produces a normalized amortized likelihood, and the use of design distributions and checkpoints addresses a real optimization failure mode. The paper also reports external calibration (L-C2ST) and prediction-distance metrics rather than only EIG, and states that code is available. However, the theoretical bridge in Theorem 3.1 is incorrect as stated, the proof in Appendix A.1 relies on an unjustified finite-sample replacement, and the empirical comparisons rest on three seeds and method-specific EIG estimators. The core idea may still be salvageable, but the manuscript in its current form substantially overclaims.","major_comments":[{"comment":"Theorem 3.1 is not correct as stated. For Eq. (6), the L→∞ limit is E_{p(θ,y)} log p_φ(y|θ) − E_{p(y)} log p_φ(y), where p_φ(y)=E_{p(θ)}p_φ(y|θ). Adding and subtracting the true conditional entropy yields I(θ;y) − D_KL(p(y|θ)∥p_φ(y|θ)) + D_KL(p(y)∥p_φ(y)). Since I is independent of φ, maximizing the InfoNCE bound is equivalent to minimizing D_KL(p(y|θ)∥p_φ(y|θ)) − D_KL(p(y)∥p_φ(y)), not the sum in Eq. (7). The error originates in Appendix A.1: after Eq. (21), E_{p(y)} log[ˆp(y)/p(y)] = −D_KL(p(y)∥ˆp(y)), but Eq. (22) replaces it with −E_{ˆp(y)}D_KL(ˆp(y)∥p(y)), reversing the direction and the expectation; Eq. (23) then treats E_{ˆp(y)}log p(y) as constant even though ˆp depends on φ. This is not a finite-L artifact; it survives at L=∞. The theorem and the remark about reverse-KL bias in Appendix A.1 need to be corrected or the theoretical claims substantially restricted.","section":"§3.1, Appendix A.1, Eq. (7)"},{"comment":"The λ analysis in Appendix B.1 is not a valid KL-divergence calculation: Eq. (30) defines D_KL(p(y|θ)∥p_φ(y|θ)^{1−λ}), where p_φ^{1−λ} is generally unnormalized, so it is not a divergence and cannot be used to conclude that λ>0 yields 'broad coverage' and λ<0 yields 'mode-seeking'. The empirical λ sweeps in Sections 5.1 and 5.3 may still be valid evidence, but the paper should either provide a corrected analysis of the regularized objective or present λ as a purely heuristic regularizer.","section":"Appendix B.1, Eq. (30)"},{"comment":"The EIG column in Table 1 is not comparable across methods because each method estimates EIG with its own estimator: MINEBED uses the MINE bound, iDAD uses InfoNCE with a critic, and SBI-BOED uses the INCE-λ surrogate. Differences as small as 0.02–0.2 can easily be estimator bias. Therefore the statement that iDAD and differentiable MINEBED 'achieve the best information gain' is not supported; a common EIG estimator (e.g., a high-L InfoNCE or nested Monte Carlo estimate evaluated on the same samples) should be applied to all final designs.","section":"Table 1, Sections 5.3–5.4"},{"comment":"The empirical claims of 'notable improvements in inference' rest on 3 seeds and T=2 (SIR) or T=3 (BMP) design rounds. For SIR, SBI-BOED's EIG is 1.01–1.63 versus 2.67–2.69 for the benchmarks, and the calibration differences are within one standard error for several rows (e.g., 0.03±0.01 vs 0.07±0.05). More seeds, more design rounds, and ideally a shared evaluation protocol are needed before claiming state-of-the-art calibration and accuracy.","section":"Sections 5.3–5.4, Table 1"}],"minor_comments":[{"comment":"There is a typo, 'rouglhy' should be 'roughly'.","section":"Section 5.3"},{"comment":"The displayed inequality in Eq. (5) is not valid in general: log[exp(g0)/(L^{-1}\\sum_{\\ell=1}^L exp g_\\ell)] ≤ log[exp(g0)/((L+1)^{-1}\\sum_{\\ell=0}^L exp g_\\ell)] holds only when exp(g0) ≤ L^{-1}\\sum_{\\ell=1}^L exp(g_\\ell); otherwise the direction is reversed. Since this inequality is not needed once p_φ is a normalized density, either remove it or state the condition.","section":"Section 3.1, Eq. (5)"},{"comment":"The checkpoint condition 'if EIG ξ > EIGξ∗' uses the surrogate's own EIG estimate; this should be clearly described as an online model-selection heuristic, and the sensitivity of the final design to this choice should be discussed.","section":"Algorithm 1, line 9"},{"comment":"The caption says 'moving average over 10 different random seed initializations,' while Table 1 reports 3 seeds; please clarify the number of seeds used for each figure and table.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem and the empirical setup is thoughtful, but the main theorem is wrong as stated and the comparison protocol needs tightening. I would be willing to review a revised version. The novelty relative to Ivanova et al. (2021) and Miller et al. (2024) should also be clarified, since the InfoNCE-with-flow idea is close to existing contrastive SBI methods; the claimed novelty is mainly the design-distribution and checkpoint heuristics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the method is sensible and the empirical results on the BMP simulator are genuinely interesting, but the paper's central theorem is not just unproven—it's stated with the wrong sign. The InfoNCE limit is I(θ;y) − KL_like + KL_marginal, so maximizing it minimizes KL_like − KL_marginal, not KL_like + KL_marginal as Theorem 3.1 claims. The Appendix A.1 proof gets there by replacing one KL with its reverse and treating a φ-dependent term as constant, which is not a finite-L artifact. I checked: the sign error survives at L=∞. So the theoretical bridge in the paper is a heuristic, not a theorem.\n\nThat said, there is real value here. Using a normalized normalizing flow as the critic in an InfoNCE-style bound, with a design distribution and checkpoints, is a sensible way to do BOED on non-differentiable simulators. The BMP experiment is exactly the kind of closed-box simulator where MINEBED and iDAD don't apply, and SBI-BOED gets better calibration and median distance than MINEBED-BO. The authors also honestly report that higher EIG doesn't correspond to better calibration, and they show the naive no-design-distribution ablation fails. Those are useful empirical facts.\n\nThe soft spots beyond the theorem: Table 1 compares EIG values estimated with different estimators per method—that's apples-to-oranges and should be fixed. Only three seeds, and no random-design baseline. L-C2ST values near zero (0.001–0.004) are suspicious and need explanation—either the metric is saturated or the posterior is overconfident in a way not captured. The text says 'all code is available online' but I don't see a link. And λ is hand-tuned, which is fine, but the sensitivity should be discussed.\n\nMy bottom line: the empirical core is promising enough to deserve refereeing. The authors should either prove a corrected version of Theorem 3.1 (probably with the right sign and conditions) or explicitly demote it to a heuristic justification. A referee should also ask for a consistent EIG estimator and a code release. If those land, this could be a useful paper for people with expensive simulators and no gradients.","headline":"Solid empirical idea undermined by a wrong-sign theorem; worth reviewing if the theory is corrected or demoted.","tokens_in":21532,"tokens_out":2852,"would_cite":false,"duration_ms":24223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62B10","62F15","62K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that maximizing the InfoNCE mutual-information bound with a normalizing flow as critic both fits an amortized likelihood and optimizes experimental designs, with better-calibrated posteriors than two gradient-based…","keywords":["simulation-based inference","Bayesian optimal experimental design","mutual information","InfoNCE bound","normalizing flows","amortized likelihood","posterior calibration","gradient-free simulator"],"falsifier":"On a simulator with a closed-form likelihood and marginal — the paper's noisy linear model is one — fit two surrogates: one by maximizing the InfoNCE bound with a small contrastive batch, and one by directly minimizing the likelihood KL divergence of Equation (3). If the two optima do not converge to the same density as the contrastive batch size $L$ grows, Theorem 3.1 is false as stated. A second check on the same model: run SBI-BOED where the optimal design is known in closed form; if the selected design does not approach it as training steps and $L$ grow, the gradient path through the surrogate is not faithfully optimizing information gain.","tokens_in":20297,"feed_emoji":"🎯","tokens_out":12403,"duration_ms":94196,"temperature":0.7,"pith_summary":"This paper attempts to show that simulation-based inference and Bayesian optimal experimental design are the same optimization viewed from two sides. Its central claim, Theorem 3.1, is that maximizing the InfoNCE lower bound on the mutual information between parameters $\\theta$ and observations $y$ is equivalent to minimizing the Kullback-Leibler divergence between the true likelihood $p(y|\\theta,\\xi)$ and a surrogate $p_\\phi(y|\\theta,\\xi)$, plus a matching divergence between the true and surrogate marginal densities. If that equivalence holds, a single normalizing flow can be trained as an amortized likelihood surrogate while the same objective's gradients, taken through the flow rather than through the simulator, choose the next experiment. The paper packages this into the SBI-BOED procedure, adds a regularization term $\\lambda$ that trades information-bound tightness for likelihood accuracy, and reports that on an epidemiology (SIR) and a cell-signaling (BMP) simulator, SBI-BOED produces better-calibrated and closer posterior predictions than the MINEBED and iDAD baselines even when its expected information gain is lower.","feed_headline":"One bound now trains likelihoods and optimizes experiments","feed_subtitle":"A normalizing flow makes even non-differentiable simulators optimizable for design and calibration.","key_machinery":"The object that carries the argument is the InfoNCE mutual-information bound re-expressed with a normalizing flow in place of the usual classifier critic: $$L_{NCE}(\\xi,\\phi;L) = \\mathbb{E}\\left[\\log\\frac{p_\\$\\varphi$(y|\\theta_0,\\xi)}{\\frac{1}{1+L}\\sum_{\\ell=0}^{L}p_\\$\\varphi$(y|\\theta_\\ell,\\xi)}\\right],$$ where the expectation is over $p(\\theta_0)p(y|\\theta_0,\\xi)p(\\theta_{1:L})$ and $p_\\phi$ is a conditional normalizing flow (a neural spline flow in the experiments). The flow does two jobs at once: as a normalized density it makes the bracketed ratio a valid lower bound on $I(\\theta;y|\\xi)$, and as a pathwise generative model $y = f^{-1}(u;\\theta,\\xi)$ it lets gradients with respect to the design $\\xi$ be computed by backpropagation through the flow, so no gradient ever needs to pass through the simulator. Theorem 3.1 identifies the $\\phi$ that maximizes this bound with the $\\phi$ that minimizes the likelihood KL divergence plus a marginal-likelihood KL term. The refinement $L_{NCE\\text{-}\\lambda}$ appends $\\lambda\\log p_\\phi(y|\\theta_0,\\xi)$, and the optimizer acts on the parameters of a truncated-normal design distribution rather than on the designs themselves, with an annealing width schedule and design checkpoints to escape reward-poor regions and local minima.","core_discovery":"On its own terms, the discovery is that the InfoNCE bound, written with a normalized density estimator as the critic, is simultaneously an objective for likelihood-based SBI and for gradient-based BOED. Concretely, the paper defines $$L_{NCE}(\\xi,\\phi;L) = \\mathbb{E}_{p(\\theta_0)p(y|\\theta_0,\\xi)p(\\theta_{1:L})}\\left[\\log \\frac{p_\\$\\varphi$(y|\\theta_0,\\xi)}{\\frac{1}{1+L}\\sum_{\\ell=0}^{L}p_\\$\\varphi$(y|\\theta_\\ell,\\xi)}\\right],$$ with $\\phi$ parameterizing a normalizing flow $p_\\phi(y|\\theta,\\xi)$, and proves in Theorem 3.1 that maximizing this lower bound over $\\phi$ is equivalent to minimizing $\\mathbb{E}_{p(\\theta)}D_{KL}(p(y|\\theta,\\xi)\\|p_\\phi(y|\\theta,\\xi))$ together with the marginal-divergence term $\\mathbb{E}_{p(y|\\xi)}D_{KL}(p(y|\\xi)\\|p_\\phi(y|\\xi))$. Because the flow's sampling path $y = f^{-1}(u;\\theta,\\xi)$ is differentiable in the design $\\xi$, gradients for design updates flow through the surrogate, so the simulator itself never needs to be differentiable. The paper's regularized objective $L_{NCE\\text{-}\\lambda}$ adds $\\lambda\\log p_\\phi(y|\\theta_0,\\xi)$, and its design-distribution trick optimizes the parameters of a truncated normal over designs with an annealing width, with checkpoints that keep the best design found. Empirically, SBI-BOED reports lower L-C2ST calibration error and smaller median prediction distance than MINEBED-BO and iDAD on both the SIR and BMP tasks, while its expected information gain is lower on SIR but higher on BMP — evidence, the paper argues, that information gain and trustworthy inference should be evaluated separately.","pith_inferences":["The equivalence in Theorem 3.1, if exact, implies the converse direction the paper does not pursue: ordinary likelihood-based SBI training is implicitly a design-optimization procedure, so calibration-aware objectives could be imported into plain SBI practice even when no experiment is being planned.","Appendix B.2's bound, $I_{NCE\\text{-}\\lambda} \\le \\log(L+1) - \\lambda\\,\\mathbb{E}_{\\theta}H(y|\\theta,\\xi)$, suggests a testable shortcut: the surrogate's conditional entropy could serve as a cheap acquisition function during design search, replacing the separate EIG estimate used for checkpoints.","The paper's dissociation between EIG and calibration, if it replicates across more tasks and seeds, would push the field toward reporting calibration metrics alongside information gain in every BOED evaluation, since an overconfident posterior can accompany a high-information design.","Because the surrogate is trained on prior samples, its accuracy should degrade as the posterior concentrates over successive design rounds; reusing posterior samples as a proposal distribution for the next round, which the paper notes but does not implement, is the natural fix for the conservative SBC curves it reports."],"forward_implications":["Black-box simulators that cannot be differentiated become optimizable for experimental design: the gradient path runs through the surrogate flow, so the design update requires only simulator samples, not simulator gradients.","A single trained object serves both purposes at once — the amortized likelihood used for posterior inference is the same surrogate whose gradients selected the design — removing the separate critic/design-policy split in earlier gradient-based BOED methods.","Designs with the highest expected information gain are not automatically the ones that yield the best-calibrated posteriors; on the SIR and BMP tasks the lower-EIG SBI-BOED surrogates beat the higher-EIG baselines on L-C2ST and median prediction distance.","The $\\lambda$ dial lets a practitioner choose a working point on the trade-off between a tight mutual-information bound and an accurate likelihood surrogate, which matters most in high-dimensional design spaces where the unregularized objective is unstable.","Any generative model with an evaluable likelihood or likelihood lower bound (diffusion models, flow matching) can substitute for the normalizing flow in the same procedure."],"supporting_citations":[{"why":"Supplies the InfoNCE mutual-information lower bound that the paper rewrites with a normalizing flow as its critic.","marker":"van den Oord et al. (2019)"},{"why":"Provides the InfoNCE-based BOED formulation (iDAD) and the SIR epidemiology task that SBI-BOED extends and benchmarks against.","marker":"Ivanova et al. (2021)"},{"why":"MINEBED-BO, the main benchmark competitor, and the noisy linear model used to study design-dimension scaling.","marker":"Kleinegesse and Gutmann (2020)"},{"why":"Neural spline flows, the conditional density estimator used as the likelihood surrogate in all experiments.","marker":"Durkan et al. (2019)"},{"why":"The contrastive likelihood-learning derivation whose form Theorem 3.1 follows in Appendix A.1.","marker":"Miller et al. (2024)"},{"why":"Establishes normalizing flows as normalized, pathwise conditional density estimators, the properties the method relies on.","marker":"Papamakarios et al. (2021)"},{"why":"Defines L-C2ST, the local calibration metric that carries the paper's main empirical comparison.","marker":"Linhart et al. (2024)"},{"why":"The unified stochastic-gradient lineage for simultaneous design and inference-object optimization that SBI-BOED adapts to the likelihood-free setting.","marker":"Foster et al. (2020)"},{"why":"Supplies the calibration-crisis framing that motivates evaluating BOED methods on posterior faithfulness, not just EIG.","marker":"Hermans et al. (2022)"}],"fun_headline_variants":["One bound unifies simulation-based inference and experiment design","InfoNCE bound links SBI to BOED for optimal experiments","Normalizing flow makes non-differentiable simulators optimizable","Optimize experiments on simulators without gradients","Mutual info bound bridges inference and experiment design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bridge between the two fields stands on one substitution: in the proof of Theorem 3.1 the true distribution of simulated data is replaced by an average over a finite batch of surrogate-likelihood samples, and the proof marks that step as approximately equal; if this substitution is not accurate, maximizing the InfoNCE bound is no longer provably the same as fitting the likelihood, and the method's value rests on its empirical results alone.","fun_headline_variants_meta":{"raw":{"variants":["One bound unifies simulation-based inference and experiment design","InfoNCE bound links SBI to BOED for optimal experiments","Normalizing flow makes non-differentiable simulators optimizable","Optimize experiments on simulators without gradients","Mutual info bound bridges inference and experiment design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":3088,"prompt_tokens":1118,"completion_tokens":1970,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":734,"completion_tokens_details":{"reasoning_tokens":1892}},"tokens_in":734,"tokens_out":1970,"duration_ms":12080,"temperature":1.0,"reasoning_tokens":1892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:11:12.560204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a simulator with a closed-form likelihood and marginal — the paper's noisy linear model is one — fit two surrogates: one by maximizing the InfoNCE bound with a small contrastive batch, and one by directly minimizing the likelihood KL divergence of Equation (3). If the two optima do not converge to the same density as the contrastive batch size $L$ grows, Theorem 3.1 is false as stated. A second check on the same model: run SBI-BOED where the optimal design is known in closed form; if the selected design does not approach it as training steps and $L$ grow, the gradient path through the surrogate is not faithfully optimizing information gain.","supporting_citations":[],"review_version":1}