{"id":"334da58d-cd9e-4f55-b94c-e2a8345673ef","arxiv_id":"2411.15826","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A normalizing-flow-based method learns non-parametric joint prior distributions for Bayesian models by matching simulated predictions to expert-elicited statistics.","lead":"Bayesian statistics lets researchers combine prior knowledge with data, but turning expert opinions into a prior distribution is hard. This paper uses a type of neural network, a normalizing flow, to learn flexible joint priors from expert-answerable questions, and tests the approach in four simulation studies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The joint-prior demonstration depends on oracle correlation input that cannot be elicited, and even marginal priors for β1 and β2 are visibly underidentified; the central claim overstates what the experiments establish.","rationale":"I read the paper as a moderate engineering extension of a previously published simulation-based elicitation framework to normalizing flows, with reproducible code, archived software, and four internally consistent oracle simulations. The central claim is that the same framework can learn non-parametric joint priors with only minor workflow adjustments. For that claim to hold, the elicited statistics must contain enough information to constrain the joint prior. The reader's weakest-assumption analysis correctly identifies where this condition is least secure: in Simulation Study 2, Scenario 3, the correlation structure of the model parameters is fed directly into the loss as an 'elicited statistic,' and the authors themselves state that such correlation information cannot currently be elicited from domain experts. The joint-prior success is therefore largely attributable to the oracle input. Furthermore, Scenario 1 already shows that even the marginal priors for β1 and β2 are underidentified given the chosen elicitation design, with large across-seed variation in Figure 9 and no quantitative recovery metric. I do not see a fatal flaw in the normalizing-flow machinery or in the optimization; the MMD-based loss, the affine coupling architecture, and the SGD training are standard and the implementation appears sound for matching the supplied summary statistics. My concern is specifically that the paper's advertised capability—learning joint priors from expert knowledge—is not established outside a setting where the dependence structure is essentially provided as the answer. This is an addressable evidence gap, so a conditional verdict is appropriate. The proposed two-condition experiment would settle whether the concern lands: if the method fails when the exact correlation is withheld or replaced by a realistic proxy, then the joint-prior claim needs to be substantially qualified; if it succeeds, the concern is resolved. The reader's verdict of CONDITIONAL with high confidence is therefore unchanged by my review.","tokens_in":24449,"tokens_out":4876,"duration_ms":52265,"concrete_test":"Re-run Simulation Study 2, Scenario 3 under two conditions: (i) remove the correlation loss component entirely, keeping only the five quantiles per group and the R2 target; (ii) replace the exact Pearson correlation matrix with a realistic dependence proxy that could be obtained from experts, for example rank-based or pairwise-comparison information derived from the true prior, as in Mikkola et al (2024). In both conditions, compute the correlation matrix of the learned joint prior across the 30 seeds and compare it with the oracle correlations using a Frobenius-norm or energy-distance metric. If condition (i) fails to recover the correlations and condition (ii) degrades substantially relative to the oracle-input result, the joint-prior capability is supported only when the dependence structure is supplied as the answer, confirming the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated contribution is learning non-parametric joint priors, yet the only scenario with dependence (Simulation Study 2, Scenario 3) supplies the exact correlation matrix of the model parameters as an 'elicited statistic' (Section 4.1). The authors explicitly concede that this quantity cannot currently be obtained from real domain experts and that suitable elicitation methods for correlation information are lacking. The joint-prior demonstration is therefore an oracle exercise in which the dependence structure is provided as part of the input, not learned from interpretable expert-elicitable quantities. The paper's claim to have demonstrated 'successful learning of joint priors based primarily on interpretable quantities that can be meaningfully provided by domain experts' (Discussion) is not supported by the experiments for the joint, as opposed to marginal, aspect of the prior. The concern is compounded by the paper's own sensitivity analysis (Appendix A.2): β1 and β2 are each informed by only one group-specific target quantity, and Figure 9 shows substantial across-replication variation in the learned marginals for these parameters. No quantitative measure of prior recovery is reported; the evaluation is visual only. Thus the evidence shows that the method can match a supplied set of summary statistics, including a directly supplied correlation matrix, but it does not demonstrate that a non-parametric joint prior can be identified from expert knowledge alone. This is an evidence gap rather than an internal inconsistency: the normalizing-flow machinery, the reproducible code, and the archived implementation (Zenodo DOI) are all credible, and the paper is appropriately cautious in several places. But the headline claim goes beyond the experimental support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the simulation-based prior-elicitation framework of Bockting et al. (2024) to non-parametric joint priors by representing the prior with a normalizing flow (NF). The method learns the flow weights by minimizing a weighted sum of discrepancies between model-implied and expert-elicited statistics, which in the simulations are five quantiles of prior predictive distributions, R^2, and pairwise parameter correlations. Four oracle-based simulation studies are presented: a binomial regression with independent normal priors, and three normal-regression scenarios with independent, skewed-normal, and correlated multivariate-normal priors. The authors assess convergence through loss trajectories and slope diagnostics, examine learned priors and model-averaged results, and provide a sensitivity analysis for target-quantity informativeness. The central claim is that the same modular, simulation-based framework can learn flexible non-parametric joint priors using only minor workflow adjustments.","tokens_in":24712,"tokens_out":4750,"duration_ms":43666,"significance":"If the central claim holds, this would be a useful methodological advance: it extends simulation-based prior elicitation beyond parametric, independent priors to flexible joint distributions while retaining interpretable, observable target quantities. The paper contributes an open-source implementation (elicito), a reproducible workflow with code and results archived, and a thoughtful set of diagnostics (convergence slopes, sensitivity analysis, model averaging, oracle-based evaluation). The emphasis on non-identifiability and the honest discussion of when learned priors are not unique are valuable for practitioners. However, the experimental support for the joint-prior aspect is weaker than the abstract suggests: the only dependence scenario supplies the exact correlation matrix as an 'elicited statistic', and the evaluation of prior recovery is entirely visual. The framework itself is sound and the paper is clearly written, but the evidence for the advertised capability needs strengthening or the claims need qualification.","major_comments":[{"comment":"The only demonstration of learning a dependent joint prior (Simulation Study 2, Scenario 3) uses the exact correlation matrix of the model parameters as an 'elicited statistic'. In Section 4.1 you state: 'we use the exact correlation structure indicated by the ground truth' and 'We acknowledge that this information cannot be reasonably demanded from a domain expert in most cases.' This means the joint-prior result is an oracle exercise: the dependence structure is provided as input, not learned from interpretable quantities that a domain expert can meaningfully provide. The Discussion (Section 5) does concede this ('One exception is the use of correlation information...'), but the Abstract and the opening claim of Section 5 ('we demonstrated successful learning of joint priors based primarily on interpretable quantities that can be meaningfully provided by domain experts') overstate the empirical support. Please reframe the abstract and conclusions to state clearly that the current evidence for learning dependence is conditional on supplying correlation targets that are not currently elicitable from real experts.","section":"Section 4.1 and Section 4.3.3"},{"comment":"The evaluation of whether the method has learned the prior is based entirely on visual inspection of marginal and joint plots; no quantitative measure of prior recovery is reported. This matters because the training objective directly minimizes the discrepancy between model-implied and expert-elicited statistics, so a close match of those statistics (e.g., Figure 8) is the fitted outcome rather than an independent confirmation that the prior has been identified. The paper's 'successful learning' claim rests on visual judgments, which are especially concerning in Scenario 1: Figure 9 shows substantial between-seed variation for beta1 and beta2, and the sensitivity analysis in Appendix A.2 shows these parameters are informed by only one group-specific target quantity each, leaving them weakly identified. I recommend reporting quantitative metrics of prior recovery (e.g., MMD or energy distance between learned and true joint priors, coverage of true parameter values, or a scalar summary of cross-seed variability) and explicitly stating in the abstract and discussion which model parameters are identifiable under the chosen elicited statistics.","section":"Section 4.1, Section 4.3.1, and Figures 8-9"}],"minor_comments":[{"comment":"There is a typo: 'Stimulation Study 2' should read 'Simulation Study 2' in the sentence about the fixed learning rate for Scenario 3.","section":"Section 3 (Implementation)"},{"comment":"The caption is missing a period after 'replications': it currently reads 'Learned joint prior for all 30 replicationsEach replication is shown in a different color.'","section":"Figure 14 caption"},{"comment":"The sentence 'we except to observe more variation in the learned prior distributions' should read 'we expect to observe'.","section":"Section 4.3.2"},{"comment":"The reference 'lgorzata Roos M, Martins TG, Held L, et al' appears to have a typographical error in the author name; it should likely be 'Małgorzata Roos' or 'Roos M'.","section":"References"},{"comment":"The statement 'With a proper discrepancy measure as loss function, it is guaranteed that the total loss approaches zero as learning progresses' is not generally true for stochastic gradient descent, since minibatch noise and finite training time prevent exact convergence to zero; please rephrase to avoid the implication of a formal guarantee.","section":"Section 2.5 (Convergence Checks)"}],"recommendation":"major_revision","confidential_remarks":"This paper presents a reasonable extension of the authors' earlier framework, with reproducible code and a thoughtful treatment of non-identifiability. The main concern is the gap between the abstract's claim of 'successful learning of joint priors based primarily on interpretable quantities' and the experimental design, which relies on oracle correlation input and lacks quantitative prior-recovery metrics. The revision should either add such metrics and an indirect correlation-elicitation scenario, or clearly delimit the claim to what the experiments actually establish. I would not recommend rejection, because the framework itself is sound and the limitations are explicitly acknowledged in places; however, the central claim needs to be aligned with the evidence before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: using normalizing flows inside the simulation-based elicitation framework so the learned prior need not be parametric or independent. The code is archived, the simulations are internally consistent, and the sensitivity analysis is a genuinely useful addition. I believe the paper deserves a serious referee; it just needs to be read with the oracle caveat front and centre.\n\nWhat the paper does well: the modular framework is cleanly extended; the MMD loss, the model-averaging over seeds, the convergence diagnostics, and the explicit discussion of non-identifiability are all sensible. The authors are candid in Section 4.1 that correlation information 'cannot be reasonably demanded from a domain expert' and that suitable elicitation methods are lacking. That is the right degree of honesty.\n\nThe soft spot is exactly where the stress-test lands. The only scenario with dependence (Study 2, Scenario 3) feeds the true correlation matrix in as an 'elicited statistic.' So the dependence structure is part of the input, not learned from interpretable expert quantities. The Discussion's phrase 'based primarily on interpretable quantities' is doing the work—'primarily' carries the correlation exception. But the paper's headline contribution is joint priors, and the joint part has no empirical support outside oracle inputs. Also, recovery of the marginals for β1 and β2 in Scenario 1 is visibly weak (Figure 9), and the evaluation is entirely visual; there is no quantitative summary of prior recovery. These are evidence gaps, not internal contradictions. The supplementary validation—training directly on model parameters—is the right check and suggests the flow itself can represent the joint prior when given enough information; that should be moved into the main text or at least summarized quantitatively.\n\nWho is this for? Someone working in prior elicitation or applied Bayesian workflow who wants a flexible way to encode expert knowledge that isn't tied to a parametric family. It will not reshape the field, but it is a working, reproducible step in a useful direction. I would send it to review with a request to address the oracle-dependence issue honestly—either by adding a realistic expert study or by reframing the contribution as 'a method that can use correlation information when it is available.'","headline":"A clearly described, reproducible extension of simulation-based prior elicitation to normalizing flows, but the joint-prior payoff currently rests on oracle correlation input that experts cannot provide.","tokens_in":25313,"tokens_out":1760,"would_cite":false,"duration_ms":16564,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15"],"pacs":[],"model":"deepseek-v4-flash","headline":"Normalizing flows turn expert answers into trainable joint priors","keywords":["prior elicitation","expert knowledge","joint prior distribution","non-parametric priors","normalizing flows","simulation-based inference","maximum mean discrepancy","Bayesian workflow"],"falsifier":"Run the elicitation pipeline on a known correlated prior, but give the 'expert' only the five quantiles per group and $R^2$—no correlation values. If the ensemble of learned flow priors across 30 seeds shows pairwise correlation estimates whose spread is comparable to the prior range rather than collapsing around the true matrix, then the method is not actually learning dependence from interpretable expert quantities and the joint-prior claim is unsupported.","tokens_in":24232,"feed_emoji":"📊","tokens_out":15301,"duration_ms":110686,"temperature":0.7,"pith_summary":"The paper adapts an existing simulation-based prior-elicitation workflow so that it can learn non-parametric joint priors over a Bayesian model's parameters, instead of only independent parametric priors. The prior is represented by a normalizing flow, a neural generative model that maps a simple base distribution to a flexible target density with exact density evaluation. The expert supplies interpretable predictive statistics—five quantiles of prior predictions, plus $R^2$—and the flow is trained so its prior predictive statistics match those answers; pairwise correlations among parameters are also supplied. Across four simulation studies, the paper reports recovery of target marginals, including skew, and capture of dependence when correlation information is provided. The wider claim is that one modular workflow can cover parametric and non-parametric, independent and joint elicitation.","feed_headline":"Flow-based elicitation learns joint priors from expert statistics","feed_subtitle":"Predictive quantiles and R-squared drive a normalizing flow that captures skew and dependence.","key_machinery":"The load-bearing object is a normalizing flow $g_\\lambda = g_{\\lambda_H} \\circ \\cdots \\circ g_{\\lambda_1}$, a composition of invertible affine coupling layers whose density is $p_\\lambda(\\theta) = p(u = g_\\lambda(\\theta))\\,|\\det g_\\lambda'(\\theta)|$. Sampling in the generative direction $u \\sim p(u)$, $\\theta = g_\\lambda^{-1}(u)$ turns the flow into a trainable joint prior. Training couples this flow to the elicitation pipeline: forward simulation from prior and likelihood produces model-implied target quantities (predictive quantiles, $R^2$, correlations), a weighted multi-objective loss (maximum mean discrepancy, with squared error for correlations) measures discrepancy from expert answers, and stochastic gradients update the flow weights. Supporting machinery includes a sensitivity analysis for testing whether each elicited statistic responds to prior changes, an oracle simulation for assessing identifiability before real experts are consulted, and loss-weighted model averaging for handling the non-uniqueness of learned priors.","core_discovery":"The paper's central claim is that the same mini-batch stochastic-gradient-descent workflow used for parametric prior elicitation can, with only minor adjustments, learn a non-parametric joint prior $p_\\lambda(\\theta)$ by treating the prior as a normalizing flow with trainable network weights $\\lambda$. A weighted sum of loss components, chiefly maximum mean discrepancy between model-implied and expert-elicited statistics, drives the training; the flow's change-of-variables formula makes density evaluation exact, and sampling proceeds in the generative direction $\\theta = g_\\lambda^{-1}(u)$. The simulations lead the authors to conclude that joint priors can be learned from primarily interpretable quantities, with the explicit caveat that the correlation information the method currently requires is not yet obtainable from real experts through a developed elicitation technique.","pith_inferences":["A realistic elicitation protocol that replaces the exact correlation oracle—for example, pairwise comparison or ranking of scenarios—would be the decisive test of whether the joint-prior claim holds with real experts.","The same pipeline could be applied to hierarchical or latent-variable models by choosing elicited statistics at the group or latent level; the sensitivity analysis would reveal whether those statistics are informative enough.","Seed-to-seed variation in learned marginals could serve as a practical identifiability diagnostic: large variation after convergence signals that the elicitation protocol needs more quantiles, more target quantities, or regularization.","A learned flow prior from one model could seed elicitation for a related model, reducing the number of expert queries needed, since the flow provides a smooth, trainable initial density."],"forward_implications":["Users can specify priors without choosing parametric families: skew and dependence are represented by the flow whenever the elicited statistics carry that information.","The same modular workflow extends to independent, joint, parametric, and non-parametric priors, so evaluation diagnostics and model averaging transfer across settings.","The oracle-based sensitivity analysis lets researchers check before consulting an expert which aspects of the prior are identifiable from the planned questions.","Loss-weighted model averaging across random-seed replications converts the non-uniqueness of learned priors into a final prior plus a diagnostic for degenerate runs.","Because the framework only needs forward simulation and a differentiable loss, alternative generative models such as diffusion or flow-matching could replace normalizing flows without changing the elicitation protocol."],"supporting_citations":[{"why":"Introduces the simulation-based elicitation framework and the parametric-prior method that this paper extends to normalizing-flow priors.","marker":"Bockting et al 2024"},{"why":"Supplies the normalizing-flow formalism, including the change-of-variables formula used for exact density evaluation.","marker":"Kobyzev et al 2020"},{"why":"Defines the four-stage elicitation process and the quantile-based elicitation techniques that structure the workflow.","marker":"Garthwaite et al 2005"},{"why":"Provides the rationale for eliciting quantiles and observable quantities rather than model parameters.","marker":"Kadane and Wolfson 1998"},{"why":"Gives the maximum mean discrepancy used as the main loss between model-implied and expert-elicited statistics.","marker":"Gretton et al 2006"},{"why":"Establishes one of the prior-predictive matching approaches that the simulation-based framework builds on.","marker":"Hartmann et al 2020"},{"why":"Demonstrates prior-predictive elicitation and the need for regularization when elicited statistics underdetermine the prior.","marker":"Manderson and Goudie 2023"},{"why":"Proves that affine coupling flows are distributional universal approximators, justifying the architecture's expressiveness for arbitrary target priors.","marker":"Draxler et al 2024"}],"fun_headline_variants":["Flow-based priors learn joint expert beliefs, but correlation elicitation lags","Normalizing flows turn expert quantiles into non-parametric joint priors","Joint prior elicitation via flows: skew and dependence, yet expert gaps remain","From expert stats to flexible joint priors with a normalizing flow","Flow method elicits joint priors, but correlation queries still untested"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's joint-prior learning depends on someone supplying the exact pairwise correlations between model parameters, and the paper states that current elicitation techniques cannot yet obtain this information from real experts.","fun_headline_variants_meta":{"raw":{"variants":["Flow-based priors learn joint expert beliefs, but correlation elicitation lags","Normalizing flows turn expert quantiles into non-parametric joint priors","Joint prior elicitation via flows: skew and dependence, yet expert gaps remain","From expert stats to flexible joint priors with a normalizing flow","Flow method elicits joint priors, but correlation queries still untested"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1197,"prompt_tokens":822,"completion_tokens":375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":278}},"tokens_in":438,"tokens_out":375,"duration_ms":3871,"temperature":1.0,"reasoning_tokens":278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:51:46.618270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the elicitation pipeline on a known correlated prior, but give the 'expert' only the five quantiles per group and $R^2$—no correlation values. If the ensemble of learned flow priors across 30 seeds shows pairwise correlation estimates whose spread is comparable to the prior range rather than collapsing around the true matrix, then the method is not actually learning dependence from interpretable expert quantities and the joint-prior claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the simulation-based elicitation framework and the parametric-prior method that this paper extends to normalizing-flow priors."},{"cited_title":"Statistician 47(1):3--19","cited_arxiv_id":null,"evidence_quote":"Provides the rationale for eliciting quantiles and observable quantities rather than model parameters."},{"cited_title":"In: Peters J, Sontag D (eds) Proc.Conf.UAI , PMLR, pp 1129--1138, ://proceedings.mlr.press/v124/hartmann20a.html","cited_arxiv_id":null,"evidence_quote":"Establishes one of the prior-predictive matching approaches that the simulation-based framework builds on."},{"cited_title":"arXiv preprint doi:10.48550/arXiv.2303.08528","cited_arxiv_id":null,"evidence_quote":"Demonstrates prior-predictive elicitation and the need for regularization when elicited statistics underdetermine the prior."}],"review_version":1}