{"id":"da20126d-3e28-4333-b954-364f6bcfe490","arxiv_id":"2507.15976","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Active learning with SNGP and BNN-NCP models constructs QuaLiKiz surrogate training sets from 100 to 10,000 points, beating random sampling and approaching prior ensemble-based ADEPT efficiency.","lead":"This paper shows that active learning with uncertainty-aware neural networks can build training datasets for plasma turbulence surrogate models using about ten thousand simulation runs, reaching an R2 around 0.75 and an F1 around 0.8 on a held-out test set. The result matters because generating training data with expensive gyrokinetic codes is often the bottleneck for deploying fast surrogate models in fusion design.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The efficiency claim is only demonstrated in-distribution: the unlabelled pool and test set share the same hand-fitted Gaussian, and Appendix B shows severe regressor collapse under distribution mismatch, so the claimed advantage may not transfer to real applications.","rationale":"The paper's strongest asset is its direct random-sampling baseline and its transparent acknowledgement of the mismatch problem in Appendix B. The central claim is not that the models are production-ready—the paper explicitly says they are not—but that uncertainty-guided acquisition builds a dataset that, at equal size, beats random acquisition from the same pool. That claim is internally supported by Figures 6-7. However, the comparison is only meaningful if the test distribution is the intended application distribution. Since the pool itself is a Gaussian fit to JET, and the test set is sampled from the same Gaussian, the measured improvement is an in-distribution interpolation result. Appendix B is the paper's own demonstration that this assumption is fragile: the regressors collapse when the application is uniform over the same bounds. Real tokamak profile distributions are non-Gaussian and device-specific, so the same failure mode is plausible in the intended use case. I therefore identify distribution mismatch as the single most load-bearing concern. Other issues—lack of repeated-seed error bars, the weak ETG regressor, and the classifier plateau—are real but either acknowledged as future work or less central to the 'notably better than random' claim. The proposed test settles the concern by measuring the method on a distribution the pool was not fitted to. The verdict is unchanged: the paper earns CONDITIONAL, with the condition being that the pool/application distribution assumption be validated or the claim be scoped accordingly.","tokens_in":28303,"tokens_out":6499,"duration_ms":80432,"concrete_test":"Evaluate the final AL-trained models (or re-run the pipeline) on a held-out application test set drawn from a different distribution—for example, a uniform distribution over the same 16D bounds as in Appendix B, or a profile-based set from another tokamak such as DIII-D or ITER—and compare F1/R2 per output against the same-size random-sampling baseline. If the average R2/F1 drops below the baseline or below the claimed ~0.75/~0.8 values for more than one output, the headline efficiency claim must be restricted to the pool distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported efficiencies—F1 ~0.8, R2 ~0.75, and the advantage over random sampling—are measured on a test set drawn from exactly the same hand-fitted multivariate Gaussian (Appendix A, Eqs. 18-26) used to generate the unlabelled pool (Section 3.2). The paper's own Appendix B shows that when the application distribution differs from the pool, regressor performance collapses catastrophically in the Gaussian-pool/uniform-application case (Figure 12), while even the classifier needs roughly double the data to recover. Because the intended use is a multi-machine surrogate built from a JET-fitted pool, this is not an edge case: real prediction tasks have profile distributions shifted from the pool. The pool is 'broadly defined' only by expanding a few variances and rotation correlations within the same Gaussian family. The central claim of efficient dataset construction is therefore conditional on the user pre-specifying an application distribution and building the pool to match it; the paper does not provide a way to construct a pool robust to unknown applications. This does not invalidate the proof-of-principle within the pool domain, but it is the load-bearing assumption behind 'notably better' as a general statement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a proof-of-principle active-learning pipeline for constructing training datasets for QuaLiKiz turbulent transport surrogates. It combines an SNGP classifier for instability presence with a BNN-NCP regressor for each flux channel, evaluates uncertainty-based acquisition functions over a large unlabelled pool generated from a hand-fitted multivariate Gaussian, and labels selected candidates with in-the-loop QuaLiKiz simulations. After 45 iterations growing the training set from 100 to roughly 10^4 points, the authors report F1 around 0.8 and R2 around 0.75 on a test set drawn from the same Gaussian distribution, and argue this is notably better than random sampling and extrapolates toward the earlier ADEPT ensemble approach. The paper also introduces a Fisher-Rao replacement for the KL-divergence terms in the BNN loss and a Sobol-weighted multi-objective acquisition function.","tokens_in":28539,"tokens_out":5738,"duration_ms":60251,"significance":"If the in-distribution result is taken at face value, the paper makes a useful contribution: it demonstrates that single-model uncertainty-aware architectures can replace deep ensembles in an active-learning loop for a high-dimensional plasma physics surrogate, with open-source code, physics-based filtering, and honest reporting of failure modes in Appendix B. The multi-objective acquisition and the Fisher-Rao based robust training loss are practical innovations. However, the headline efficiency claims are conditional on the unlabelled pool and test set sharing the same hand-fitted Gaussian, the metrics are single-run point estimates with some channels much weaker than others, and the ADEPT parity is extrapolated rather than measured. The significance is therefore that of a well-scoped proof-of-principle, not a validated general-purpose methodology.","major_comments":[{"comment":"The efficiency claim is demonstrated only in-distribution. The unlabelled pool (Section 3.2) and the test set are both drawn from the same hand-fitted multivariate Gaussian defined in Appendix A (Eqs. 18-20), and Appendix B's Figure 12 shows that when the application distribution is uniform while the pool is Gaussian, the regressor performance collapses catastrophically. Because the stated goal is a multi-machine surrogate built from a JET-fitted pool, this is a load-bearing assumption behind the Section 5 statement that the AL result is notably better than randomly sampling an equivalently sized dataset. The paper itself acknowledges this in Appendix B by advising that the high-probability region of the Gaussian must cover the application domain, but it does not remove the limitation. I request either an additional experiment measuring AL versus random sampling under a distribution shift, or a revised claim that explicitly scopes the efficiency result to the case where the pool distribution matches the application distribution.","section":"Section 3.2 / Appendix A / Appendix B"},{"comment":"The headline F1 around 0.8 and R2 around 0.75 are single-run point estimates with no error bars or multiple-seed variability. Figure 7 shows the classifier F1 is essentially flat as the training set grows, and Figure 6 (top right) shows the ETG Qe regressor is markedly weaker than the other channels, sometimes at or below zero. The abstract and conclusion state these metrics across all outputs, which overstates what is shown. Please report per-output final metrics, ideally with means and standard deviations over at least three to five seeds, and clarify that the summary metrics are not uniform across modes and channels.","section":"Section 4.1, Figures 6-7"},{"comment":"The comparison with the ADEPT ensemble is extrapolative, not measured. The text says that by extrapolating the observed performance trajectory, the AL methodology appears to approach the previous ADEPT performance, and no ADEPT model is evaluated on the same test set in this paper. This extrapolation is reasonable as a discussion point, but it should be presented explicitly as an extrapolation rather than as an established equivalence. If feasible, a head-to-head comparison on a common test set would substantially strengthen the claim.","section":"Section 5"},{"comment":"The efficiency comparison does not include an analysis of sensitivity to acquisition hyperparameters or to the manual per-iteration adjustment of the candidate buffer size. Because these choices directly affect the number and location of labels, their absence leaves open the possibility that part of the reported AL advantage over random sampling stems from these manual choices rather than from the uncertainty-aware acquisition itself. A brief sensitivity study, or at least an explicit statement of which results depend on manual tuning, would make the efficiency claim more robust.","section":"Section 3.3 and Section 4.1"}],"minor_comments":[{"comment":"The caption refers to the ion heat flux, Pi_i, but Pi_i was defined in Section 3.2 as the ion momentum flux; please correct the caption.","section":"Figure 16 caption"},{"comment":"The sentence 'QuaLiKiz executions continued to simultaneous produce the necessary output' contains a typo; 'simultaneous' should be 'simultaneously'.","section":"Section 3.3"},{"comment":"The caption states that gamma_abs,d and gamma_rel,d are equal across channels, but the main text explains why negative values are useful; adding one sentence on this to the caption would improve readability.","section":"Table 5 caption"},{"comment":"The caption says that models with R2 <= 0 are set to zero, which visually inflates the apparent improvement; please state how many such points occur, or plot the negative values explicitly.","section":"Figure 6 caption"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the distribution-mismatch concern from the stress-test note is legitimate and is already acknowledged by the authors in Appendix B; I would make it the central point of the revision. The paper is a credible proof-of-principle and the open-source code is a strength. My recommendation of major_revision is driven by the need to either demonstrate robustness under distribution shift or substantially narrow the claims, and to report uncertainty on the headline metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is that the ADEPT ensemble trick can be replaced by single uncertainty-aware models: SNGP for classification, BNN-NCP for regression, with a Fisher-Rao loss replacing KL divergence and a Sobol-weighted multi-objective acquisition. That is a genuine, if incremental, advance. The paper earns credit for measuring against a random-sampling baseline, for documenting the pool construction and hyperparameters in enough detail to reproduce, and for shipping code. It also explicitly flags its own limitations: the BNN occasionally collapses, large fluxes are underpredicted, the ETG Qe channel is weak, and the classifier barely improves with dataset size. That honesty is rare and useful.\n\nThe central result holds up in-distribution: AL beats random sampling for most regressors and classifiers, reaching roughly F1=0.8 and R2=0.75 on a test set drawn from the same hand-fitted multivariate Gaussian used to generate the pool. I do not buy the stress-test claim that this invalidates the paper—it is a proof-of-principle, and the authors themselves warn that the pool must match the application domain. But the caveat is real and should be stated loudly: the headline efficiency is conditional on that match. Appendix B shows regressor collapse when the application distribution differs from a Gaussian pool, and their response (oversampling the same Gaussian to 5e8 points) does not solve the mismatch problem—it just makes the Gaussian denser. So the 'broadly defined pool' claim is overstated.\n\nOther soft spots, in proportion: the headline metrics are single-run point estimates with no seed averaging or error bars; the ADEPT parity claim is extrapolated, not measured head-to-head; and the acquisition function hyperparameters are tuned on a validation set drawn from the same distribution, which contributes a mild circularity. None of these break the paper. The flat classifier F1 and weak ETG Qe are honestly reported and plausibly explainable.\n\nWho should read it: anyone building surrogates for expensive plasma physics codes, and machine-learning practitioners working on active learning for scientific simulators. It deserves a serious referee. I would accept it for peer review and ask for seed-averaged results and a more honest treatment of the distribution-mismatch limitation. The core method is sound, and the paper is a legitimate upgrade to the ADEPT program rather than a new paradigm.","headline":"A solid, honest proof-of-principle that active learning with uncertainty-aware single models can build QuaLiKiz surrogates as efficiently as the ADEPT ensemble pipeline, with the caveat that the efficiency is only demonstrated in the same Gaussian-distributed pool used for training.","tokens_in":29186,"tokens_out":915,"would_cite":true,"duration_ms":12109,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active learning with uncertainty-aware networks can build compact tokamak turbulence surrogate datasets, reaching F1 around 0.8 and R2 around 0.75 with about ten thousand labels and beating random sampling.","keywords":["active learning","uncertainty-aware neural networks","plasma turbulent transport","QuaLiKiz surrogate","tokamak fusion","Bayesian neural network","Fisher-Rao distance","dataset construction"],"falsifier":"Run the identical active-learning pipeline with the unlabelled pool drawn from a uniform distribution over the same input bounds while the test set remains Gaussian, and compare final F1 and R2; the paper's own mismatch study says the Gaussian-pool/uniform-application combination collapses the regressor, so if the uncertainty-aware pipeline shows the same collapse the reported efficiency is pool-specific, and if it does not the method is more general than the paper claims.","tokens_in":28079,"feed_emoji":"⚛️","tokens_out":11446,"duration_ms":111681,"temperature":0.7,"pith_summary":"Active learning, with a physics simulator acting as the label provider, is proposed as a way to build small training sets for surrogate models of plasma turbulent transport. The paper demonstrates this on the QuaLiKiz quasilinear gyrokinetic transport code, starting from 100 labelled inputs and adding roughly 200 points per iteration over 45 iterations to reach a final set of about $10^{4}$ points. The resulting classifier and regressor networks reach $F_1$ around 0.8 and $R^2$ around 0.75 on an independent test set that follows the same distribution as the unlabelled pool, and this is better than randomly sampling an equivalently sized dataset from that pool. The relevance is that more expensive gyrokinetic codes could, in principle, be treated the same way, making surrogate training affordable at higher fidelity while replacing an ensemble of networks with a single uncertainty-aware model per output.","feed_headline":"Active learning beats random sampling for fusion surrogate data","feed_subtitle":"One model per output matches ensemble-level accuracy, using 45 iterations and ~10^4 labels.","key_machinery":"The pipeline is built on Bayesian last layer architectures: a spectral-normalized Gaussian process classifier produces a mode-presence probability with an uncertainty output, and a Bayesian neural network with noise-contrastive prior regressor returns a mean prediction, a model-uncertainty estimate, and a data-uncertainty estimate. A custom acquisition function combines classifier uncertainty with a weighted sum of absolute and relative regressor uncertainties, and a low-discrepancy sampler generates random weight vectors so that multiple transport outputs compete fairly in a multi-objective selection. A closed-form Fisher-Rao distance metric replaces the asymmetric KL-divergence in the regressor loss, improving training robustness, and physics-based filters clean the labelled outputs before they are appended. The loop works by evaluating the current models on a random subset of a very large unlabelled pool, scoring candidates, sending the top-scoring inputs to QuaLiKiz for labelling, and growing the training set.","core_discovery":"The central claim is that a single-model, uncertainty-aware architecture can drive an active-learning loop that constructs a QuaLiKiz surrogate dataset about as efficiently as the earlier ensemble-based pipeline, but with only one network to train, store, and load per output variable. Starting from a 100-point seed set and adding roughly 200 labelled points per iteration over 45 iterations, the classifier-regressor pairs reach $F_1$ around 0.8 and $R^2$ around 0.75 on a test set drawn from the same distribution that generated the unlabelled pool. The paper presents this as a proof-of-principle for smaller datasets, explicitly framed as a proxy for more expensive gyrokinetic codes, and states that the final performance is notably better than random acquisition from the pool at the same dataset size.","pith_inferences":["An implication the paper leaves implicit is that the 45-iteration schedule and fixed candidate buffer may not be the main driver of efficiency; the same pipeline with adaptive batch sizes or an adaptive acquisition-function schedule could show faster convergence, since the paper itself notes diminishing returns.","A testable extension would swap the pool distribution while holding all other components fixed; the appendix's mismatch study suggests regressors are the fragile part, so a production version should validate the pool against the target machine's expected operating space before trusting the reported numbers.","Because classifier and regressor outputs are blended, classifier errors can feed into regressor acquisition in later iterations; the paper flags this feedback risk but does not quantify it, so tracking per-iteration classifier F1 as a real-time gate could be a cheap safeguard.","The single multi-output Bayesian network experiment failing while single-output models succeed suggests the covariance information between transport channels is not captured; an explicit covariance-aware loss or a small final correlation layer would be a natural next step to recover that information."],"forward_implications":["If the claim holds, a surrogate for a costly gyrokinetic code can be trained with roughly 10,000 high-value simulator runs instead of relying on a comparably performing random dataset of the same size.","Deployment keeps one trained network per output instead of an ensemble, reducing memory, training time, and loading overhead in integrated transport solvers.","The multi-objective acquisition scheme means one active-learning campaign can simultaneously improve classification and multiple flux regressors for all three turbulence modes.","The Fisher-Rao loss replacement, if it generalizes, gives other uncertainty-aware regression models a more stable training signal in low-data regimes.","The same simulator-in-the-loop design can be ported to other expensive physics codes as long as a suitable unlabelled pool and physics filters exist."],"supporting_citations":[{"why":"The earlier ensemble-based active-learning pipeline whose dataset efficiency this work claims to match, on a problem with one extra input dimension.","marker":"[27]"},{"why":"Supplies the experimental tokamak plasma dataset whose statistics define the multivariate Gaussian sampling distribution and the physics-based filter conventions.","marker":"[17]"},{"why":"Establishes the classifier/regressor split and the strategy of training regressors only on unstable points, which the pipeline inherits.","marker":"[19]"},{"why":"Defines the spectral-normalized Gaussian process classifier architecture used for mode-presence prediction with uncertainty.","marker":"[51]"},{"why":"Defines the Bayesian neural network with noise-contrastive prior regressor that supplies model and data uncertainty estimates.","marker":"[52]"},{"why":"Is the prior fusion-data application of the Bayesian neural network whose hyperparameters and training recipe the paper adapts.","marker":"[57]"},{"why":"Provides the closed-form Fisher-Rao distance used to replace KL-divergence in the regressor loss.","marker":"[67]"},{"why":"Is the QuaLiKiz quasilinear gyrokinetic model used as the in-the-loop labeller for the active-learning pipeline.","marker":"[26]"}],"fun_headline_variants":["Active learning trims data for plasma surrogate models","Uncertainty-aware active learning boosts fusion surrogate efficiency","One model active learning matches ensemble for plasma surrogates","Efficient data curation for fusion turbulence surrogates","Active learning shows merit for fusion model training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-fitted multivariate Gaussian distribution used to generate the unlabelled pool describes the domain where the surrogate will actually be applied; the test set is drawn from that same distribution, and Appendix B shows regressor performance degrades sharply when the application distribution does not match the pool.","fun_headline_variants_meta":{"raw":{"variants":["Active learning trims data for plasma surrogate models","Uncertainty-aware active learning boosts fusion surrogate efficiency","One model active learning matches ensemble for plasma surrogates","Efficient data curation for fusion turbulence surrogates","Active learning shows merit for fusion model training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1617,"prompt_tokens":1044,"completion_tokens":573,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":498}},"tokens_in":660,"tokens_out":573,"duration_ms":5762,"temperature":1.0,"reasoning_tokens":498,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:21:00.356557+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical active-learning pipeline with the unlabelled pool drawn from a uniform distribution over the same input bounds while the test set remains Gaussian, and compare final F1 and R2; the paper's own mismatch study says the Gaussian-pool/uniform-application combination collapses the regressor, so if the uncertainty-aware pipeline shows the same collapse the reported efficiency is pool-specific, and if it does not the method is more general than the paper claims.","supporting_citations":[],"review_version":1}