REVIEW 3 major objections 7 minor 21 references
Bayesian Active Learning By Distribution Disagreement
T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that for active learning in regression with normalizing flows, the best acquisition signal is disagreement among MC-dropout predictive distributions, not the spread of a single distribution.
desk verdict BALSA gives AL practitioners a useful new disagreement heuristic for normalizing flows, but the unnormalized KL scores need a theoretical fix before the SOTA claim is trustworthy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the BALSA acquisition score, a sum of distributional distances over k Monte-Carlo-dropout parameter samples instead of BALD's entropy subtraction. Two distance computations carry the method: the grid version discretizes the normalized target range into 200 bins, averages the resulting likelihood vectors to form $\bar{p}|x$, and scores $\sum_i \mathrm{KL}(\hat{p}_{\theta_i}|x,\bar{p}|x)$; the pair version avoids averaging and scores $\sum_i \mathrm{KL}(\hat{p}_{\theta_i}|x,\hat{p}_{\theta_{i+1}}|x)$. An EMD variant computes pairwise Earth mover distances over iid samples. The paper defines these scores on unnormalized likelihood vectors, arguing from an ablation that renormalization changes performance only slightly.
What would settle it
Run BALSA KL Grid on the same unlabeled pool with grid resolutions of 50, 200, and 1000, or compare the unnormalized score with the renormalized version from Appendix B; if the top-ranked acquisition points change materially across resolutions, or the renormalized ranking differs from the unnormalized one, the central claim that the score measures epistemic disagreement is not supported.
Extended reading notes
Core claim
The paper's central claim is that for pool-based active learning in regression with normalizing flows, the quantity to acquire on is the disagreement between predictive distributions obtained under different dropout masks, not the width or entropy of any single predictive distribution. The authors define BALSA by replacing the uncertainty function in BALD with a distance between distributions: the grid variant sums KL divergences between each dropout-sampled likelihood vector and their average, and the pair variant sums KL divergences between consecutive sampled distributions. In the experiments, BALSA KL Pairs ranks first on average, followed by BALSA KL Grid, BALD H, and Coreset, while Shannon entropy, standard deviation, and least-confidence baselines rank poorly. For Gaussian neural networks, Coreset becomes the best method, narrowly ahead of BALSA KL Pairs.
Load-bearing premise
The paper's KL-based acquisition scores are computed on unnormalized grid likelihood vectors, and it assumes those scores rank candidate points the same way a properly normalized divergence would, without proving that the ranking is invariant to the missing normalization.
Editorial extensions
If this is right
- For normalizing-flow regressors, common uncertainty heuristics such as Shannon entropy, standard deviation, and least confidence are unreliable acquisition functions and can underperform random sampling.
- Distribution-disagreement acquisition (BALSA KL Pairs, BALSA KL Grid) is, on average, the best way to spend a labeling budget on the four tested datasets.
- Under the MAE metric, geometric coreset sampling becomes the top method, so the best acquisition rule depends on the target error measure.
- At larger query sizes (50 and 200), uncertainty-based methods keep their advantage while clustering methods such as Coreset and TypiClust lose ground, contradicting the usual classification result.
- BALSA applies to any model with dropout layers and a predictive distribution, not only normalizing flows.
Reading between the lines
- The pairwise-disagreement recipe should transfer to other regressors with predictive distributions, such as deep ensembles or heteroscedastic networks, where the same distinction between aleatoric and epistemic spread applies; this is an extension the paper notes only for dropout-compatible training schemes.
- Because the grid version's score is computed on unnormalized likelihood vectors, a natural stress test is to check whether acquisition rankings are stable under grid resolution; the paper does not report this.
- A testable practical extension: combine BALSA's disagreement score with a diversity term for batch acquisition, since the paper shows uncertainty methods degrade at larger query sizes for lack of diversity.
- The paper's ranking method (AUC over repeated runs with Wilcoxon tests) could be applied to other new acquisition functions to see whether the failure of entropy-type heuristics is specific to flows or generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes BALSA (Bayesian Active Learning by Distribution Disagreement), an adaptation of BALD for regression models that output full predictive distributions, such as normalizing flows and Gaussian neural networks. BALSA measures epistemic uncertainty by comparing MC-dropout-sampled predictive distributions via KL divergence (grid or pair variants) or Earth Mover's Distance. The authors evaluate their methods on four regression datasets, two architectures, three query sizes, and multiple baselines, and report that BALSA KL Pairs is the best algorithm on average by NLL-based AUC rankings, followed by BALSA KL Grid, BALD H, and Coreset. They also document that standard heuristics (Shannon entropy, standard deviation, least-confidence) underperform, and they provide ablation studies for dual dropout mode and re-normalization.
Significance. If the claims hold, the paper makes a useful empirical contribution to active learning for regression with deep generative models, an area that is less studied than classification AL. The experimental protocol is solid in structure: 30 repetitions, AUC-based ranking, Wilcoxon-Holm significance tests, CD diagrams, code release, and ablations across datasets and architectures. The idea of using distribution-level disagreement from MC dropout is natural and the pairwise variant avoids the problematic averaging of predictive distributions. However, the central SOTA claim rests on a formal gap: the BALSA KL equations operate on unnormalized grid likelihood vectors, and the normalization ablation is incomplete and internally inconsistent. Additionally, an undefined algorithm ('BALSA Anomaly') appears in the reported comparisons. These issues need to be resolved before the empirical ranking can be fully trusted.
major comments (3)
- [Section 5, Eqs. (3)-(4); Section 7 ablation; Appendix B] The BALSA KL Grid and BALSA KL Pair scores are defined as KL divergences between raw grid likelihood vectors p-tilde, which are not normalized probability distributions. KL divergence is only a valid divergence for normalized distributions; for unnormalized vectors the expression can be negative, scale-dependent, and dominated by the amount of probability mass inside the fixed grid rather than by distributional disagreement. The paper omits normalization based on an ablation, but that ablation (Figure 5) only tests the re-normalized version of BALSA KL Grid, not BALSA KL Pair, which is the top-ranked method. Moreover, the statement in Section 5 that the unnormalized version performs 'comparable to or worse' than the renormalized one is inconsistent with Appendix B's claim that the normalized and unnormalized versions perform 'identical.' Even the 'normalized' version in Appendix B divides only the averaged vector by trapz(p-bar), leaving the individual likelihood vectors unnormalized, so it still does not compute a proper KL divergence. The authors must either fully normalize both arguments with the appropriate trapezoidal weights, provide a rigorous argument that the acquisition ranking is invariant to these normalizations, or report an ablation for BALSA KL Pair. Because BALSA KL Pair is the headline SOTA algorithm, this gap is load-bearing for the paper's central claim.
- [Figure 3 and Section 7] The legend of Figure 3 (and presumably the CD diagram in Figure 2) includes an algorithm called 'BALSA Anomaly' that is never defined in the text, in Table 1, or in any appendix. The paper's benchmark claims to compare a fixed set of algorithms, and including an undefined method makes the results irreproducible and the ranking ambiguous. The authors should either define this algorithm and its hyperparameters or remove it from all figures and analyses.
- [Section 7 and Abstract] The abstract and conclusion claim 'SOTA results for BALSA across 4 different datasets and 2 different architectures,' but Section 7 reports that on MAE-based rankings Coreset is the best algorithm, narrowly beating BALSA KL Pairs. The 'SOTA' claim is therefore metric-dependent, and the paper should qualify it accordingly, ideally reporting both NLL and MAE rankings in the main results and discussing the sensitivity of the conclusions to the evaluation metric. The CRPS-based ranking is said to be identical to NLL but is only described briefly in Appendix E; providing the full CRPS results would strengthen the claim.
minor comments (7)
- [Section 5, Eqs. (7)-(8)] The definitions of BALD sigma and BALD LC are unclear: Eq. (7) subtracts standard deviations of single samples y'_{theta_i}, but the standard deviation of a single sample is not defined; presumably it is the standard deviation of a predictive distribution obtained from a single MC sample, but the notation should be clarified. Eq. (8) subtracts least-confidence values, but the result can be negative and the interpretation as an acquisition score should be justified.
- [Table 1 and Section 6] The 'dual' mode for BALSA EMD and BALSA KL uses a dropout rate marked with '*' and 0.1, but the procedure for selecting this rate is not described in Section 6. The text mentions a fixed evaluation rate of 0.05 and says it is the highest optimal rate, yet the table lists 0.1 for dual modes; this inconsistency should be resolved.
- [Section 5] The sentence 'Since there exist no sound way of averaging iid samples (and their likelihoods) from arbitrary distributions to obtain p-bar|x' contains a grammatical error ('no sound way') and the claim itself is too strong: averaging likelihoods is a well-defined operation, even if the result is not a normalized distribution. Please rephrase to avoid overstatement.
- [Appendix A] The notation in Appendix A is inconsistent with the main text (e.g., 'balsa' in lowercase, and the integral limits are missing). Also, the derivation shows that BALSA KL Grid differs from BALD, but the paper does not discuss whether this difference is beneficial or harmful; a short remark would help readers interpret the relationship.
- [Section 2] The set-difference notation 'Dtrain/L(0)' and 'U(i)/{x_b}' should use the standard backslash symbol to avoid confusion with division.
- [Throughout] The names 'BALSA KL Pairs' and 'BALSA KL Pair' are used interchangeably; please pick one. Similarly, 'BALD Std' appears in Figure 3 while Table 1 uses 'BALD sigma' and 'BALSA EMD dual' vs 'BALSA EMD_dual' should be harmonized.
- [Reproducibility Statement] The reproducibility statement says the setup is identical to reference [21], which is a preprint by the same authors. Please ensure that the evaluation protocol is fully described in the paper or in a stable public benchmark, as relying on a citation to an unpublished preprint makes independent verification difficult.
Circularity Check
No significant circularity: BALSA scores are defined from predictive distributions without fitted targets; the SOTA claim is empirical against external baselines, with only minor self-citation in evaluation protocol.
full rationale
BALSA's acquisition functions (Eqs. 3-5) are explicit functions of dropout-sampled predictive distributions and contain no parameter fitted to the AL objective; the Appendix A derivation shows BALSA KL Grid differs algebraically from BALD, so the method does not reduce to its input by construction. The reported SOTA is an empirical ranking over 30 repetitions on four datasets, compared against external baselines (Coreset, CoreGCN, TypiClust, Random, Std, LC, Entropy) and is code-reproducible. The evaluation protocol is inherited from the authors' own benchmark [21] ('our setup is identical to [21]'), a self-citation, but it sets only the experimental methodology (CD-diagrams, Wilcoxon tests) alongside independent guidelines [10,13]; it does not supply the mathematical content of BALSA, so it is not load-bearing. The paper's own admission that BALSA KL omits renormalization of grid likelihood vectors ('As a vector of averaged likelihoods is no longer normalized... we focus on the un-normalized version'), together with the inconsistent ablation claims in Section 7 and Appendix B, is a correctness risk about whether Eqs. 3-4 are true divergences, but it is not a circular reduction: the score is not defined in terms of the measured test NLL/MAE. Optimizing the MC dropout rate for AL performance is hyperparameter selection rather than a fitted input renamed as a prediction. Overall, no derivation step reduces by definition to its inputs.
Assumptions & free parameters
free parameters (3)
- Grid resolution for BALSA KL and entropy baselines =
200
- MC dropout evaluation rate for BALD and BALSA variants =
0.05
- Number of Monte Carlo dropout samples k =
not reported
assumptions (4)
- domain assumption MC dropout approximates sampling from the Bayesian parameter posterior.
- ad hoc to paper KL divergence between unnormalized grid likelihood vectors is a valid acquisition score.
- domain assumption Disagreement between dropout-sampled predictive distributions separates epistemic from aleatoric uncertainty.
- domain assumption A 200-point grid with trapezoidal integration adequately approximates continuous integrals over the target distribution.
Cite this review
Pith. "Pith review of Bayesian Active Learning By Distribution Disagreement." pith.science (2026). https://pith.science/paper/PRNGADSK
@misc{pith2026250101248,
author = {Pith},
title = {Pith review of: Bayesian Active Learning By Distribution Disagreement},
year = {2026},
howpublished = {\url{https://pith.science/paper/PRNGADSK}},
note = {Machine review of arXiv:2501.01248}
}
read the original abstract
Active Learning (AL) for regression has been systematically under-researched due to the increased difficulty of measuring uncertainty in regression models. Since normalizing flows offer a full predictive distribution instead of a point forecast, they facilitate direct usage of known heuristics for AL like Entropy or Least-Confident sampling. However, we show that most of these heuristics do not work well for normalizing flows in pool-based AL and we need more sophisticated algorithms to distinguish between aleatoric and epistemic uncertainty. In this work we propose BALSA, an adaptation of the BALD algorithm, tailored for regression with normalizing flows. With this work we extend current research on uncertainty quantification with normalizing flows \cite{berry2023normalizing, berry2023escaping} to real world data and pool-based AL with multiple acquisition functions and query sizes. We report SOTA results for BALSA across 4 different datasets and 2 different architectures.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Lucas Berry and David Meger. Escaping the sample trap: Fast and accurate epistemic uncertainty estimation with pairwise-distance estimators. arXiv preprint arXiv:2308.13498, 2023
arXiv 2023
-
[2]
Normalizing flow ensembles for rich aleatoric and epistemic uncertainty model- ing
Lucas Berry and David Meger. Normalizing flow ensembles for rich aleatoric and epistemic uncertainty model- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6806–6814, 2023
work page 2023
-
[3]
Sequential graph convolutional network for active learning
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9583– 9592, 2021
work page 2021
-
[4]
Neural spline flows
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. Advances in neural information processing systems, 32, 2019
2019
- [5]
-
[6]
Deepar: Probabilistic forecasting with autoregressive recur- rent networks
Valentin Flunkert, David Salinas, and Jan Gasthaus. Deepar: Probabilistic forecasting with autoregressive recur- rent networks. CoRR, abs/1704.04110, 2017
arXiv 2017
-
[7]
Deep bayesian active learning with image data
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Interna- tional conference on machine learning, pages 1183–1192. PMLR, 2017
work page 2017
-
[8]
Active learning on a budget: Opposite strategies suit high and low budgets
Guy Hacohen, Avihu Dekel, and Daphna Weinshall. Active learning on a budget: Opposite strategies suit high and low budgets. arXiv preprint arXiv:2202.02794, 2022
arXiv 2022
Show all 21 references
-
[9]
Superconductivty Data
Kam Hamidieh. Superconductivty Data. UCI Machine Learning Repository, 2018. DOI: https://doi.org/10.24432/C53P47
2018 doi
-
[10]
Randomness is the root of all evil: More reliable evaluation of deep active learning
Yilin Ji, Daniel Kaestner, Oliver Wirth, and Christian Wressnegger. Randomness is the root of all evil: More reliable evaluation of deep active learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3943–3952, 2023
2023
-
[11]
Regression tree-based active learning
Ashna Jose, Jo ˜ao Paulo Almeida de Mendonc ¸a, Emilie Devijver, No¨el Jakse, Val´erie Monbet, and Roberta Poloni. Regression tree-based active learning. Data Mining and Knowledge Discovery, 38(2):420–460, 2024
2024
-
[12]
Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019
2019
-
[13]
Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment
Carsten L ¨uth, Till Bungert, Lukas Klein, and Paul Jaeger. Navigating the pitfalls of active learning evaluation: A systematic framework for meaningful performance assessment. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[14]
Hyperparameter tuning mlp’s for probabilistic time series forecasting
Kiran Madhusudhanan, Shayan Jawed, and Lars Schmidt-Thieme. Hyperparameter tuning mlp’s for probabilistic time series forecasting. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 264–275. Springer, 2024
2024
-
[15]
Diamonds Data
Andreas Mueller. Diamonds Data. OpenML, 2019. OpenML ID: 42225
2019
-
[16]
Masked autoregressive flow for density estimation
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017
2017
-
[17]
Ac- tiveglae: A benchmark for deep active learning with transformers
Lukas Rauch, Matthias Aßenmacher, Denis Huseljic, Moritz Wirth, Bernd Bischl, and Bernhard Sick. Ac- tiveglae: A benchmark for deep active learning with transformers. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 55–74. Springer, 2023
2023
-
[18]
Bayesian active learning with fully bayesian gaussian processes
Christoffer Riis, Francisco Antunes, Frederik H ¨uttel, Carlos Lima Azevedo, and Francisco Pereira. Bayesian active learning with fully bayesian gaussian processes. Advances in Neural Information Processing Systems , 35:12141–12153, 2022
2022
-
[19]
Active learning for convolutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489, 2017
2017 arXiv
-
[20]
Parkinsons Telemonitoring
Athanasios Tsanas and Max Little. Parkinsons Telemonitoring. UCI Machine Learning Repository, 2009. DOI: https://doi.org/10.24432/C5ZS3N
2009 doi
-
[21]
A cross-domain benchmark for active learning, 2024
Thorben Werner, Johannes Burchert, Maximilian Stubbemann, and Lars Schmidt-Thieme. A cross-domain benchmark for active learning, 2024. 10 Bayesian Active Learning By Distribution Disagreement A Difference between BALD and BALSA KL BALD(x | ˆp1:K) := KX k=1 H(¯p(y | x)) − H(ˆpk...
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.