{"id":"6a73a37e-eda7-41e7-a082-52082d491ee3","arxiv_id":"2501.01248","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BALSA, a BALD variant that scores points by KL or Earth-Mover disagreement between dropout-sampled predictive distributions, outperforms standard uncertainty and clustering acquisition functions for regression with normalizing flows.","lead":"BALSA adapts the BALD active-learning algorithm to regression models that output full probability distributions, using how much the distributions disagree under Monte Carlo dropout as the signal for which points to label. The authors report that it beats standard uncertainty heuristics and clustering baselines on four real datasets and two model architectures.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BALSA KL Grid/Pair score raw grid likelihood vectors without normalization; if the grid misses probability mass, Eqs. 3-4 are not a divergence and can rank points by grid truncation instead of disagreement.","rationale":"The reader's CONDITIONAL verdict identified the same weakest assumption, and I agree. I focus on the normalization issue because it attacks the definition of the proposed acquisition function itself, not merely the comparison protocol. If the unnormalized KL is only a constant multiple of the true divergence, the concern collapses; if not, the SOTA claim is unsupported. The paper's own statements are inconsistent: Sec. 7 says the normalized version is 'slightly lower' while Appendix B says it performs 'identical.' I also weighed two secondary concerns: 'BALSA Anomaly' appears in every trajectory but is never defined in Section 5 or Table 1, and the fixed dropout rate 0.05 was selected by AL performance across datasets, which risks target leakage. Both are serious reproducibility/fairness issues but do not directly invalidate the BALSA KL construction; they should be fixed in revision. Credit is due for 30 repetitions, Wilcoxon-Holm CD diagrams, a public code repository, four datasets, and two architectures, which is why I do not move the verdict to REJECT. A single controlled normalization experiment would settle whether the central result is robust or an artifact of the unnormalized grid.","tokens_in":11708,"tokens_out":11390,"duration_ms":122419,"concrete_test":"On one dataset (e.g., Diamonds with normalizing flows), fix a trained model and the acquisition pool. Compute BALSA KL Grid and BALSA KL Pair scores under three variants: (i) raw likelihood vectors as printed in Eqs. 3-4; (ii) vectors renormalized by trapz area as in Appendix B; (iii) proper trapezoid-weighted discrete KL with a finer grid (e.g., 1000 points), including estimated mass outside [0,1]. Compare acquisition rankings via Kendall tau and final AL AUC over the same 30 seeds. If (i) and (iii) have Kendall tau below about 0.8 or produce a different winning algorithm, the reported SOTA depends on the unnormalized-grid artifact; if they agree, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equations (3) and (4) define BALSA KL Grid and BALSA KL Pair as sums of KL divergences between raw grid likelihood vectors p^⊣, with normalization explicitly omitted because an ablation 'performed comparable or better.' KL is only a divergence for normalized probability vectors. With grid spacing Δ, the trapezoid approximation to the continuous KL is Δ Σ p(y_n) log(p(y_n)/q(y_n)) only if both densities integrate to one over the sampled interval. The flows are not truncated to [0,1] even though targets are rescaled to [0,1], so the mass inside the grid, S_p = ΔΣ p(y_n), can vary across x and across dropout samples. A properly normalized discrete KL then contains log(S_q/S_p) and 1/S_p terms that the raw sum omits. Consequently the scores in Eqs. 3-4 can be dominated by how much probability mass falls inside the fixed grid rather than by distributional disagreement. The ablation in Sec. 7 and the Appendix B claim that normalized and unnormalized versions are 'identical' do not prove ranking invariance, and the two statements are inconsistent. Because BALSA KL Pair is the headline SOTA algorithm, this missing normalization is the weakest load-bearing point in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BALSA (Bayesian Active Learning by Distribution Disagreement), an adaptation of BALD for regression models that output full predictive distributions, such as normalizing flows and Gaussian neural networks. BALSA measures epistemic uncertainty by comparing MC-dropout-sampled predictive distributions via KL divergence (grid or pair variants) or Earth Mover's Distance. The authors evaluate their methods on four regression datasets, two architectures, three query sizes, and multiple baselines, and report that BALSA KL Pairs is the best algorithm on average by NLL-based AUC rankings, followed by BALSA KL Grid, BALD H, and Coreset. They also document that standard heuristics (Shannon entropy, standard deviation, least-confidence) underperform, and they provide ablation studies for dual dropout mode and re-normalization.","tokens_in":11964,"tokens_out":6119,"duration_ms":59596,"significance":"If the claims hold, the paper makes a useful empirical contribution to active learning for regression with deep generative models, an area that is less studied than classification AL. The experimental protocol is solid in structure: 30 repetitions, AUC-based ranking, Wilcoxon-Holm significance tests, CD diagrams, code release, and ablations across datasets and architectures. The idea of using distribution-level disagreement from MC dropout is natural and the pairwise variant avoids the problematic averaging of predictive distributions. However, the central SOTA claim rests on a formal gap: the BALSA KL equations operate on unnormalized grid likelihood vectors, and the normalization ablation is incomplete and internally inconsistent. Additionally, an undefined algorithm ('BALSA Anomaly') appears in the reported comparisons. These issues need to be resolved before the empirical ranking can be fully trusted.","major_comments":[{"comment":"The BALSA KL Grid and BALSA KL Pair scores are defined as KL divergences between raw grid likelihood vectors p-tilde, which are not normalized probability distributions. KL divergence is only a valid divergence for normalized distributions; for unnormalized vectors the expression can be negative, scale-dependent, and dominated by the amount of probability mass inside the fixed grid rather than by distributional disagreement. The paper omits normalization based on an ablation, but that ablation (Figure 5) only tests the re-normalized version of BALSA KL Grid, not BALSA KL Pair, which is the top-ranked method. Moreover, the statement in Section 5 that the unnormalized version performs 'comparable to or worse' than the renormalized one is inconsistent with Appendix B's claim that the normalized and unnormalized versions perform 'identical.' Even the 'normalized' version in Appendix B divides only the averaged vector by trapz(p-bar), leaving the individual likelihood vectors unnormalized, so it still does not compute a proper KL divergence. The authors must either fully normalize both arguments with the appropriate trapezoidal weights, provide a rigorous argument that the acquisition ranking is invariant to these normalizations, or report an ablation for BALSA KL Pair. Because BALSA KL Pair is the headline SOTA algorithm, this gap is load-bearing for the paper's central claim.","section":"Section 5, Eqs. (3)-(4); Section 7 ablation; Appendix B"},{"comment":"The legend of Figure 3 (and presumably the CD diagram in Figure 2) includes an algorithm called 'BALSA Anomaly' that is never defined in the text, in Table 1, or in any appendix. The paper's benchmark claims to compare a fixed set of algorithms, and including an undefined method makes the results irreproducible and the ranking ambiguous. The authors should either define this algorithm and its hyperparameters or remove it from all figures and analyses.","section":"Figure 3 and Section 7"},{"comment":"The abstract and conclusion claim 'SOTA results for BALSA across 4 different datasets and 2 different architectures,' but Section 7 reports that on MAE-based rankings Coreset is the best algorithm, narrowly beating BALSA KL Pairs. The 'SOTA' claim is therefore metric-dependent, and the paper should qualify it accordingly, ideally reporting both NLL and MAE rankings in the main results and discussing the sensitivity of the conclusions to the evaluation metric. The CRPS-based ranking is said to be identical to NLL but is only described briefly in Appendix E; providing the full CRPS results would strengthen the claim.","section":"Section 7 and Abstract"}],"minor_comments":[{"comment":"The definitions of BALD sigma and BALD LC are unclear: Eq. (7) subtracts standard deviations of single samples y'_{theta_i}, but the standard deviation of a single sample is not defined; presumably it is the standard deviation of a predictive distribution obtained from a single MC sample, but the notation should be clarified. Eq. (8) subtracts least-confidence values, but the result can be negative and the interpretation as an acquisition score should be justified.","section":"Section 5, Eqs. (7)-(8)"},{"comment":"The 'dual' mode for BALSA EMD and BALSA KL uses a dropout rate marked with '*' and 0.1, but the procedure for selecting this rate is not described in Section 6. The text mentions a fixed evaluation rate of 0.05 and says it is the highest optimal rate, yet the table lists 0.1 for dual modes; this inconsistency should be resolved.","section":"Table 1 and Section 6"},{"comment":"The sentence 'Since there exist no sound way of averaging iid samples (and their likelihoods) from arbitrary distributions to obtain p-bar|x' contains a grammatical error ('no sound way') and the claim itself is too strong: averaging likelihoods is a well-defined operation, even if the result is not a normalized distribution. Please rephrase to avoid overstatement.","section":"Section 5"},{"comment":"The notation in Appendix A is inconsistent with the main text (e.g., 'balsa' in lowercase, and the integral limits are missing). Also, the derivation shows that BALSA KL Grid differs from BALD, but the paper does not discuss whether this difference is beneficial or harmful; a short remark would help readers interpret the relationship.","section":"Appendix A"},{"comment":"The set-difference notation 'Dtrain/L(0)' and 'U(i)/{x_b}' should use the standard backslash symbol to avoid confusion with division.","section":"Section 2"},{"comment":"The names 'BALSA KL Pairs' and 'BALSA KL Pair' are used interchangeably; please pick one. Similarly, 'BALD Std' appears in Figure 3 while Table 1 uses 'BALD sigma' and 'BALSA EMD dual' vs 'BALSA EMD_dual' should be harmonized.","section":"Throughout"},{"comment":"The reproducibility statement says the setup is identical to reference [21], which is a preprint by the same authors. Please ensure that the evaluation protocol is fully described in the paper or in a stable public benchmark, as relying on a citation to an unpublished preprint makes independent verification difficult.","section":"Reproducibility Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical protocol is a strength, but the central claim depends on an unnormalized KL-based acquisition score whose validity is not established for the top-performing BALSA KL Pair variant. The inconsistency between Section 5 and Appendix B regarding the normalization ablation should be addressed head-on. I also note that the evaluation framework cites the authors' own benchmark paper [21] as the standard; this is not a fatal issue, but the editor may wish to consider whether the benchmark is sufficiently established in the community to serve as the sole protocol reference. The undefined 'BALSA Anomaly' algorithm is a more straightforward reproducibility problem that should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about active learning for regression with full predictive distributions. The paper's real contribution is a set of disagreement acquisition functions—KL Grid, KL Pair, and EMD—that compare dropout-sampled predictive distributions from normalizing flows, and it tests them on four real datasets with two architectures, 30 repetitions, and proper significance testing. That is more than most AL papers do, and the code is public. The empirical result is interesting: pairwise KL disagreement wins on NLL for flows, while Coreset wins on MAE. The paper is honest about differences.\n\nThe soft spot is the normalization issue. Equations (3) and (4) write KL between raw grid likelihood vectors that are not normalized, and KL is only a divergence for probability vectors. If the grid misses mass—and the flows are not truncated—the score can be dominated by how much mass falls in the grid rather than by disagreement. The paper says in Sec 7 that unnormalized performed 'comparable or worse,' and Appendix B says 'identical,' which is inconsistent. This matters because BALSA KL Pair is the headline algorithm. The fix is not hard: normalize the vectors or prove ranking invariance, and report the grid resolution effect. Without that, the SOTA claim is not fully backed.\n\nSmaller issues: 'BALSA Anomaly' appears in the legends but is never defined—unacceptable for reproducibility. The MC dropout rate for evaluation was tuned to AL performance across datasets; that is a potential leakage that should be disclosed more prominently.\n\nThe central idea—using distribution disagreement rather than spread—is sound and likely useful to practitioners. But as written, the key formulas are not well-founded, and the missing definition of BALSA Anomaly makes part of the comparison unreproducible. I would send this to peer review with a request for major revision, not desk-reject it. A good referee can push the authors to fix the normalization and define every reported algorithm, and the empirical benchmark is worth having.","headline":"BALSA gives AL practitioners a useful new disagreement heuristic for normalizing flows, but the unnormalized KL scores need a theoretical fix before the SOTA claim is trustworthy.","tokens_in":12476,"tokens_out":1926,"would_cite":true,"duration_ms":18756,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that for active learning in regression with normalizing flows, the best acquisition signal is disagreement among MC-dropout predictive distributions, not the spread of a single distribution.","keywords":["active learning","regression","normalizing flows","BALD","epistemic uncertainty","Monte-Carlo dropout","BALSA","acquisition function"],"falsifier":"Run BALSA KL Grid on the same unlabeled pool with grid resolutions of 50, 200, and 1000, or compare the unnormalized score with the renormalized version from Appendix B; if the top-ranked acquisition points change materially across resolutions, or the renormalized ranking differs from the unnormalized one, the central claim that the score measures epistemic disagreement is not supported.","tokens_in":11481,"feed_emoji":"🎯","tokens_out":4876,"duration_ms":44402,"temperature":0.7,"pith_summary":"Normalizing flows output a full predictive distribution, which makes it tempting to use familiar active-learning heuristics such as Shannon entropy, standard deviation, or least-confidence sampling to choose which labels to request. This paper argues that those heuristics fail for regression because they cannot separate aleatoric noise from epistemic model uncertainty, and that only the latter can be reduced by labeling more data. It proposes BALSA, a variant of the BALD algorithm that treats disagreement among Monte-Carlo-dropout predictive distributions as the acquisition signal, and reports that BALSA attains state-of-the-art results on four datasets and two architectures. If the claim holds, practitioners with flow-based regression models have a simple recipe for spending their labeling budget.","feed_headline":"Distribution disagreement wins active learning for flow regression","feed_subtitle":"Pairwise KL between dropout-sampled predictive distributions beats entropy, std, and least confidence on four benchmarks.","key_machinery":"The load-bearing object is the BALSA acquisition score, a sum of distributional distances over k Monte-Carlo-dropout parameter samples instead of BALD's entropy subtraction. Two distance computations carry the method: the grid version discretizes the normalized target range into 200 bins, averages the resulting likelihood vectors to form $\\bar{p}|x$, and scores $\\sum_i \\mathrm{KL}(\\hat{p}_{\\theta_i}|x,\\bar{p}|x)$; the pair version avoids averaging and scores $\\sum_i \\mathrm{KL}(\\hat{p}_{\\theta_i}|x,\\hat{p}_{\\theta_{i+1}}|x)$. An EMD variant computes pairwise Earth mover distances over iid samples. The paper defines these scores on unnormalized likelihood vectors, arguing from an ablation that renormalization changes performance only slightly.","core_discovery":"The paper's central claim is that for pool-based active learning in regression with normalizing flows, the quantity to acquire on is the disagreement between predictive distributions obtained under different dropout masks, not the width or entropy of any single predictive distribution. The authors define BALSA by replacing the uncertainty function in BALD with a distance between distributions: the grid variant sums KL divergences between each dropout-sampled likelihood vector and their average, and the pair variant sums KL divergences between consecutive sampled distributions. In the experiments, BALSA KL Pairs ranks first on average, followed by BALSA KL Grid, BALD H, and Coreset, while Shannon entropy, standard deviation, and least-confidence baselines rank poorly. For Gaussian neural networks, Coreset becomes the best method, narrowly ahead of BALSA KL Pairs.","pith_inferences":["The pairwise-disagreement recipe should transfer to other regressors with predictive distributions, such as deep ensembles or heteroscedastic networks, where the same distinction between aleatoric and epistemic spread applies; this is an extension the paper notes only for dropout-compatible training schemes.","Because the grid version's score is computed on unnormalized likelihood vectors, a natural stress test is to check whether acquisition rankings are stable under grid resolution; the paper does not report this.","A testable practical extension: combine BALSA's disagreement score with a diversity term for batch acquisition, since the paper shows uncertainty methods degrade at larger query sizes for lack of diversity.","The paper's ranking method (AUC over repeated runs with Wilcoxon tests) could be applied to other new acquisition functions to see whether the failure of entropy-type heuristics is specific to flows or generalizes."],"forward_implications":["For normalizing-flow regressors, common uncertainty heuristics such as Shannon entropy, standard deviation, and least confidence are unreliable acquisition functions and can underperform random sampling.","Distribution-disagreement acquisition (BALSA KL Pairs, BALSA KL Grid) is, on average, the best way to spend a labeling budget on the four tested datasets.","Under the MAE metric, geometric coreset sampling becomes the top method, so the best acquisition rule depends on the target error measure.","At larger query sizes (50 and 200), uncertainty-based methods keep their advantage while clustering methods such as Coreset and TypiClust lose ground, contradicting the usual classification result.","BALSA applies to any model with dropout layers and a predictive distribution, not only normalizing flows."],"supporting_citations":[{"why":"Supplies the normalizing-flow-ensemble uncertainty setup that BALSA extends to pool-based active learning.","marker":"[2]"},{"why":"Supplies the Monte-Carlo-dropout approximation of flow ensembles that BALSA's sampling builds on.","marker":"[1]"},{"why":"Supplies BALD, the algorithm BALSA adapts from classification to predictive distributions.","marker":"[7]"},{"why":"Supplies the neural spline flow architecture used in all experiments.","marker":"[4]"},{"why":"Supplies the repeated-run evaluation protocol that the experiments follow.","marker":"[10]"},{"why":"Supplies the CD-diagram ranking protocol and experimental setup.","marker":"[21]"},{"why":"Supplies the coreset geometric baseline that is the main competitor to BALSA.","marker":"[19]"}],"fun_headline_variants":["Distribution disagreement beats entropy for flow regression","BALSA: active learning via KL divergence between dropout samples","Dropout disagreement outperforms uncertainty heuristics in regression","For flow regression, query by disagreement, not entropy","BALSA: distribution divergence for active learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's KL-based acquisition scores are computed on unnormalized grid likelihood vectors, and it assumes those scores rank candidate points the same way a properly normalized divergence would, without proving that the ranking is invariant to the missing normalization.","fun_headline_variants_meta":{"raw":{"variants":["Distribution disagreement beats entropy for flow regression","BALSA: active learning via KL divergence between dropout samples","Dropout disagreement outperforms uncertainty heuristics in regression","For flow regression, query by disagreement, not entropy","BALSA: distribution divergence for active learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000429,"raw_usage":{"total_tokens":2148,"prompt_tokens":855,"completion_tokens":1293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1219}},"tokens_in":471,"tokens_out":1293,"duration_ms":8960,"temperature":1.0,"reasoning_tokens":1219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:31:37.083112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run BALSA KL Grid on the same unlabeled pool with grid resolutions of 50, 200, and 1000, or compare the unnormalized score with the renormalized version from Appendix B; if the top-ranked acquisition points change materially across resolutions, or the renormalized ranking differs from the unnormalized one, the central claim that the score measures epistemic disagreement is not supported.","supporting_citations":[{"cited_title":"Normalizing flow ensembles for rich aleatoric and epistemic uncertainty model- ing","cited_arxiv_id":null,"evidence_quote":"Supplies the normalizing-flow-ensemble uncertainty setup that BALSA extends to pool-based active learning."},{"cited_title":"Deep bayesian active learning with image data","cited_arxiv_id":null,"evidence_quote":"Supplies BALD, the algorithm BALSA adapts from classification to predictive distributions."},{"cited_title":"Randomness is the root of all evil: More reliable evaluation of deep active learning","cited_arxiv_id":null,"evidence_quote":"Supplies the repeated-run evaluation protocol that the experiments follow."},{"cited_title":"A cross-domain benchmark for active learning, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the CD-diagram ranking protocol and experimental setup."}],"review_version":1}