{"id":"db634410-111d-4da1-ad27-4126653f2314","arxiv_id":"1909.00218","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Epistemic uncertainty, the reducible part of predictive uncertainty, outperforms total and aleatoric uncertainty as a query-selection score in active learning.","lead":"This paper proposes choosing the next example to label in active learning by the amount of model uncertainty that is due to lack of knowledge (epistemic), rather than by total prediction uncertainty. Tests on eight datasets suggest this selection rule helps for local, non-parametric classifiers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's u_e measures disagreement within a fixed hypothesis space H, not reducible uncertainty; under model misspecification it can vanish exactly where labels would most reduce error, as Section 5.2 concedes for logistic regression.","rationale":"The reader's weakest assumption is that the normalized-likelihood construction inherited from [19] truly separates reducible model uncertainty from irreducible noise. My stress-test identifies the same assumption as the most load-bearing point: u_e is a likelihood-weighted disagreement measure over a fixed H, not a direct measure of reducible error. The paper's own Section 5.2 observation that epistemic uncertainty 'disappears quite quickly' for logistic regression is an admission that the measure is sensitive to the inductive bias of H. This matters because the central claim is general: Section 6 states that the epistemic part is the better criterion for selecting instances. The empirical evidence for that claim is strongest for flexible local learners and weakest for a parametric learner, which is consistent with the concern. The proposed synthetic test would settle whether u_e correlates with actual error reduction under misspecification; if it does not, the central claim must be restricted to flexible model classes or supplemented with a model-selection mechanism. Since this is an addressable empirical and conceptual issue rather than a demonstrated contradiction, the existing CONDITIONAL verdict remains appropriate. No adversarial intent is implied; the paper is transparent about the limitation, but the conclusion currently overstates the scope of the evidence.","tokens_in":9897,"tokens_out":5377,"duration_ms":55888,"concrete_test":"Generate a binary classification dataset with a known nonlinear ground-truth boundary (e.g., XOR or a radial pattern) and fit logistic regression. For a large unlabeled pool, compute u_e(x) (Eq. 9) and, using the true generative model, estimate the oracle expected error reduction EER(x) of querying x by simulating label acquisition and retraining. Compute the rank correlation between u_e(x) and EER(x); also run the active-learning loop comparing max-u_e selection with max-EER selection and with standard uncertainty sampling. If u_e is near zero for the highest-EER points, or max-u_e fails to beat standard uncertainty sampling in this misspecified setting, then the claim that u_e measures reducible uncertainty fails in exactly the regime Section 5.2 flags.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that u_e captures reducible uncertainty and is therefore the better query criterion. But u_e (Eqs. 9-10) is built from normalized likelihoods pi_H(h) over a fixed H and support degrees (Eqs. 7-8). It measures disagreement among plausible models inside H, not reducibility with respect to the true data-generating process. If H is misspecified, plausible models can agree at a point even though labelling it would substantially reduce error; u_e is then near zero precisely where the largest error reduction is available. The paper itself concedes this in Section 5.2: logistic regression 'comes with a very strong learning bias in the form of a linearity assumption. Therefore, the epistemic (or model) uncertainty disappears quite quickly.' This is not a peripheral implementation detail: for parametric learners with restrictive inductive bias, the query rule (11) can degenerate toward standard or random sampling, as Figure 4 shows. The experiments cover flexible model classes (Parzen windows, decision trees) or datasets roughly matching the linear assumption, but no experiment checks whether u_e actually tracks expected error reduction under misspecification. Without this check, the Section 6 conclusion that 'the epistemic part is the better criterion' is not established beyond the chosen instantiations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes epistemic uncertainty sampling, a query strategy for active learning in binary classification. Building on the normalized-likelihood framework of Senge et al. (2014), it defines epistemic uncertainty u_e(x) (Eq. 9) and aleatoric uncertainty u_a(x) (Eq. 10) as functions of support degrees of the two classes, and suggests querying the instance maximizing u_e (Eq. 11). Closed-form support degrees are derived for Parzen window classifiers and decision trees (Section 4.1), while logistic regression is handled by discretizing the parameter constraint alpha (Section 4.2). Experiments on eight UCI data sets compare epistemic, aleatoric, standard uncertainty, and random sampling; epistemic sampling performs best for the two local learners and comparably to standard uncertainty for logistic regression.","tokens_in":10195,"tokens_out":3654,"duration_ms":33670,"significance":"If the central conjecture is valid, the paper makes a useful conceptual contribution by separating reducible from irreducible uncertainty in active learning and providing a concrete, algorithm-agnostic measure. The derivations for Parzen windows and decision trees are closed-form and clear. However, the claim that u_e tracks reducible uncertainty is asserted rather than proven, and the empirical evidence lacks error bars and significance tests. The paper's own logistic-regression results show the measure can collapse under model misspecification, so the generality of the conclusion is not established. The work is nevertheless a reasonable starting point for further study.","major_comments":[{"comment":"The central claim that u_e measures reducible uncertainty is not established. Equations (9)-(10) define u_e as the minimum of support degrees obtained from normalized likelihoods over a fixed hypothesis space H. This measures internal disagreement among plausible models, not reducibility with respect to the true data-generating process. Under model misspecification, plausible models can agree at a point even though labeling it would substantially reduce error, and u_e can vanish exactly where error reduction is most needed. The authors concede this behavior for logistic regression in Section 5.2, where u_e 'disappears quite quickly' under the linearity assumption. Since the conclusion in Section 6 states that 'the epistemic part is the better criterion' without qualification, this is a load-bearing gap. The authors should either state precise conditions on H under which arg max u_e corresponds to maximal expected error reduction, or add an experiment that directly measures whether u_e tracks error reduction under misspecification.","section":"Section 3, Eqs. (9)-(10)"},{"comment":"The global optimum of the non-concave problem (19) is approximated by scanning 50 discretized alpha values and taking the best restricted maximum, without any certificate of global optimality. Because u_e is used as the query score, a poor approximation could change instance rankings and thereby affect the experimental conclusions. The authors should provide evidence that the discretization is adequate, for example by comparing against finer grids or multiple initializations, or by using a certified global optimization technique on a subset of instances.","section":"Section 4.2, Eq. (23)"},{"comment":"The experimental section reports only averaged accuracies over 10-fold cross-validation, without standard deviations, confidence intervals, or significance tests. The claimed advantage of epistemic over standard uncertainty sampling is described as 'not huge' but 'visible and consistent' (Section 5.1). Since the pool is also randomly subsampled at 10% in each iteration, the results may have substantial variability. Given that the paper's main evidence is empirical, the absence of any statistical assessment makes it difficult to evaluate whether the observed differences are reliable. Paired tests (e.g., Wilcoxon signed-rank across datasets) and error bars or per-fold spreads should be reported.","section":"Section 5, Figures 2-4"}],"minor_comments":[{"comment":"The notation is confusing: 'p' denotes both the number of positive instances and the probability ph(1|x). Renaming the count variable (e.g., to 'k' or 'n_+') would improve readability.","section":"Section 4.1, after Eq. (14)"},{"comment":"There appears to be a typo: the update for pi(0|x) uses '1 - 2*alpha_p' but should likely use '1 - 2*alpha_n' to match the negative-class support in Eq. (24).","section":"Algorithm 3, line 10"},{"comment":"Algorithm 2 sets epsilon to the average K-th nearest neighbor distance over training instances, but the text states this makes the average number of neighbors 'just K'. This is only approximately true, and the approximation is not quantified; a brief clarification would help.","section":"Section 5.1, Algorithm 2"},{"comment":"The color map has no colorbar or scale, so the reader cannot quantitatively interpret the uncertainty values shown. Adding a colorbar would improve the figure's informativeness.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' earlier framework [19] for the definition of epistemic and aleatoric uncertainty, but this appears to be a legitimate foundation rather than an inappropriate self-citation. The main concern is whether the central claim is sufficiently supported given the misspecification issue and the absence of statistical rigor in the experiments. The paper is a conference-level contribution that could be acceptable after revision addressing these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a competent extension of a known distinction to active learning, not a new conceptual breakthrough. The authors take the epistemic/aleatoric decomposition from their own earlier framework [19], define an epistemic uncertainty measure u_e, and propose querying the pool element with maximal u_e. For Parzen windows and decision trees the computations are closed-form and the experiments are clean enough to show a consistent advantage over entropy-based uncertainty sampling. That is a genuine, if modest, contribution.\n\nWhat I like: the instantiations are worked out carefully, the aleatoric control is a nice touch, and the authors do not oversell the results in the main body. They include the logistic regression case where the advantage disappears, and in Section 5.2 they attribute it to the linearity bias of the model. That is honest reporting.\n\nThe soft spots are real but not fatal. The biggest issue is the conceptual gap between 'epistemic uncertainty' as disagreement within the chosen hypothesis space H and 'reducible uncertainty' with respect to the true data-generating process. Under misspecification, u_e can be near zero exactly where a label would reduce error the most. The paper concedes this implicitly in the logistic regression section; I would have liked an explicit test of whether u_e tracks error reduction on the datasets, rather than just an accuracy comparison. A second issue is the missing comparison to BALD, which is the established method for querying by epistemic uncertainty in Bayesian active learning. That omission weakens the novelty claim. Third, the logistic regression support calculation is a discretized search over 50 alpha values on a non-concave objective; there is no certificate that the global optimum is found. That is a minor implementation concern but should be stated as an approximation.\n\nGiven the paper's scope, none of these should block publication. The central claim is supported for the flexible model classes where it is most plausible, and the experiments, while lacking error bars, are consistent across eight datasets. I would send it to a serious referee, asking for error bars, a BALD comparison, and a short discussion of what u_e measures under misspecification.\n\nFor a reader in active learning or uncertainty quantification, this is worth a look. I don't see myself citing it unless I work on uncertainty sampling specifically, but it is a reasonable paper.","headline":"A competent and honest application of a known epistemic/aleatoric split to uncertainty sampling, with a real limitation: the key measure can vanish under model misspecification.","tokens_in":10676,"tokens_out":2670,"would_cite":false,"duration_ms":26056,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Active learners should query instances with high epistemic uncertainty—ignorance about the model—not high aleatoric uncertainty from noise.","keywords":["active learning","uncertainty sampling","epistemic uncertainty","aleatoric uncertainty","likelihood-based plausibility","Parzen window classifier","decision tree","logistic regression"],"falsifier":"Run the comparison on a synthetic binary problem where one region has clean labels and another has labels flipped with probability $p$, for several values of $p$. If instances with high $u_e$ are not the ones whose addition reduces test error fastest, or if the accuracy advantage of the $u_e$ rule fails to grow with $p$, the central conjecture is false.","tokens_in":9716,"feed_emoji":"🎯","tokens_out":12101,"duration_ms":99044,"temperature":0.7,"pith_summary":"Active learning is about spending labels where they will teach the most. This paper argues that standard uncertainty sampling conflates two different things: epistemic uncertainty, which comes from not knowing the right model and can be reduced by more data, and aleatoric uncertainty, which comes from inherent randomness in the labels and cannot. The paper's conjecture is that, in uncertainty sampling, an instance's usefulness is better reflected by its epistemic component, and it instantiates the query rule $x^* = \\arg\\max_{x\\in U} u_e(x)$ using likelihood-based plausibility scores. Experiments with Parzen-window classifiers, decision trees, and logistic regression on eight datasets support the conjecture, with epistemic uncertainty sampling performing best and aleatoric-only sampling worst. If the conjecture holds, active learners can become more label-efficient simply by asking whether the uncertainty they see is reducible before querying.","feed_headline":"Query model ignorance, not noise, in active learning","feed_subtitle":"Separating model ignorance from noise makes label queries more informative than entropy does.","key_machinery":"The machinery is the normalized likelihood $\\pi_H(h)=L(h)/L(h_{ml})$ of a model $h$ given the training data, combined with the two degrees of support $\\pi(1\\mid x)=\\sup_{h\\in H}\\min[\\pi_H(h), 2h(x)-1]$ and $\\pi(0\\mid x)=\\sup_{h\\in H}\\min[\\pi_H(h), 1-2h(x)]$. These support degrees turn training data into a plausibility ranking over hypotheses and then split total uncertainty into $u_e$ and $u_a$. Everything else is computation: for Parzen windows and decision-tree leaves, constant hypotheses per region reduce the supports to one-dimensional scalar maximizations solved with Brent's method; for logistic regression, the supports are found by maximizing a strictly concave regularized log-likelihood over parallel hyperplanes, with the level $\\alpha$ discretized and solved by Newton-CG or sequential least squares.","core_discovery":"The paper's central claim is that in uncertainty sampling the utility of an unlabeled instance is better captured by its epistemic uncertainty than by its aleatoric uncertainty, so the learner should query $x^* = \\arg\\max_{x\\in U} u_e(x)$ instead of maximizing predictive entropy. With $u_e(x)=\\min[\\pi(1\\mid x),\\pi(0\\mid x)]$ and $u_a(x)=1-\\max[\\pi(1\\mid x),\\pi(0\\mid x)]$, the first is high when plausible models disagree strongly about which class is more likely, while the second is high when every plausible model puts the class probability near one half. The paper derives these degrees of support from normalized likelihoods, instantiates the query rule for local learners and logistic regression, and presents experiments in which epistemic uncertainty sampling achieves the best accuracy and aleatoric sampling the worst, with standard entropy and random sampling in between. The paper interprets the smaller gains under logistic regression as evidence that the method's advantage is tied to how long the hypothesis space retains genuine model uncertainty.","pith_inferences":["A natural stress test not run in the paper: on a synthetic binary task with known label-flip noise in one region, the gap between $u_e$ sampling and $u_a$ sampling should widen as the flip rate rises, because the aleatoric rule will systematically drain the budget in the noisy region.","The logistic-regression finding suggests a general dependency: the amount of epistemic uncertainty an instance carries depends on the expressiveness of the hypothesis space, so kernel-based or deep models, which keep model uncertainty alive longer, would plausibly show larger gains from epistemic sampling than linear models do.","The same $u_e$ measure could serve as a stopping rule: once the pool-wide epistemic uncertainty is low, the expected benefit of further queries is small, which would let an active learner stop early and save labeling cost.","If the conjecture is correct, the success of entropy-based uncertainty sampling in earlier work can be reinterpreted as approximating epistemic uncertainty; explicit separation should matter most when label noise is heterogeneous across the input space."],"forward_implications":["Replacing entropy or least-confidence with $u_e(x)$ as the utility score should give an active learner comparable or better accuracy at the same label budget, since the experiments show the epistemic rule dominating standard uncertainty sampling.","Aleatoric-only sampling is a cautionary baseline: querying the instances where all plausible models agree on a near-random class probability spends labels where the label noise itself is the problem.","The difference between epistemic and aleatoric scores is a practical diagnostic: a large gap means the learner's uncertainty is mostly reducible, while a small gap means further labels are unlikely to sharpen the model.","For algorithms with strong inductive bias, such as linear logistic regression, epistemic uncertainty can be exhausted quickly, so epistemic sampling converges to standard uncertainty sampling; the paper's generic construction means the rule can be attached to any hypothesis space for which the supports can be computed."],"supporting_citations":[{"why":"Supplies the normalized-likelihood formalization and the definitions of epistemic and aleatoric uncertainty that the query rule is built on.","marker":"[19]"},{"why":"Defines uncertainty sampling and the standard utility measures such as entropy that the paper proposes to replace.","marker":"[20]"},{"why":"Motivates the need to look beyond total uncertainty in sampling, the starting point of the paper.","marker":"[21]"},{"why":"Provides the relative-likelihood foundation used to turn likelihoods into plausibility degrees.","marker":"[2]"},{"why":"Justifies upper probabilities based only on the likelihood function, the basis for the support degrees.","marker":"[23]"},{"why":"Supplies the Parzen-window active-learning setting used as the first instantiation.","marker":"[4]"},{"why":"Establishes strict concavity of the regularized logistic log-likelihood, which makes the support optimization tractable.","marker":"[17]"}],"fun_headline_variants":["Separate model ignorance from noise for smarter queries","Query epistemic uncertainty, not aleatoric noise","Active learning: target reducible uncertainty","Ask where models disagree, not where data is noisy","Epistemic over aleatoric: better active learning queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the numbers it computes from the training data truly separate 'uncertainty that more data can fix' from 'uncertainty that is just noise', rather than merely reflecting the arbitrary choice of model family.","fun_headline_variants_meta":{"raw":{"variants":["Separate model ignorance from noise for smarter queries","Query epistemic uncertainty, not aleatoric noise","Active learning: target reducible uncertainty","Ask where models disagree, not where data is noisy","Epistemic over aleatoric: better active learning queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1218,"prompt_tokens":917,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":533,"tokens_out":301,"duration_ms":3666,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:57:34.858838+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the comparison on a synthetic binary problem where one region has clean labels and another has labels flipped with probability $p$, for several values of $p$. If instances with high $u_e$ are not the ones whose addition reduces test error fastest, or if the accuracy advantage of the $u_e$ rule fails to grow with $p$, the central conjecture is false.","supporting_citations":[{"cited_title":"Upper probabilities based only on the likelihood function","cited_arxiv_id":null,"evidence_quote":"Justifies upper probabilities based only on the likelihood function, the basis for the support degrees."},{"cited_title":"Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty","cited_arxiv_id":null,"evidence_quote":"Supplies the normalized-likelihood formalization and the definitions of epistemic and aleatoric uncertainty that the query rule is built on."},{"cited_title":"Active learning literature survey","cited_arxiv_id":null,"evidence_quote":"Defines uncertainty sampling and the standard utility measures such as entropy that the paper proposes to replace."},{"cited_title":"Evidence-based uncertainty sampling for active learning","cited_arxiv_id":null,"evidence_quote":"Motivates the need to look beyond total uncertainty in sampling, the starting point of the paper."},{"cited_title":"On the foundations of statistical inference","cited_arxiv_id":null,"evidence_quote":"Provides the relative-likelihood foundation used to turn likelihoods into plausibility degrees."},{"cited_title":"Active learning for parzen window classifier","cited_arxiv_id":null,"evidence_quote":"Supplies the Parzen-window active-learning setting used as the first instantiation."},{"cited_title":"Regularized logistic regression is strictly convex","cited_arxiv_id":null,"evidence_quote":"Establishes strict concavity of the regularized logistic log-likelihood, which makes the support optimization tractable."}],"review_version":1}