{"id":"55ec96ec-cc46-46b3-a0ac-a826f835347f","arxiv_id":"2507.06061","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new Bayesian quantifier, PQ, produces tighter and well-calibrated prediction intervals for class prevalence estimates, beating existing methods across simulated and real datasets.","lead":"This paper introduces Precise Quantifier (PQ), a Bayesian method that estimates the share of a class in a new unlabeled dataset with narrower uncertainty intervals than earlier quantifiers while keeping the intervals well calibrated. The authors test it on simulations and three review datasets and identify the classifier's strength and the sizes of labeled and unlabeled samples as the main drivers of precision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PQ's coverage is only validated under the weak prior shift assumption, which all experiments satisfy by construction; the paper provides no evidence on how coverage behaves when this key assumption is violated.","rationale":"No significant objection to the internal mathematics was found: the posterior predictive derivation in Eqs. (10)–(12) is coherent, the MCMC procedure is standard, and the experiments are reasonably extensive. The most load-bearing concern is external validity: the method's coverage guarantee is conditional on the weak prior shift assumption, and the paper never tests what happens when it fails. All simulated data respects the assumption by construction, and the real-data protocol draws validation and test from the same source, so the assumption is satisfied trivially. This is not a criticism of the method's logic but of the evidence for its practical claim. A single covariate-shift experiment with shifted class-conditional means would directly settle whether coverage degrades. The reader's weakest assumption identifies the same issue, so I agree. The appropriate verdict remains conditional: the paper should add such a robustness check (or clearly scope the claim) before the central claim is fully supported.","tokens_in":12305,"tokens_out":13668,"duration_ms":159258,"concrete_test":"Run a controlled covariate-shift experiment using the simulated setup of §3.1: keep the validation set with positive-class mean 2.5 and negative mean 0 (unit variance), but generate the 1010 test sets with the positive-class mean shifted to 1.5 (unit variance), so P_T(f(X)|Y=1) differs from P_V(f(X)|Y=1) and the weak prior shift assumption is violated. Apply PQ with the default Nbin=4 and the same MCMC sampling, and compute the coverage of the central 50% PIs across the 1010 test sets. If coverage falls materially below the nominal 50% (e.g., below 40%), the central claim of well-calibrated coverage is not robust to violation of the key assumption; if coverage remains near 50%, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that PQ yields shorter intervals with well-calibrated coverage—rests on the weak prior probability shift assumption introduced in Section 2: P_V(f(X)|Y) = P_T(f(X)|Y). This equality is used directly in Eq. (3) and in the multinomial binning model (Eqs. (1)–(2)); if it fails, the likelihood is misspecified and the posterior predictive intervals from Eqs. (10)–(12) need not attain nominal coverage. Yet no experiment in Section 4 or elsewhere tests robustness to violation of this assumption. In the simulated data, the class-conditional feature distributions are identical by construction for validation and test; in the review datasets, validation and test are both drawn from the same QuaPy test split, so the weak prior shift is satisfied by design. The discussion mentions general dataset shift but stops short of any stress test. Since the paper is proposing a method for real-world prevalence estimation, where covariate shift is common, the absence of any evaluation under a violated weak prior shift leaves the headline result conditional on an unverified assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes Precise Quantifier (PQ), a Bayesian aggregative method for binary prevalence estimation. PQ discretizes classifier scores into Nbin bins, models positive/negative bin counts with multinomial likelihoods on validation and test data (Eqs. 1–2), places a Bernoulli prior on individual test labels, and obtains a posterior predictive distribution for the test prevalence θ via MCMC (Eqs. 10–12). The paper compares PQ to BayesianCC, EMQ, PACC, and HDy on two simulated classification problems and three review datasets (hp, kindle, imdb), measuring PI/CI length and coverage. The authors report that PQ yields shorter intervals than the other methods while achieving nominal coverage, and they analyze how classifier strength, validation size, and test size affect precision.","tokens_in":12520,"tokens_out":8069,"duration_ms":85661,"significance":"If the results are reproducible, PQ is a useful addition to quantification uncertainty: it is computationally efficient (only 2Nbin+1 continuous parameters), is agnostic to classifier internals, and the paper provides open-source code. The simulation and real-data protocols are extensive, and the binning sensitivity analysis (Figure 3) is a good practice. The main claims, however, are established only under the weak prior probability shift assumption and only for 50% coverage, which tempers the significance until these gaps are addressed. The derivation itself is a standard Bayesian posterior predictive and I do not see a circularity problem.","major_comments":[{"comment":"The likelihood and the multinomial model rely on the weak prior probability shift assumption P_V(f(X)|Y)=P_T(f(X)|Y), which is never violated in any experiment: the simulated data use identical class-conditional distributions for validation and test by construction, and the review datasets are drawn from the same QuaPy test split (Section 3.2). The paper's headline claim that PQ's coverage is well-calibrated is therefore conditional on an assumption that is always satisfied by design. Please add at least one experiment in which the test class-conditional score distributions differ from validation (e.g., a covariate shift or label-conditioned score shift) and report the resulting coverage; if the method degrades, temper the general claim and state the assumption explicitly in the abstract.","section":"Section 2, Eqs. (1)–(3)"},{"comment":"All coverage results are reported only for central 50% prediction intervals. The text in Section 2 states that more posterior samples are needed for 95% intervals, but no 95% coverage evaluation is presented. Because 'well-calibrated coverage' is a central claim, the paper should demonstrate at least one other confidence level (e.g., 95%) to rule out calibration being specific to the 50% level.","section":"Section 4, Figures 1–2"},{"comment":"The claim that PQ is 'more precise than existing quantifiers' is too strong. Section 4 concedes that EMQ can be as precise or more precise than PQ on small test sets (Figure 1C), and EMQ is excluded from the ranking only because its coverage is unreliable. Please qualify the claim, e.g., 'more precise than existing quantifiers with well-calibrated coverage,' or modify the abstract and introduction to match the reported results. This is not a mathematical error but it affects the paper's leading claim.","section":"Abstract"}],"minor_comments":[{"comment":"Typo: 'treatmant' should be 'treatment'.","section":"Section 1"},{"comment":"Typo: 'datasests' should be 'datasets'.","section":"Table 1 caption"},{"comment":"Typo: 'Not suprising' should be 'Not surprisingly'.","section":"Section 4"},{"comment":"The notation 'prior prevalence θpr' is confusing because θpr is estimated from the data (Eq. 8) rather than fixed a priori; consider renaming it 'label prevalence hyperparameter' or 'mixing proportion' to avoid implying it is a fixed prior on θ.","section":"Section 2"},{"comment":"It is unclear whether the validation set for quantifiers is disjoint from the early-stopping split of the BERT training data; please clarify the data splitting procedure.","section":"Section 3.2"},{"comment":"For coverage evaluation, specify whether the 'true prevalence' is the nominal prevalence value used to generate the test set or the empirical fraction of positives in the drawn sample; because sampling is without replacement from finite pools, these can differ slightly.","section":"Section 3.3"},{"comment":"When EMQ is excluded because coverage can be below 50%, report the actual coverage values or a statistical comparison rather than relying on the informal phrase 'coverage was not reliable.'","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of the journal and likely of interest to the quantification community. The missing stress test for the weak prior shift is the most consequential gap; the comparators (PACC, HDy, EMQ) also rely on the same assumption, so the issue is not unique to PQ, but the paper's uncertainty-calibration claims demand a robustness check. The authors should also be asked to provide 95% coverage results before acceptance. The open-source code and reproducible protocol are strengths that should be acknowledged in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PQ is a legitimately new Bayesian quantifier: instead of hard classes, it models soft classifier outputs with non-parametric histograms and builds a posterior predictive for prevalence (Eqs. 10-12). The derivation is sound and the Stan implementation is a sensible way to avoid sampling per-point discrete labels. The experiments are broad — simulated normals, three review datasets — with a sensible artificial-prevalence protocol that fixes test size, and the code is public. Credit where due: for the 50% intervals they actually evaluate, PQ genuinely does tend to give shorter intervals than BayesianCC and bootstrap CIs from PACC/HDy, with coverage tracking the nominal level.\n\nThe soft spots. First, the abstract overclaims. 'More precise than existing quantifiers' is not true in the paper's own numbers: EMQ matches or beats PQ on small test sets (Fig 1C), and the authors then discount EMQ because its coverage is unreliable. That's fine, but the headline should say 'among methods with reliable coverage,' or similar. Second, all coverage results are for 50% intervals. That's an unusual choice, and the text admits more MCMC samples are needed for 95% tails; they never show that the calibration holds at the level practitioners actually use. Third, the weak prior probability shift assumption — P_V(f(X)|Y)=P_T(f(X)|Y) — is the load-bearing assumption, and every experiment satisfies it by construction. Validation and test are drawn from the same QuaPy test split; the simulated data share class-conditional features by design. No experiment violates the assumption, so we learn nothing about coverage under covariate shift. This is a real gap, though it's the same assumption many quantifiers (PACC, HDy) rely on, so it's a limitation shared with the field rather than a fatal flaw.\n\nI'd send this to a serious referee. The method is clearly described, the comparison is mostly fair, and the missing pieces — 95% coverage, a covariate-shift stress test, and a toned-down abstract — are addressable in revision.","headline":"A genuine new Bayesian quantifier with solid 50%-interval results, but the abstract overstates precision and the key prior-shift assumption is never stress-tested.","tokens_in":13015,"tokens_out":2032,"would_cite":true,"duration_ms":21662,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"A new Bayesian method, Precise Quantifier (PQ), yields tighter and equally well-calibrated prediction intervals for class prevalence than existing Bayesian and bootstrap quantifiers.","keywords":["prevalence estimation","quantification learning","Bayesian prediction intervals","uncertainty quantification","prior probability shift","bootstrap confidence intervals","classifier score distributions","MCMC"],"falsifier":"Run PQ on test sets drawn from a different time period, population, or domain than the validation set, with known prevalence, and compare the empirical coverage of nominal 50% intervals across many replications; if coverage drops systematically below 50% when score distributions shift, the central coverage claim fails.","tokens_in":12129,"feed_emoji":"🎯","tokens_out":9634,"duration_ms":93115,"temperature":0.7,"pith_summary":"This paper sets out to show that uncertainty in class-prevalence estimates can be reported more tightly than standard practice allows. It introduces Precise Quantifier (PQ), a Bayesian method that models the classifier's score distribution for each class with bins and derives a posterior predictive distribution for the test-set prevalence. In experiments on simulated data and three real-world review datasets, PQ's prediction intervals were shorter than BayesianCC's intervals and shorter than bootstrap confidence intervals from PACC and HDy, while coverage stayed at the expected confidence level. The paper also identifies the factors that set interval width: classifier discriminatory power, validation-set size, and test-set size. If the claim holds, anyone estimating prevalence from unlabeled data -- epidemiologists, social scientists, or engineers -- could report narrower, trustworthy ranges instead of wider bootstrap intervals.","feed_headline":"Bayesian quantifier PQ beats existing prevalence intervals","feed_subtitle":"Its 50% prediction intervals stay shorter than Bayesian and bootstrap rivals while coverage holds at 50%.","key_machinery":"PQ rests on the weak prior probability shift assumption, $P_V(f(X)|Y)=P_T(f(X)|Y)$, which says the classifier's score distributions for each class are the same in validation and test data even though prevalence differs. Classifier scores are sorted into $N_{\\mathrm{bin}}$ equal-size bins per class; positive and negative validation sets contribute multinomial counts and the unlabeled test set contributes a mixture likelihood $\\theta_{\\mathrm{pr}} p^+_k+(1-\\theta_{\\mathrm{pr}})p^-_k$ per bin. With uniform and Dirichlet priors, the posterior over $\\theta_{\\mathrm{pr}}$ and the bin probabilities is sampled by MCMC, and each posterior draw yields one predictive prevalence via the bin-level binomial sum. This formulation propagates validation and test uncertainty together while sampling only $2N_{\\mathrm{bin}}+1$ continuous parameters rather than one latent label per test point.","core_discovery":"The paper's central discovery is that a multi-level Bayesian quantification model, PQ, gives prediction intervals for prevalence that are narrower than those of existing Bayesian and bootstrap quantifiers while keeping coverage calibrated. PQ treats the test prevalence $\\theta$ as the normalized sum of bin-level binomial counts, giving the posterior predictive distribution $\\theta | \\theta_{\\mathrm{pr}},\\{p^+\\},\\{p^-\\} \\sim \\frac{1}{n_T}\\sum_{k} \\mathrm{Binomial}(t_k, \\theta_{\\mathrm{pr}} p^+_k/(\\theta_{\\mathrm{pr}} p^+_k+(1-\\theta_{\\mathrm{pr}})p^-_k))$. In the paper's comparisons, PQ's central 50% intervals were on average the shortest among methods that reached 50% coverage, and its point estimates had bias comparable to the other well-calibrated methods. The experiments further show that stronger classifiers, larger validation sets, and larger test sets all narrow the intervals, with test-size gains largely realized by 500 samples.","pith_inferences":["Inference: if validation and test data come from different domains, the weak prior probability shift assumption is the only bridge between them, so PQ's coverage should degrade under covariate shift; this is directly testable and would bound the method's practical reach.","Inference: PQ's binning suggests a natural extension to a continuous Dirichlet-process or kernel-density model for class-conditional scores, which could remove the sensitivity to bin count while keeping the same posterior predictive mechanism.","Inference: extending PQ to multi-class would require a genuine multivariate model per bin rather than one-versus-all binarization, since the paper notes that one-versus-all can violate the probability shift assumption."],"forward_implications":["Practitioners can report an interval for prevalence rather than a point estimate, with the width automatically reflecting how informative the classifier is and how much labeled and unlabeled data are available.","Bayesian quantification of this kind beats bootstrap resampling of point estimates in precision under the tested conditions, so interval users gain without losing coverage.","EMQ should not be trusted for interval estimates by default: it matched or beat PQ's precision only on some small test sets, and its coverage fell below the nominal level.","Improving the classifier, enlarging the validation set, or enlarging the test set each buys narrower intervals, and most of the test-size benefit is reached around 500 samples.","The bin count matters: four bins balanced precision and coverage, and larger bin counts can degrade coverage when the validation set is small."],"supporting_citations":[{"why":"Supplies BayesianCC, the competing Bayesian quantifier whose prediction intervals PQ is compared against.","marker":"[8]"},{"why":"Defines bootstrap confidence intervals for quantifiers and provides the normal-distribution simulation setup used in the experiments.","marker":"[6]"},{"why":"Defines the weak prior probability shift assumption that PQ's likelihood and calibration rely on.","marker":"[12]"},{"why":"Introduces the artificial-prevalence protocol and the classify-and-count bias that motivates bias-adjusted prevalence estimation.","marker":"[1]"},{"why":"Provides the framework used to implement all quantifiers and the source of the three review datasets.","marker":"[13]"},{"why":"Defines PACC and PCC, two baseline methods compared on precision and coverage.","marker":"[16]"},{"why":"Defines HDy, a baseline method compared on precision and coverage.","marker":"[17]"},{"why":"Defines EMQ, the baseline method whose occasional high precision but unreliable coverage is contrasted with PQ.","marker":"[18]"},{"why":"Motivates the preset test-set sizes in the evaluation protocol, which shape the reported precision curves.","marker":"[15]"}],"fun_headline_variants":["PQ quantifier shrinks prevalence intervals while keeping coverage calibrated","Bayesian PQ gives shorter calibrated prevalence intervals","Precise Quantifier: narrower prevalence intervals, same coverage","PQ model tightens prevalence intervals without losing coverage calibration","Narrower prevalence intervals from Bayesian PQ with calibrated coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the classifier's score distribution for each class is the same in validation and test data; if that score distribution shifts between settings, PQ's interval coverage can no longer be expected to reach the nominal level.","fun_headline_variants_meta":{"raw":{"variants":["PQ quantifier shrinks prevalence intervals while keeping coverage calibrated","Bayesian PQ gives shorter calibrated prevalence intervals","Precise Quantifier: narrower prevalence intervals, same coverage","PQ model tightens prevalence intervals without losing coverage calibration","Narrower prevalence intervals from Bayesian PQ with calibrated coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000894,"raw_usage":{"total_tokens":3855,"prompt_tokens":950,"completion_tokens":2905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":2827}},"tokens_in":566,"tokens_out":2905,"duration_ms":19463,"temperature":1.0,"reasoning_tokens":2827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:11:47.358641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PQ on test sets drawn from a different time period, population, or domain than the validation set, with known prevalence, and compare the empirical coverage of nominal 50% intervals across many replications; if coverage drops systematically below 50% when score distributions shift, the central coverage claim fails.","supporting_citations":[{"cited_title":"Confidence intervals for class prevalences under prior probability shift","cited_arxiv_id":null,"evidence_quote":"Defines bootstrap confidence intervals for quantifiers and provides the normal-distribution simulation setup used in the experiments."},{"cited_title":"Counting positives accurately despite inaccurate classification","cited_arxiv_id":null,"evidence_quote":"Introduces the artificial-prevalence protocol and the classify-and-count bias that motivates bias-adjusted prevalence estimation."},{"cited_title":"The importance of the test set size in quantification assessment","cited_arxiv_id":null,"evidence_quote":"Defines PACC and PCC, two baseline methods compared on precision and coverage."},{"cited_title":"Quan- tification via probability estimators","cited_arxiv_id":null,"evidence_quote":"Defines HDy, a baseline method compared on precision and coverage."},{"cited_title":"Guzmán- Martínez, and Enrique Alegre","cited_arxiv_id":null,"evidence_quote":"Defines EMQ, the baseline method whose occasional high precision but unreliable coverage is contrasted with PQ."}],"review_version":1}