Learning from Double Positive and Unlabeled Data for Potential-Customer Identification

Kentaro Baba; Masahiro Kato; Ryo Inokuchi; Takashi Imai; Yuki Ikeda

REVIEW 4 major objections 4 minor 17 references

Learning from Double Positive and Unlabeled Data for Potential-Customer Identification

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Double PU learning identifies interested, non-loyal customers from positive and unlabeled data alone.

desk verdict A correct double-PU risk for compound labels, undermined by a real-data experiment that silently assumes known class priors. read the letter →

arxiv 2506.00436 v2 pith:JI7FC56H submitted 2025-05-31 cs.LG cs.AIecon.EMstat.MEstat.ML

Masahiro Kato , Yuki Ikeda , Kentaro Baba , Takashi Imai , Ryo Inokuchi This is my paper

classification cs.LGcs.AIecon.EMstat.MEstat.ML

keywords doublePUlearningpositiveandunlabeleddatapotential-customeridentificationclassificationriskclasspriortargetedmarketingunbiasedestimationcost-sensitive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a learning-from-positive-and-unlabeled-data (PU learning) method to identify potential customers: people who are interested in a product but are not loyal to the company, so marketing to them is worth the cost. It introduces a target label W that is +1 exactly when interest label Y = +1 and loyalty label Z = -1, and shows how to train a classifier for W using three available datasets: people known to be interested, people known to be both interested and loyal, and a completely unlabeled population. The key move is an identity expressing the usual classification risk for W as a combination of expectations over these three distributions, so minimizing the empirical version gives an unbiased risk estimator. The authors call this double PU learning because the objective implicitly contains two PU losses, one separating interest and one removing loyal customers. They validate the approach in simulations and on bank marketing data, where the learned classifier beats random guessing with a ROC-AUC of 0.6013.

What carries the argument

The load-bearing object is Lemma 1, the identity $\alpha p(x \mid w=+1) = \beta p(x \mid y=+1) - \gamma p(x \mid y=+1, z=+1)$, which converts the inaccessible positive-$W$ distribution into a difference of two accessible distributions. This identity carries the argument: it rewrites the risk $R_{0\text{-}1}(g)$ as a linear combination of expectations over $p(x \mid y=+1)$, $p(x)$, and $p(x \mid y=+1, z=+1)$, each with a sample counterpart, so the resulting empirical risk is unbiased. Surrogate losses replace the zero-one loss, and a non-negative correction of the PU term is applied to prevent the negative-loss divergence known as train-loss hacking.

What would settle it

Construct synthetic data with known beta and gamma, train the double PU objective with deliberately wrong priors while keeping the same samples, and observe the learned decision boundary and the risk estimate move away from the Bayes classifier. A clean observation: on a fully labeled benchmark, compute the true beta and gamma, then train with perturbed values and show that the empirical risk no longer equals the true classification risk and the test AUC degrades.

Watch

Extended reading notes

Core claim

The paper's central claim is that the classification risk for the potential-customer label W can be written entirely in terms of expectations over the three observable distributions: $R_{0\text{-}1}(g) = \beta E_{Y=+1}[\ell(g(X))] + E_U[\ell(-g(X))] - \beta E_{Y=+1}[\ell(-g(X))] - \gamma E_{(Y,Z)=(+1,+1)}[\ell(g(X))] + \gamma E_{(Y,Z)=(+1,+1)}[\ell(-g(X))]$, where $\beta = P(Y=+1)$ and $\gamma = P(Y=+1, Z=+1)$. Because each expectation in this expression is directly estimable from positive-interest data, unlabeled data, and positive-loyalty data, minimizing the empirical risk trains a classifier that separates $W=+1$ from $W=-1$ without any explicit negative labels. The $\gamma$ terms subtract the interested-and-loyal block from the interested region, which is the second PU step inside a single-stage optimization.

Load-bearing premise

The class priors beta, the share of people interested in the product, and gamma, the share interested and loyal, are assumed known exactly, and the unbiasedness of the risk estimator collapses if those numbers are wrong.

Editorial extensions

If this is right

If $\beta$ and $\gamma$ are known or accurately estimated, marketers can train a classifier for interested-but-not-loyal customers from data that contains no explicit negatives.
The objective inherits two standard PU risks, so existing PU machinery, including the non-negative risk correction and cost-sensitive weighting, applies directly to double PU learning.
The cost-sensitive version allows asymmetric costs, which matches applications such as credit marketing where lending to a defaulter is far more expensive than missing a customer.
Because the construction is purely distributional, it extends beyond loyalty and interest to any setting with two nested positive subsets and an unlabeled draw.
The unbiased estimator gives finite-sample stability, so the method can be used with flexible model classes rather than only linear classifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper fixes beta and gamma in the real-data experiment without an estimation procedure; a natural extension is to plug in class-prior estimates obtained from the positive and unlabeled samples, at the cost of additional assumptions.
The identity invites an alternative interpretation as a difference of two PU classifiers, an interest classifier minus a loyal-customer classifier, which could yield calibrated scores for targeting instead of a hard 0.5 decision boundary.
In censoring PU settings where the unlabeled set is a random subset of the full population, the same risk expression applies when class priors are known, so double PU could be deployed without separate collection of positive-interest and positive-loyalty datasets.
A testable extension is to compare double PU against a two-stage pipeline that first learns interest and then removes loyal customers, to see whether the single-stage unbiased objective improves finite-sample behavior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The double PU risk in Lemma 1 is the real contribution: it is a correct linear combination of two known unbiased PU estimators to target the conjunctive label W=+1. The derivation in Sections III-A through III-C checks out, and the paper is honest that this is an extension rather than a new framework. The non-negative correction and the cost-sensitive variant are sensible standard additions.

The soft spot is exactly where the stress-test note lands. The method assumes beta = P(Y=+1) and gamma = P(Y=+1,Z=+1) are known, but in the motivating scenario they are not observed. Section V simply says 'We set beta = 0.4738 and gamma = 0.0046' with no estimation procedure, and those numbers almost certainly come from the fully labeled UCI data, so the real-data experiment leaks information relative to the intended deployment setting. If the priors are misspecified, every term in Eq. (1) is misweighted and the estimator is biased; the non-negative correction does not repair that. The paper itself acknowledges in Section VI that priors 'can be estimated under certain conditions, doing so introduces extra assumptions,' but the experiments do not carry that caveat through. This is a load-bearing weakness in the empirical section, not in the math.

The rest of the empirical work is thin. The simulation shows one decision boundary with no error bars or repeated trials. The bank marketing experiment has a confusing split (80% of the data becomes the 'positive loyalty' set while only 10% is unlabeled, which inverts the usual sampling scheme), no baselines, no comparison to standard PU or supervised classifiers, and a single AUC of 0.6013 without variance. The cost-sensitive weighting is set with a 100:1 ratio but no cost curve or sensitivity analysis. The dataset also tests a different semantic (default vs. no-default) than the loyalty story in the abstract, which is fine in principle but should be stated more carefully.

What is genuinely new is narrow but real: a weakly supervised method for a compound label with a clean unbiased risk. If the priors are available or reliably estimated, the method should work. The math is solid, the writing is clear, and the related work is appropriate. The paper deserves referee time, but the experiments need to estimate priors or study sensitivity to them, add baselines, and report variance.

For a serious reader in PU learning or weak supervision, this is worth a look. I would not cite it yet given the empirical gaps, but I would accept it for review.

Referee Report

4 major / 4 minor

Summary. The paper proposes a method, called double PU learning, for classifying potential customers defined by Y=+1 and Z=-1 (interested in a product but not loyal to the company) using three training datasets: positive interest samples, positive loyalty samples, and unlabeled samples. The main theoretical contribution is Lemma 1, which rewrites the classification risk for the derived label W as a linear combination of expectations over the three available distributions, weighted by the class priors beta = P(Y=+1) and gamma = P(Y=+1,Z=+1). Based on this identity, the authors define an empirical risk minimization objective with surrogate losses, discuss a non-negative correction and a cost-sensitive extension, and report a simulation and a real-data experiment on a UCI bank marketing dataset.

Significance. If the derivation is taken at face value, the paper contributes a clean extension of case-control PU learning to settings with two positive label sources: the algebraic identity in Lemma 1 is parameter-free given beta and gamma, and it yields a single-stage training objective without requiring negative labels. This is a useful conceptual contribution for targeted-marketing applications. The paper also explicitly discusses the known-prior assumption and relates the setting to censoring and case-control PU learning. The main weakness is that the empirical validation is not yet sufficient to support the practical claim: the real-data experiment relies on unstated values of beta and gamma, the data split is ambiguous, and no baselines or error bars are reported.

major comments (4)

[Section V] The priors beta=0.4738 and gamma=0.0046 are stated without any estimation procedure. Because the empirical risk in Eq. (1) is unbiased only when beta and gamma equal the true population probabilities, and because the motivating data-availability setting of Section II-B does not include fully labeled data, the real-data experiment must either estimate these priors from the PU data or include a sensitivity analysis over plausible values. As written, the reported AUC validates the classifier under unverified constants; if the constants are wrong, every term in Eq. (1) is misweighted and the risk estimator is biased. If these values were computed from the fully labeled UCI data, that information is not available in the intended deployment scenario.
[Section V] The construction of the three training sets is ambiguous and likely inconsistent. After an 80/20 train/test split, the text says the training data are 'further divided' into sets of 10%, 10%, and 80% of the total, but these percentages overlap (the third set uses 80% of the total, which equals the full training set), and it is unclear whether D_{(y,z)=(+1,+1)} is a subset of D_{y=+1} or an independent sample. This makes the experiment irreproducible and prevents the reader from understanding how much label information is actually used.
[Sections IV and V] The empirical evidence that the method 'works' is thin. The simulation in Section IV shows a single decision boundary with no quantitative accuracy, no repetitions, and no comparison to alternatives; the real-data experiment in Section V reports a single test ROC-AUC of 0.6013 with no standard error, no confidence interval, and no baselines such as a fully supervised classifier, standard PU learning, or logistic regression on the observed labels. A single AUC slightly above 0.5 does not establish that double PU learning is effective for the problem.
[Section V] The real-data experiment defines W as 'accepted a marketing offer and not in default,' which is explicitly different from the loyalty-based definition in the introduction. As the authors note, this is a proxy task, but they do not justify why performance on this proxy transfers to the motivating potential-customer setting or present an experiment on the actual loyalty definition. This weakens the connection between the empirical claim and the paper's central application.

minor comments (4)

[Section III-E] The proposed non-negative correction clips only the E_U[l(-g)] - beta E_{Y=+1}[l(-g)] part of the risk; the two gamma-weighted terms, -gamma E_{++}[l(g)] + gamma E_{++}[l(-g)], can still be unbounded below for flexible models such as neural networks. If the intention is to transfer the standard non-negative correction from the PU literature, the formula or its justification should be adjusted.
[Section IV] The description of the simulation's labeling process, in particular the phrases 'randomized 30% of both (Y,Z)=(+1,-1) and (Y,Z)=(+1,+1) to unlabeled for Y' and 'further randomized 50% of (Y,Z)=(+1,+1) to unlabeled for Z,' is difficult to parse; a precise algorithm for generating D_{Y=+1}, D_U, and D_{++} would improve reproducibility.
[Table I] For the log loss, the domain is listed as (0,1), but the decision function g(X) is otherwise treated as real-valued; the authors should clarify how the classifier output is mapped to (0,1) for this loss.
[Throughout] The notation D_{y=+1} is used both for the conceptual positive dataset and for the sample set {X_j}; using distinct symbols for the distribution and the finite sample would improve clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reduction: Lemma 1 is a direct algebraic rewrite of the classification risk under the known-prior assumption; the fixed beta and gamma values in Section V are an experimental limitation, not a fitted-input prediction.

full rationale

The derivation chain is self-contained. Lemma 1 rewrites R0-1(g) using the identity alpha p(x|w=+1)=beta p(x|y=+1)-gamma p(x|y=+1,z=+1), which follows from the definitions of W, beta, and gamma; no term is defined in terms of the target classifier or the fitted parameters. Equation (1) is the sample analog of this identity with a surrogate loss, so the unbiasedness claim is a direct algebraic consequence rather than a restatement of an input. The only potentially questionable step is Section V, where beta=0.4738 and gamma=0.0046 are set without an estimation procedure; Section VI acknowledges that priors are usually unknown and that estimation requires extra assumptions. This is a limitation of the empirical demonstration, not circularity, because the values are not fitted to the test predictions and the paper's central claim is explicitly conditional on known beta and gamma. Self-citations [5]-[7] are contextual (train-loss hacking, selection-bias PU, class-prior estimation) and do not carry the derivation. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method relies on two class priors that are treated as known, plus standard sampling and loss assumptions. No new physical or conceptual entities are introduced; W is a derived label, not an invented entity.

free parameters (3)

class prior beta = 0.4738 (Section V)
Probability of Y=+1; treated as known in the risk. In the experiment it is set to this value without a described estimation procedure, and the motivating scenario does not provide it.
class prior gamma = 0.0046 (Section V)
Probability of (Y=+1,Z=+1); similarly assumed known and set in the experiment without stated estimation.
cost weights c_FN and c_FP = c_FP/c_FN = 100 (Section V)
Chosen by hand to reflect business cost; affects the classifier and is application-specific.

assumptions (4)

domain assumption The three datasets are independently drawn from p(x|y=+1), p(x), and p(x|y=+1,z=+1)
Section II-B; unbiasedness of the empirical risk relies on these sampling assumptions, but in the real experiment the datasets are carved from the same training set and may not be independent.
domain assumption A surrogate loss is an appropriate replacement for the zero-one loss
Section III-C; no consistency theorem is provided for the chosen surrogate losses, so the empirical minimizer may not converge to the optimal classifier.
domain assumption W=+1 defined as Y=+1 and Z=-1 is the correct business target
Section I; the mapping from the business concept of potential customers to the label definition is assumed without empirical validation.
standard math Standard probability identities hold
Lemma 1 uses basic manipulations such as the law of total probability and conditional density factorization.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning from Double Positive and Unlabeled Data for Potential-Customer Identification." pith.science (2026). https://pith.science/paper/JI7FC56H

@misc{pith2026250600436,
  author       = {Pith},
  title        = {Pith review of: Learning from Double Positive and Unlabeled Data for Potential-Customer Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JI7FC56H}},
  note         = {Machine review of arXiv:2506.00436}
}

read the original abstract

In this study, we propose a method for identifying potential customers in targeted marketing by applying learning from positive and unlabeled data (PU learning). We consider a scenario in which a company sells a product and can observe only the customers who purchased it. Decision-makers seek to market products effectively based on whether people have loyalty to the company. Individuals with loyalty are those who are likely to remain interested in the company even without additional advertising. Consequently, those loyal customers would likely purchase from the company if they are interested in the product. In contrast, people with lower loyalty may overlook the product or buy similar products from other companies unless they receive marketing attention. Therefore, by focusing marketing efforts on individuals who are interested in the product but do not have strong loyalty, we can achieve more efficient marketing. To achieve this goal, we consider how to learn, from limited data, a classifier that identifies potential customers who (i) have interest in the product and (ii) do not have loyalty to the company. Although our algorithm comprises a single-stage optimization, its objective function implicitly contains two losses derived from standard PU learning settings. For this reason, we refer to our approach as double PU learning. We verify the validity of the proposed algorithm through numerical experiments, confirming that it functions appropriately for the problem at hand.

Figures

Figures reproduced from arXiv: 2506.00436 by the authors.

**Figure 2.** Quadrants divided by Y and Z. The upper left quadrant represents W = +1, and the other quadrants are W = −1. a) Classification risk.: Let g : X → R be a classifier and G be some set of g. Let EW=+1 and EW=−1 be expectations over p [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The standard PU classification. b) Unbiased double PU risk.: The blue region in Figure 3 includes (Y, Z) = (+1, +1) in addition to (Y, Z) = (+1, −1) (W = +1). We aim to remove this region. This step can be interpreted as the second PU classification, since we minimize another empirical risk by regarding Y = +1 as unlabeled data and (Y, Z) = (+1, +1) as labeled data. As a result, we obtain a classifier that classifi… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The double PU classification. Thus, our double PU learning algorithm trains a classifier for identifying potential customers by minimizing the unbiased empirical risk. This unbiasedness is important because it guarantees finite-sample stability of the learning process…

**Figure 5.** Figure 5: Results of the numerical experiment. customer. We first divided the original dataset into training and test dataset by 80% and 20%. Next, we further divided the training dataset into three datasets where the first one (10% of total) used only samples with Y = +1 (corre…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

[1]

Learning from positive and unlabeled data under the selected at random assumption

Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data under the selected at random assumption. In Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 94, pages 8–22, 2018

work page 2018
[2]

Semi- supervised novelty detection

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi- supervised novelty detection. Journal of Machine Learn- ing Research, 11(Nov):2973–3009, 2010

work page 2010
[3]

Niu, and Masashi Sugiyama

Marthinus Christoffel du Plessis, Gang. Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In ICML, pages 1386–1394, 2015

work page 2015
[4]

Learning classifiers from only positive and unlabeled data

Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In KDD, pages 213– 220, 2008

work page 2008
[5]

Non-negative bregman divergence minimization for deep direct density ratio estimation

Masahiro Kato and Takeshi Teshima. Non-negative bregman divergence minimization for deep direct density ratio estimation. In ICLR, volume 139, pages 5320–5333, 2021

work page 2021
[6]

Learning from positive and unlabeled data with a selec- tion bias

Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selec- tion bias. In ICLR, 2019

work page 2019
[7]

Alternate Estimation of a Classifier and the Class-Prior from Positive and Unlabeled Data

Masahiro Kato, Liyuan Xu, Gang Niu, and Masashi Sugiyama. Alternate estimation of a classifier and the class-prior from positive and unlabeled data, 2018. arXiv: 1809.05710

work page Pith review arXiv 2018
[8]

Positive-unlabeled learning with non-negative risk estimator

Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In NeurIPS, pages 1675–1685, 2017

work page 2017

Show all 17 references

[9]

Case-control studies with contaminated controls

Tony Lancaster and Guido Imbens. Case-control studies with contaminated controls. Journal of Econometrics , 71(1):145–160, 1996. 2As [13] discusses, most case-control PU algorithm can be applied to censoring PU learning setting when the class prior is known

1996
[10]

Positive unlabeled learning for data stream classification

Xiao-Li Li, Philip S Yu, Bing Liu, and See-Kiong Ng. Positive unlabeled learning for data stream classification. In SDM, pages 259–270, 2009

2009
[11]

Sur les applications de la theorie des probabilites aux experiences agricoles: Essai des principes

Jerzy Neyman. Sur les applications de la theorie des probabilites aux experiences agricoles: Essai des principes. Statistical Science, 5:463–472, 1923

1923
[12]

Positive unlabeled leaning for time series classification

Minh Nhut Nguyen, Xiaoli-Li Li, and See-Kiong Ng. Positive unlabeled leaning for time series classification. In IJCAI, pages 1421–1426, 2011

2011
[13]

Theoretical com- parisons of positive-unlabeled learning against positive- negative learning

Gang Niu, Marthinus Christoffel du Plessis, Tomoya Sakai, Yao Ma, and Masashi Sugiyama. Theoretical com- parisons of positive-unlabeled learning against positive- negative learning. In NeurIPS, volume 29, 2016

2016
[14]

D.B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688–701, 1974

1974
[15]

Novelty detection: Unlabeled data definitely help

Clayton Scott and Gilles Blanchard. Novelty detection: Unlabeled data definitely help. In AISTATS, pages 464– 471, 2009

2009
[16]

van der Vaart

A.W. van der Vaart. Asymptotic Statistics . Cambridge Series in Statistical and Probabilistic Mathematics. Cam- bridge University Press, 1998

1998
[17]

Presence-only data and the em algorithm

Gill Ward, Trevor Hastie, Simon Barry, Jane Elith, and John R Leathwick. Presence-only data and the em algorithm. Biometrics, 65(2):554–563, 2009

2009

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Learning from positive and unlabeled data under the selected at random assumption

Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data under the selected at random assumption. In Proceedings of the Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, volume 94, pages 8–22, 2018

work page 2018

[2] [2]

Semi- supervised novelty detection

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi- supervised novelty detection. Journal of Machine Learn- ing Research, 11(Nov):2973–3009, 2010

work page 2010

[3] [3]

Niu, and Masashi Sugiyama

Marthinus Christoffel du Plessis, Gang. Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In ICML, pages 1386–1394, 2015

work page 2015

[4] [4]

Learning classifiers from only positive and unlabeled data

Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In KDD, pages 213– 220, 2008

work page 2008

[5] [5]

Non-negative bregman divergence minimization for deep direct density ratio estimation

Masahiro Kato and Takeshi Teshima. Non-negative bregman divergence minimization for deep direct density ratio estimation. In ICLR, volume 139, pages 5320–5333, 2021

work page 2021

[6] [6]

Learning from positive and unlabeled data with a selec- tion bias

Masahiro Kato, Takeshi Teshima, and Junya Honda. Learning from positive and unlabeled data with a selec- tion bias. In ICLR, 2019

work page 2019

[7] [7]

Alternate Estimation of a Classifier and the Class-Prior from Positive and Unlabeled Data

Masahiro Kato, Liyuan Xu, Gang Niu, and Masashi Sugiyama. Alternate estimation of a classifier and the class-prior from positive and unlabeled data, 2018. arXiv: 1809.05710

work page Pith review arXiv 2018

[8] [8]

Positive-unlabeled learning with non-negative risk estimator

Ryuichi Kiryo, Gang Niu, Marthinus Christoffel du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In NeurIPS, pages 1675–1685, 2017

work page 2017