REVIEW 4 major objections 4 minor 45 references
Learning from Label Proportions and Covariate-shifted Instances
T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read When target labels are available only as bag averages and source data is fully labeled under covariate shift, the paper's BagCSI loss bounds target error and improves prediction.
desk verdict New hybrid LLP problem with solid empirics, but Theorem 3.2 as stated is not proven—the proof only yields a threshold lower bound on bag loss, not the claimed multiplicative control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the BagCSI loss (equation 3), a weighted sum of the target bag-level mean-squared error $\bar{\varepsilon}(B,h)$, the source instance-level mean-squared error $\hat{\varepsilon}(S,h)$, and the squared label-weighted feature-alignment gap $\xi^2(S,B)$, where $\xi(S,B)$ is the norm of the difference between the bag-label-weighted average of target features and the instance-label-weighted average of source features. This last term is what converts bag labels into a domain-adaptation signal. The argument is carried by Lemma 3.1 and Theorem 3.2: the lemma bounds the gap between target bag loss and source instance loss in terms of $\xi$, and the theorem shows that bag loss controls target instance error up to the factor $16k$, with the sample-size requirement set by the pseudo-dimension of the regressor class. The pseudo-label variant $\psi(S,B)$ replaces true target instance labels with pseudo-labels that shift each bag's predictions to match its bag average, so the same alignment idea works without target instance labels.
What would settle it
A concrete check: take any real dataset, create a source/target split where $p(Y|X)$ is known to differ (for example, the same product features with prices set by different markets), and compare BagCSI against a target-bags-only baseline; if the alignment term increases target MSE, the paper's reliance on shared $p(Y|X)$ is the failure point. Alternatively, with target instance labels available for evaluation, compute $\bar{\varepsilon}(B,h)$ and $16k \bar{\varepsilon}(B,h)$ for a trained $h$ and check whether $\varepsilon(D_T,h)$ violates the bound under the theorem's sampling scheme.
Extended reading notes
Core claim
The discovery is a loss and training procedure that uses target bag labels inside the domain-alignment term, not just in the prediction loss. The BagCSI loss is $\lambda_1$ times the target bag mean-squared error, plus $\lambda_2$ times the source instance mean-squared error, plus $\lambda_3 \xi^2(S,B)$, where $\xi$ is the norm of the difference between the average of bag-label-weighted target feature means and the average of source label-feature products. The paper proves Lemma 3.1, that the target bag loss minus source instance loss is bounded by $\xi$ times the regressor norm plus label-independent terms, and Theorem 3.2, the first bag-to-instance generalization bound for regression in LLP using pseudo-dimension: for random bags of size $k$, with high probability every regressor with target error at least $\nu$ satisfies $\varepsilon(D_T,h) \leq 16k \bar{\varepsilon}(B,h)$. The PL-WFA variant replaces bag labels with per-instance pseudo-labels that respect the bag average, giving the same alignment effect when instance-level target predictions are available. The paper's experiments show BL-WFA and PL-WFA matching or beating LLP and domain-adaptation baselines, with larger gains at larger bag sizes.
Load-bearing premise
The claim rests on the assumption that only the feature distribution changes between source and target, while the mapping from features to labels $p(Y|X)$ stays the same; if the label mechanism also changes, source instance labels can mislead the target predictor.
Editorial extensions
If this is right
- Training with BagCSI is an upper-bound-motivated surrogate for target instance error, so minimizing the three terms together should give a predictor whose target error is controlled by the quantities in the bound.
- For a fixed bag size, the target bag-level loss alone certifies instance-level error up to factor $16k$, so smaller bags give tighter guarantees and larger bags need the source term to compensate.
- The PL-WFA variant extends the alignment idea to settings where a trained hypothesis can distribute each bag's average among its instances, making the method applicable in iterative or self-training pipelines.
- The baselines adapted from prior work (AF, LR, DANN variants, DMFA) are natural comparisons, and the paper's tables show PL-WFA and BL-WFA improving on them at large bag sizes, including on correlated and mixed-size bags.
- The linear degradation of the bound with bag size is not an artifact: the appendix example shows the $k$ blowup is unavoidable for random bags.
Reading between the lines
- Inference: The BagCSI alignment term is essentially first-moment matching of label-weighted features; a natural extension is to second-order or class-conditional alignment, which the paper does not explore.
- Inference: The method's practical value hinges on the shared-$p(Y|X)$ assumption; when the label mechanism differs across domains, the alignment term could pull the predictor toward the source label mechanism, so a diagnostic comparing source and target residual structure would be useful.
- Inference: In privacy-driven applications, bag sizes are often large, and the theorem implies the source term becomes increasingly important exactly in that regime, so the practical payoff is largest where bag information is weakest.
- Inference: A testable extension is to measure the slack in Theorem 3.2 directly by computing $16k \bar{\varepsilon}(B,h)$ for a trained $h$ and comparing it to the target instance error on held-out labels.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper defines covariate-shifted hybrid LLP: the learner has instance-labeled source data from a covariate-shifted distribution and bag-labeled target data, and wants an instance-level predictor for the target. It proposes the BagCSI loss (Eq. 3), which combines source instance MSE, target bag MSE, and a label-weighted feature-alignment term ξ(S,B), together with a pseudo-label variant PL-WFA. The theoretical contribution is Lemma 3.1, bounding the bag-loss/source-loss gap, and Theorem 3.2, a bag-to-instance generalization bound for regression with pseudo-dimension-based sample complexity. Experiments on synthetic data, Wine, IPUMS, and Criteo compare BL-WFA and PL-WFA against LLP and domain-adaptation baselines across random, correlated, and mixed-size bags.
Significance. If Theorem 3.2 is established, the paper provides a useful bag-to-instance regression guarantee and a principled motivation for using source instance labels together with target bag labels in domain adaptation. The empirical study is broad and reproducible, with code released, multiple bag-generation protocols, and consistent gains at large bag sizes. However, the written proof of Theorem 3.2 stops short of the stated multiplicative bound, and the claim that BagCSI is an upper bound on target error is an overstatement; these issues need to be repaired before the theoretical motivation is sound. The empirical claims also rely on assumptions that are not validated on the real datasets, so the paper's scope should be stated more carefully.
major comments (4)
- [Section 5] The proof as written concludes ε(B,h) ≥ ν/(16k) for all h with ε(DT,h) ≥ ν. This is only a threshold lower bound on the bag loss and does not by itself imply the theorem's multiplicative claim ε(DT,h) ≤ 16k ε̄(B,h) for every such h. For example, a regressor with ε(DT,h)=10ν would need ε̄(B,h) ≥ 10ν/(16k), whereas the written chain permits ε̄(B,h)=ν/(16k). The gap is repairable: Theorem 17.1 of Anthony and Bartlett gives the stronger uniform-convergence consequence ε̂(Z,h) ≥ ε(DT,h) − ν/4, which, combined with (11), yields ε̄(B,h) ≥ (ε(DT,h) − ν/4)/(12k) ≥ ε(DT,h)/(16k) for all ε(DT,h) ≥ ν. This step should be added explicitly; as printed, the theorem is not proved.
- [Section 6] The sentence "We have shown above that BagCSI loss is an upper bound over ε(DT,h)" is not supported by the preceding derivation. Lemma 3.1 bounds a difference of empirical losses, not ε(DT,h); the passage from the lemma to Eq. (3) replaces ξ(S,B) by ξ²(S,B), drops R(h,S,T) and λ′(S,T), and treats ∥r_h∥₂ as an unknown bounded constant. Thus BagCSI is a heuristic surrogate motivated by the bound, not a certified upper bound. Please revise this claim and the surrounding motivation in Section 3.1 accordingly.
- [Section 7] The covariate-shift assumption that p(Y|X) is identical on source and target is stated in Section 1 but is not validated on any real dataset. For the Wine split (France versus the rest), the label is price and the features include review-text indicators, so the label mechanism may well differ across domains; in that case source instance labels could actively mislead the target predictor. The synthetic experiments control this assumption, but the real-data conclusions should be qualified, or a diagnostic should be added to check whether p(Y|X) appears stable across the chosen domains.
- [Section 7] Theorem 3.2 assumes labels and regressor outputs in [0,1], and its proof uses |z_i| ≤ 1 in Eq. (6) and in the covering arguments. The real-world experiments use unbounded labels (Wine price, Criteo sales amount) and a final linear layer without output clamping, so the theoretical guarantee does not directly apply to those settings. The paper should either normalize labels to the theorem's range or explicitly state that the experiments are outside the theoretical scope.
minor comments (4)
- [References] The references for Quadrianto et al. (2009a) and (2009b) are identical, as are Li and Culotta (2023a) and (2023b); these duplicates should be merged.
- [Section 8] Inference 3 says proposed methods are better than using instance-level labeled target data, but Table 3 reports target instance loss 0.19 ± 0.02 while PL-WFA and BL-WFA at bag size 256 are 18.40 and 13.76, respectively. This statement is contradicted by the table; presumably "source instance loss" was intended. Please correct it.
- [Section 3] The notation B is overloaded: it denotes both an individual bag B ⊆ R^d and the sample of bags B(m,k) used in ε̄(B,h). Please use a separate symbol, e.g., script B for the bag sample.
- [Throughout] There are several typos and small formatting issues: "consistenty" in the Conclusion, "± 10." in Table 4, and "discarding features with labels in the top 5 percentile" in Appendix E.2 should be "discarding instances". These should be cleaned up.
Circularity Check
No significant circularity: BagCSI is motivated by, not equivalent to, the bound; self-citations are contextual.
full rationale
The derivation chain is not circular. The BagCSI loss (Eq. 3) is introduced after Lemma 3.1 and Theorem 3.2 as an optimization surrogate for the upper bound; the bound is proved from definitions via Cauchy-Schwarz, Hoeffding concentration, and covering arguments, and it does not assume the loss or its fitted weights. The xi(S,B) term in Eq. 2 is a deterministic function of the training data (source instance labels and target bag labels), not of a fitted parameter, so minimizing it is not 'predicting' a fitted value. The PL-WFA pseudo-label loop is anchored by true bag labels and the source instance loss, and evaluation is on target instance labels, which never enter training. The self-citations (Saket et al. 2022; Nandy et al. 2022; Brahmbhatt et al. 2024) are contextual - they concern LLP methods and Criteo preprocessing - and are not used to justify the core bound or to forbid alternatives. One caveat belongs to correctness, not circularity: the proof of Theorem 3.2 (Section 5, Eq. 11) appears to establish only the threshold lower bound epsilon(B,h) >= nu/(16k) for h with epsilon(DT,h) >= nu, rather than the claimed multiplicative inequality epsilon(DT,h) <= 16k * epsilon(B,h); this is a possible soundness gap, but it is not an equivalence-by-construction or a fitted-input-as-prediction, so it does not raise the circularity score.
Assumptions & free parameters
free parameters (4)
- λ1, λ2, λ3 (BagCSI loss weights) =
Not reported per configuration; selected by grid search
- κ (adaptive scaling for ξ² term) =
κ = \barε(B,h)/ξ²(S,B) computed during training
- Learning rate and optimizer =
Adam or SGD selected by grid search
- Wine outlier cutoff percentile =
Top 5 percentile labels removed
assumptions (6)
- domain assumption Covariate shift preserves conditional label distribution p(Y|X)
- domain assumption Target instances are randomly aggregated into fixed-size bags
- standard math Function class has finite pseudo-dimension p with covering number bound N∞(ξ,F,N) ≤ (eN/ξp)^p
- domain assumption Labels are bounded in [0,1] for the theory
- ad hoc to paper Regularization term R(h,S,T) can be omitted from the loss
- ad hoc to paper Representation is linear in features: f(x)=r_f^T φ(x) with bounded ||r_f||₂
Cite this review
Pith. "Pith review of Learning from Label Proportions and Covariate-shifted Instances." pith.science (2026). https://pith.science/paper/OQJZZRLO
@misc{pith2026241112334,
author = {Pith},
title = {Pith review of: Learning from Label Proportions and Covariate-shifted Instances},
year = {2026},
howpublished = {\url{https://pith.science/paper/OQJZZRLO}},
note = {Machine review of arXiv:2411.12334}
}
read the original abstract
In many applications, especially due to lack of supervision or privacy concerns, the training data is grouped into bags of instances (feature-vectors) and for each bag we have only an aggregate label derived from the instance-labels in the bag. In learning from label proportions (LLP) the aggregate label is the average of the instance-labels in a bag, and a significant body of work has focused on training models in the LLP setting to predict instance-labels. In practice however, the training data may have fully supervised albeit covariate-shifted source data, along with the usual target data with bag-labels, and we wish to train a good instance-level predictor on the target domain. We call this the covariate-shifted hybrid LLP problem. Fully supervised covariate shifted data often has useful training signals and the goal is to leverage them for better predictive performance in the hybrid LLP setting. To achieve this, we develop methods for hybrid LLP which naturally incorporate the target bag-labels along with the source instance-labels, in the domain adaptation framework. Apart from proving theoretical guarantees bounding the target generalization error, we also conduct experiments on several publicly available datasets showing that our methods outperform LLP and domain adaptation baselines as well techniques from previous related work.
Reference graph
Works this paper leans on
-
[1]
and Bartlett, P
Anthony, M. and Bartlett, P. L. (2009). Neural Network Learning: Theoretical Foundations . Cambridge University Press, USA, 1st edition
2009
-
[2]
Apple (n.d.). Apple storekit ad network. https://developer.apple.com/documentation/storekit/skadnetwork/
-
[3]
Ardehaly, E. M. and Culotta, A. (2016). Domain adaptation for learning from label proportions using self-training. In IJCAI , pages 3670--3676
work page 2016
-
[4]
Ardehaly, E. M. and Culotta, A. (2017). Co-training for demographic classification using deep learning from label proportions. In ICDM , pages 1017--1024
work page 2017
-
[5]
Baručić, D. and Kybic, J. (2022). Fast learning from label proportions with small bags. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 3156--3160
work page 2022
-
[6]
N., Katramados, I., Hogeweg, L., Thomsen, L
Bortsova, G., Dubost, F., rting, S. N., Katramados, I., Hogeweg, L., Thomsen, L. H., Wille, M. M. W., and de Bruijne, M. (2018). Deep learning from label proportions for emphysema quantification. In MICCAI , volume 11071 of Lecture Notes in Computer Science , pages 768--776. Springer
work page 2018
-
[7]
Brahmbhatt, A., Pokala, M., Saket, R., and Raghuveer, A. (2024). Llp-bench: A large scale tabular benchmark for learning from label proportions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages 4374--4381
work page 2024
-
[8]
I., Choi, H., Dick, T., Gentile, C., and Mu \ n oz Medina , A
Busa - Fekete, R. I., Choi, H., Dick, T., Gentile, C., and Mu \ n oz Medina , A. (2023). Easy learning from label proportions. CoRR , abs/2302.03115
arXiv 2023
Show all 45 references
-
[9]
Chen, L., Fu, T., Karbasi, A., and Mirrokni, V. (2023). Learning from aggregated data: Curated bags versus random bags. arXiv
2023
-
[10]
Chen, L., Huang, Z., and Ramakrishnan, R. (2004). Cost-based labeling of groups of mass spectra. In Proc. ACM SIGMOD International Conference on Management of Data , pages 167--178
2004
-
[11]
Chen, S., Liu, B., Qian, M., and Zhang, C. (2009). Kernel k-means based framework for aggregate outputs classification. In Saygin, Y., Yu, J. X., Kargupta, H., Wang, W., Ranka, S., Yu, P. S., and Wu, X., editors, ICDM , pages 356--361
2009
-
[12]
Wine ratings
Dara (2018). Wine ratings. https://www.kaggle.com/datasets/dbahri/wine-ratings/data. Licensed under CC BY-NC-SA 4.0
2018
-
[13]
and K \" u ck, H
de Freitas, N. and K \" u ck, H. (2005). Learning about individuals from group statistics. In UAI , pages 332--339
2005
-
[14]
M., Nachman, B., Rubbo, F., and Schwartzman, A
Dery, L. M., Nachman, B., Rubbo, F., and Schwartzman, A. (2017). Weakly supervised classification in high energy physics. Journal of High Energy Physics , 2017(5):1--11
2017
-
[15]
Dulac - Arnold, G., Zeghidour, N., Cuturi, M., Beyer, L., and Vert, J. P. (2019). Deep multi-class learning from label proportions. CoRR , abs/1905.12909
2019 arXiv
-
[16]
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., March, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. Journal of machine learning research , 17(59):1--35
2016
-
[17]
Private aggregation api of chrome privacy sandbox
Google (n.d.). Private aggregation api of chrome privacy sandbox. https://developer.chrome.com/docs/ privacy-sandbox/aggregation-service/
-
[18]
Hern \' a ndez - Gonz \' a lez, J., Inza, I., and Lozano, J. A. (2013). Learning bayesian network classifiers from label proportions. Pattern Recognit. , 46(12):3425--3440
2013
-
[19]
A., Iñarra, M
Hernández-González, J., Inza, I., Crisol-Ortíz, L., Guembe, M. A., Iñarra, M. J., and Lozano, J. A. (2018). Fitting the data from embryo implantation prediction: Learning from label proportions. Statistical methods in medical research , 27(4):1056--1066
2018
-
[20]
Kotzias, D., Denil, M., de Freitas, N., and Smyth, P. (2015). From group to individual labels using deep features. In Proc. SIGKDD , pages 597--606
2015
-
[21]
and Culotta, A
Li, X. and Culotta, A. (2023a). Domain adaptation for learning from label proportions using domain-adversarial neural network. SN Comput. Sci. , 4(5):615
2023
-
[22]
and Culotta, A
Li, X. and Culotta, A. (2023b). Domain adaptation for learning from label proportions using domain-adversarial neural network. SN Computer Science , 4(5):615
2023
-
[23]
Lin, K.-Y., Lin, H.-Y., Hsu, Y.-P., and Huang, Y.-C. (2024). Age aware scheduling for differentially-private federated learning. In 2024 IEEE International Symposium on Information Theory (ISIT) , pages 398--403
2024
-
[24]
Liu, J., Wang, B., Qi, Z., Tian, Y., and Shi, Y. (2019). Learning from label proportions with generative adversarial networks. In Proc. NeurIPS , pages 7167--7177
2019
-
[25]
Liu, J., Wang, B., Shen, X., Qi, Z., and Tian, Y. (2021). Two-stage training for learning from label proportions. In Zhou, Z., editor, Proc. IJCAI , pages 2737--2743
2021
-
[26]
Long, M., Cao, Y., Wang, J., and Jordan, M. (2015). Learning transferable features with deep adaptation networks. In International conference on machine learning , pages 97--105. PMLR
2015
-
[27]
Long, M., Zhu, H., Wang, J., and Jordan, M. I. (2017). Deep transfer learning with joint adaptation networks. In International conference on machine learning , pages 2208--2217. PMLR
2017
-
[28]
R., Christensen, J
Musicant, D. R., Christensen, J. M., and Olson, J. F. (2007). Supervised learning by training on aggregate outputs. In ICDM , pages 252--261. IEEE Computer Society
2007
-
[29]
Nandy, J., Saket, R., Jain, P., Chauhan, J., Ravindran, B., and Raghuveer, A. (2022). Domain-agnostic contrastive representations for learning from label proportions. In Proc. CIKM , pages 1542--1551
2022
-
[30]
O'Brien, C., Thiagarajan, A., Das, S., Barreto, R., Verma, C., Hsu, T., Neufield, J., and Hunt, J. J. (2022). Challenges and approaches to privacy preserving post-click conversion prediction
2022
-
[31]
rting, S., Petersen, J., Wille, M., Thomsen, L., and de Bruijne , M. (2016). Quantifying emphysema extent from weakly labeled ct scans of the lungs using label proportions learning. In The Sixth International Workshop on Pulmonary Image Analysis , pages 31--42
2016
-
[32]
S., and Rivera, P
Patrini, G., Nock, R., Caetano, T. S., and Rivera, P. (2014). (almost) no label no cry. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems , pages 190--198
2014
-
[33]
J., Caetano, T
Quadrianto, N., Smola, A. J., Caetano, T. S., and Le, Q. V. (2009a). Estimating labels from label proportions. J. Mach. Learn. Res. , 10:2349--2374
2009
-
[34]
J., Caetano, T
Quadrianto, N., Smola, A. J., Caetano, T. S., and Le, Q. V. (2009b). Estimating labels from label proportions. J. Mach. Learn. Res. , 10:2349--2374
2009
-
[35]
Rueping, S. (2010). SVM classifier estimation from group probabilities. In Proc. ICML , pages 911--918
2010
-
[36]
Ruggles, S., Flood, S., Sobek, M., Backman, D., Chen, A., Cooper, G., Richards, S., Rodgers, R., and Schouweiler, M. (2024). IPUMS USA: Version 15.0 [dataset]
2024
-
[37]
Saket, R., Raghuveer, A., and Ravindran, B. (2022). On combining bags to better learn from label proportions. In AISTATS , volume 151 of Proceedings of Machine Learning Research , pages 5913--5927. PMLR
2022
-
[38]
and Morik, K
Stolpe, M. and Morik, K. (2011). Learning from label proportions by optimizing cluster model selection. In Gunopulos, D., Hofmann, T., Malerba, D., and Vazirgiannis, M., editors, ECML PKDD Proceedings, Part III , volume 6913, pages 349--364. Springer
2011
-
[39]
and Yadav, P
Tallis, M. and Yadav, P. (2018). Reacting to variations in product demand: An application for conversion rate (cr) prediction in sponsored search. arXiv preprint arXiv:1806.08211
2018 arXiv
-
[40]
Wojtusiak, J., Irvin, K., Birerdinc, A., and Baranova, A. V. (2011). Using published medical results and non-homogenous data in rule learning. In Proc. International Conference on Machine Learning and Applications and Workshops , volume 2, pages 84--89. IEEE
2011
-
[41]
X., Choromanski, K., Kumar, S., Jebara, T., and Chang, S
Yu, F. X., Choromanski, K., Kumar, S., Jebara, T., and Chang, S. F. (2014). On learning from label proportions. CoRR , abs/1402.5902
2014 arXiv
-
[42]
X., Liu, D., Kumar, S., Jebara, T., and Chang, S
Yu, F. X., Liu, D., Kumar, S., Jebara, T., and Chang, S. (2013). SVM for learning with label proportions. In ICML , volume 28 of JMLR Workshop and Conference Proceedings , pages 504--512
2013
-
[43]
Wine reviews
Zackthoutt (2017). Wine reviews. https://www.kaggle.com/datasets/zynicide/wine-reviews. Licensed under CC BY-NC-SA 4.0
2017
-
[44]
Zhang, J., Wang, Y., and Scott, C. (2022a). Learning from label proportions by learning with label noise. In Proc. NeurIPS
2022
-
[45]
Zhang, M., Wei, E., Berry, R., and Huang, J. (2022b). Age-dependent differential privacy. SIGMETRICS Perform. Eval. Rev. , 50(1):115–116
2022
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.