Recognition: no theorem link
Online Set Learning from Precision and Recall Feedback
Pith reviewed 2026-05-12 02:57 UTC · model grok-4.3
The pith
A hypothesis class is learnable from precision and recall feedback if and only if it has finite VC dimension.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In this online set learning model with stochastic precision and recall feedback, a hypothesis class is learnable if and only if it has finite Vapnik-Chervonenkis (VC) dimension. Algorithms exist that achieve sublinear regret despite the feedback dependencies invalidating ERM, extending the classical characterization to this partial-information setting.
What carries the argument
The mixed feedback model, where each round independently selects precision feedback (sample from the prediction and check target membership) or recall feedback (sample from the target and check prediction membership) with equal probability, together with the VC dimension of the hypothesis class as the learnability criterion.
If this is right
- Standard empirical risk minimization and proper learning rules can fail due to feedback dependencies.
- Sublinear regret is achievable in both realizable and agnostic settings with tailored algorithms.
- Optimal regret rates remain undetermined.
- The result gives a complete qualitative characterization of learnability in the model.
Where Pith is reading between the lines
- The same VC-based threshold may apply to other online problems that mix different forms of partial feedback.
- Practical performance could be tested on simple classes such as intervals on the line.
- Relaxing the equal-probability assumption on feedback types might preserve or alter the characterization.
Load-bearing premise
Each round the learner receives either precision or recall feedback independently with equal probability, and a reward is received exactly when the revealed item matches the target set membership.
What would settle it
A concrete hypothesis class with finite VC dimension for which every algorithm suffers linear regret, or a class with infinite VC dimension for which some algorithm achieves sublinear regret.
read the original abstract
We consider the problem of learning an unknown subset $N_\text{target}$ of a domain in an online setting. In each round $t$, the learner predicts a set of items ${N}_t$ and receives one of two types of feedback, each with equal probability: precision feedback, in which a randomly chosen item from the predicted set $N_t$ is revealed and the learner is told whether it belongs to $N_\text{target}$ (incurring a reward if it does), or recall feedback, in which a randomly chosen item from the target set $N_\text{target}$ is revealed and the learner is told whether it belongs to $N_t$ (incurring a reward if it does). The goal is to maximize the cumulative reward over time. This simple online set learning problem abstracts a variety of learning scenarios with precision- and recall-type feedback. We show that a hypothesis class (a family of subsets of the domain) is learnable in this setting if and only if it has finite Vapnik-Chervonenkis (VC) dimension, mirroring the classical PAC characterization. However, the resulting algorithmic structure is markedly more intricate: in contrast to standard Probably Approximately Correct (PAC) learning -- where the algorithmic landscape is governed by the simple principle of Empirical Risk Minimization (ERM) -- our partial feedback model can invalidate ERM and even all proper learning rules. We develop algorithms to address the dependencies induced by the feedback, obtaining regret guarantees in both the realizable and agnostic settings. Our results provide a qualitative characterization of learnability in this model, addressing its most basic question, while pointing to a range of natural and intriguing open questions, including the determination of optimal regret rates.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces an online learning setting in which a learner repeatedly predicts a set N_t and receives, with equal probability, either precision feedback (a random element from N_t is revealed and a reward is given if it belongs to the unknown target set N_target) or recall feedback (a random element from N_target is revealed and a reward is given if it belongs to N_t). The central result is that a hypothesis class of subsets is learnable (admits an algorithm with sublinear regret) if and only if it has finite VC-dimension. The manuscript supplies improper algorithms that achieve this in both the realizable and agnostic regimes, together with regret bounds derived from VC uniform convergence and a potential-function argument that accounts for the dependence between the sampled item and the current hypothesis.
Significance. If the claimed equivalence holds, the work supplies a qualitative characterization of learnability that directly parallels the classical PAC theorem yet requires substantially more intricate algorithmic machinery; in particular, it demonstrates that ERM and all proper learners can be invalidated by the adaptive sampling induced by the partial feedback. The result is therefore of conceptual interest for any domain in which precision- or recall-style observations arise, and the explicit construction of non-proper algorithms with provable regret bounds constitutes a concrete technical contribution.
major comments (2)
- [§4.2] §4.2, Algorithm 1 (Realizable case): the potential-function analysis that bounds the dependence between the randomly chosen feedback type and the current hypothesis appears to rely on a uniform-convergence argument over the shattered sets; a concrete walk-through of how the VC-dimension bound is applied to the adaptive sampling distribution would strengthen the claim that the regret remains O(√T).
- [Theorem 3] Theorem 3 (Agnostic regret bound): the statement that the algorithm achieves sublinear regret without assuming realizability is load-bearing for the “if” direction; the proof sketch in the appendix should explicitly separate the bias term arising from the agnostic case from the variance term induced by the random feedback type.
minor comments (3)
- [§2] The notation for the target set (N_target) and the predicted set (N_t) is introduced in the abstract but is not restated at the beginning of §2; a single sentence reminding the reader of the two random variables would improve readability.
- [Figure 1] Figure 1 caption refers to “the dependence graph” without defining the nodes or edges; adding a brief legend or a sentence in the text would clarify the diagram.
- [§6] The open questions listed in §6 are interesting but are stated at a high level; indicating which of them are already resolved by the regret bounds in Theorems 2 and 3 would help readers gauge the remaining gaps.
Simulated Author's Rebuttal
We thank the referee for their careful reading, positive evaluation, and constructive suggestions. We address each major comment below and will incorporate the requested clarifications into the revised manuscript.
read point-by-point responses
-
Referee: [§4.2] §4.2, Algorithm 1 (Realizable case): the potential-function analysis that bounds the dependence between the randomly chosen feedback type and the current hypothesis appears to rely on a uniform-convergence argument over the shattered sets; a concrete walk-through of how the VC-dimension bound is applied to the adaptive sampling distribution would strengthen the claim that the regret remains O(√T).
Authors: We agree that an explicit walk-through will strengthen the exposition. In the revision we will expand the analysis in §4.2 to provide a step-by-step derivation: we first recall the uniform-convergence guarantee over all sets shattered by the hypothesis class (of size at most the shattering coefficient), then show how the current hypothesis and the random feedback type induce a distribution over items whose deviation from the uniform measure on shattered sets is controlled by the same VC bound. The resulting additive term in the potential-function decrease is therefore O(√(d log T / t)) per round, which sums to O(√T) overall and preserves the claimed regret. revision: yes
-
Referee: [Theorem 3] Theorem 3 (Agnostic regret bound): the statement that the algorithm achieves sublinear regret without assuming realizability is load-bearing for the “if” direction; the proof sketch in the appendix should explicitly separate the bias term arising from the agnostic case from the variance term induced by the random feedback type.
Authors: We will revise the appendix proof of Theorem 3 to make this separation explicit. The regret will be decomposed as E[regret] = bias term (from the agnostic gap between the best fixed hypothesis and the target) + variance term (from the random choice of precision versus recall feedback). The bias term is bounded via the standard agnostic uniform-convergence rate for VC classes, while the variance term is controlled by a martingale argument that accounts for the dependence between the sampled item and the current hypothesis; each component is shown to be O(√T), yielding the overall sublinear bound without realizability. revision: yes
Circularity Check
No circularity: standard VC necessity plus independent algorithmic sufficiency
full rationale
The necessity direction ('only if finite VC') is the classical shattering argument that already forces linear regret under full supervision; the paper simply notes it carries over to weaker feedback. The sufficiency direction ('if finite VC') is established by an explicit improper algorithm that maintains a version space, samples adaptively according to the current prediction set and random feedback type, and bounds regret via VC uniform convergence plus a potential that tracks the dependence between observed items and hypotheses. No parameter is fitted to the target regret and then renamed a prediction; no self-citation supplies the central equivalence; the derivation does not reduce to its inputs by construction. The result is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Finite VC dimension is necessary and sufficient for learnability in the defined online feedback model
Reference graph
Works this paper leans on
-
[1]
Advances in Neural Information Processing Systems , pages=
Uniform convergence may be unable to explain generalization in deep learning , author=. Advances in Neural Information Processing Systems , pages=
- [2]
-
[3]
Advances in Neural Information Processing Systems , pages=
Online learning: Random averages, combinatorial parameters, and learnability , author=. Advances in Neural Information Processing Systems , pages=
-
[4]
Conference on Learning Theory , pages=
The extended littlestone’s dimension for learning with mistakes and abstentions , author=. Conference on Learning Theory , pages=
-
[5]
arXiv preprint arXiv:2005.11818 , year=
Proper Learning, Helly Number, and an Optimal SVM Bound , author=. arXiv preprint arXiv:2005.11818 , year=
-
[6]
Relating data compression and learnability , author=. 1986 , publisher=
work page 1986
-
[7]
The Journal of Machine Learning Research , volume=
Refined error bounds for several learning algorithms , author=. The Journal of Machine Learning Research , volume=. 2016 , publisher=
work page 2016
-
[8]
arXiv preprint arXiv:1909.03547 , year=
Convex Set Disjointness, Distributed Learning of Halfspaces, and LP Feasibility , author=. arXiv preprint arXiv:1909.03547 , year=
-
[9]
Empirical Bernstein Bounds and Sample Variance Penalization
Empirical Bernstein bounds and sample variance penalization , author=. arXiv preprint arXiv:0907.3740 , year=
-
[10]
High-dimensional probability: An introduction with applications in data science , author=. 2018 , publisher=
work page 2018
-
[11]
Liwei Wang , title =. J. Mach. Learn. Res. , volume =. 2011 , url =
work page 2011
-
[12]
Alkis Kalavasis, Grigoris Velegkas, and Amin Karbasi
On the limits of language generation: Trade-offs between hallucination and mode collapse , author=. arXiv preprint arXiv:2411.09642 , year=
-
[13]
Advances in Neural Information Processing Systems , volume=
Language generation in the limit , author=. Advances in Neural Information Processing Systems , volume=
-
[14]
Information and control , volume=
Language identification in the limit , author=. Information and control , volume=. 1967 , publisher=
work page 1967
-
[15]
2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages=
A theory of PAC learnability of partial concept classes , author=. 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS) , pages=. 2022 , organization=
work page 2021
-
[16]
Conference on Learning Theory , pages=
The optimal approximation factor in density estimation , author=. Conference on Learning Theory , pages=. 2019 , organization=
work page 2019
-
[17]
Combinatorial methods in density estimation , author=. 2001 , publisher=
work page 2001
- [18]
-
[19]
Advances in neural information processing systems , volume=
Assessing generative models via precision and recall , author=. Advances in neural information processing systems , volume=
-
[20]
International Journal of Indian Culture and Business Management , volume=
Evaluation of information retrieval: precision and recall , author=. International Journal of Indian Culture and Business Management , volume=. 2016 , publisher=
work page 2016
-
[21]
International conference on machine learning , pages=
Cascading bandits: Learning to rank in the cascade model , author=. International conference on machine learning , pages=. 2015 , organization=
work page 2015
-
[22]
Learning from positive and unlabeled data: A survey , author=. Machine Learning , volume=. 2020 , publisher=
work page 2020
-
[23]
International conference on algorithmic learning theory , pages=
Denis, Fran. International conference on algorithmic learning theory , pages=. 1998 , organization=
work page 1998
-
[24]
Positive and unlabeled examples help learning , author=. Algorithmic Learning Theory: 10th International Conference, ALT’99 Tokyo, Japan, December 6--8, 1999 Proceedings 10 , pages=. 1999 , organization=
work page 1999
-
[25]
Multi-Label Learning with Stronger Consistency Guarantees , author=. 2024 , archivePrefix=
work page 2024
-
[26]
A Review on Multi-Label Learning Algorithms , year=
Zhang, Min-Ling and Zhou, Zhi-Hua , journal=. A Review on Multi-Label Learning Algorithms , year=
-
[27]
Comprehensive comparative study of multi-label classification methods , journal =. 2022 , author =
work page 2022
-
[28]
International Conference on Algorithmic Learning Theory , pages=
Learning from positive and unlabeled examples , author=. International Conference on Algorithmic Learning Theory , pages=. 2000 , organization=
work page 2000
-
[29]
arXiv preprint arXiv:2008.05756 , year=
Metrics for multi-class classification: an overview , author=. arXiv preprint arXiv:2008.05756 , year=
-
[30]
Foundations and Trends in Machine Learning , title =. 2019 , volume =
work page 2019
-
[31]
Bandit Algorithms , publisher=
Lattimore, Tor and Szepesvári, Csaba , year=. Bandit Algorithms , publisher=
-
[32]
Nesime Tatbul and Tae Jun Lee and Stan Zdonik and Mejbah Alam and Justin Gottschlich , title =. Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montr
work page 2018
-
[33]
Precision and recall for regression , author=. Discovery Science: 12th International Conference, DS 2009, Porto, Portugal, October 3-5, 2009 12 , pages=. 2009 , organization=
work page 2009
-
[34]
BoosTexter: A Boosting-based System for Text Categorization , author=. Machine Learning , year=
-
[35]
AAAI’99 workshop on text learning , year=
Multi-label text classification with a mixture model trained by EM , author=. AAAI’99 workshop on text learning , year=
-
[36]
Advances in Neural Information Processing Systems , title =
Elisseeff, Andr\'. Advances in Neural Information Processing Systems , title =
-
[37]
Advances in Neural Information Processing Systems , title =
Petterson, James and Caetano, Tib\'. Advances in Neural Information Processing Systems , title =
-
[38]
Multilabel Classification using Bayesian Compressed Sensing , year =
Kapoor, Ashish and Viswanathan, Raajay and Jain, Prateek , booktitle =. Multilabel Classification using Bayesian Compressed Sensing , year =
-
[39]
Proceedings of the AAAI conference on artificial intelligence , volume=
Precision-recall versus accuracy and the role of large data sets , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
-
[40]
Generalization Bounds for the Area Under the
Shivani Agarwal and Thore Graepel and Ralf Herbrich and Sariel Har. Generalization Bounds for the Area Under the. J. Mach. Learn. Res. , volume =
-
[41]
Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems,
Corinna Cortes and Mehryar Mohri , title =. Advances in Neural Information Processing Systems 16 [Neural Information Processing Systems,
-
[42]
Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems,
Corinna Cortes and Mehryar Mohri , title =. Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems,
-
[43]
Saharon Rosset , editor =. Model selection via the. Machine Learning, Proceedings of the Twenty-first International Conference
-
[44]
The Thirty Seventh Annual Conference on Learning Theory , pages=
Inherent limitations of dimensions for characterizing learnability of distribution classes , author=. The Thirty Seventh Annual Conference on Learning Theory , pages=. 2024 , organization=
work page 2024
-
[45]
arXiv preprint arXiv:2305.16501 , year=
Strategic Classification under Unknown Personalized Manipulation , author=. arXiv preprint arXiv:2305.16501 , year=
-
[46]
Valiant, L. G. , title =. 1984 , publisher =
work page 1984
-
[47]
Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm , author=. Machine learning , volume=. 1988 , publisher=
work page 1988
-
[48]
Blumer, Anselm and Ehrenfeucht, Andrzej and Haussler, David and Warmuth, Manfred K , journal=. Learnability and the. 1989 , publisher=
work page 1989
-
[49]
Information and Computation , volume=
A general lower bound on the number of examples needed for learning , author=. Information and Computation , volume=. 1989 , publisher=
work page 1989
-
[50]
Fundamental Bounds on Online Strategic Classification , booktitle =
Saba Ahmadi and Avrim Blum and Kunhe Yang , editor =. Fundamental Bounds on Online Strategic Classification , booktitle =
-
[51]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Incentive-aware PAC learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[52]
Proceedings of the 24th Annual Conference on Learning Theory , year =
On the Consistency of Multi-Label Learning , author =. Proceedings of the 24th Annual Conference on Learning Theory , year =
-
[53]
Multilabel reductions: what is my loss optimising? , year =
Menon, Aditya K and Rawat, Ankit Singh and Reddi, Sashank and Kumar, Sanjiv , booktitle =. Multilabel reductions: what is my loss optimising? , year =
-
[54]
International Conference on Machine Learning , pages=
Pac-learning for strategic classification , author=. International Conference on Machine Learning , pages=. 2021 , organization=
work page 2021
-
[55]
Proceedings of the 2016 ACM conference on innovations in theoretical computer science , pages=
Strategic classification , author=. Proceedings of the 2016 ACM conference on innovations in theoretical computer science , pages=
work page 2016
-
[56]
Advances in Neural Information Processing Systems , volume=
Learning strategy-aware linear classifiers , author=. Advances in Neural Information Processing Systems , volume=
-
[57]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Learning losses for strategic classification , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[58]
Proceedings of the 22nd ACM Conference on Economics and Computation , pages=
The strategic perceptron , author=. Proceedings of the 22nd ACM Conference on Economics and Computation , pages=
-
[59]
Proceedings of the 2018 ACM Conference on Economics and Computation , pages=
Strategic classification from revealed preferences , author=. Proceedings of the 2018 ACM Conference on Economics and Computation , pages=
work page 2018
-
[60]
Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=
The social cost of strategic classification , author=. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=
-
[61]
Stackelberg games for adversarial prediction problems , author=. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining , pages=
-
[62]
Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=
The disparate effects of strategic manipulation , author=. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=
-
[63]
Advances in Neural Information Processing Systems , volume=
Who Leads and Who Follows in Strategic Classification? , author=. Advances in Neural Information Processing Systems , volume=
-
[64]
Proceedings of the 23rd ACM Conference on Economics and Computation , pages=
Learning in Stackelberg Games with Non-myopic Agents , author=. Proceedings of the 23rd ACM Conference on Economics and Computation , pages=
-
[65]
International Conference on Machine Learning , pages=
Alternative microfoundations for strategic classification , author=. International Conference on Machine Learning , pages=. 2021 , organization=
work page 2021
-
[66]
ACM Transactions on Economics and Computation (TEAC) , volume=
How do classifiers induce agents to invest effort strategically? , author=. ACM Transactions on Economics and Computation (TEAC) , volume=. 2020 , publisher=
work page 2020
-
[67]
Nika Haghtalab and Nicole Immorlica and Brendan Lucier and Jack Z. Wang , title =. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence,
-
[68]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Multiagent evaluation mechanisms , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[69]
arXiv preprint arXiv:2203.00124 , year=
On classification of strategic agents who can both game and improve , author=. arXiv preprint arXiv:2203.00124 , year=
-
[70]
International Conference on Machine Learning , pages=
Information discrepancy in strategic learning , author=. International Conference on Machine Learning , pages=. 2022 , organization=
work page 2022
-
[71]
Adversarial classification , author=. Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining , pages=
-
[72]
Incentive compatible regression learning , journal =. 2010 , author =
work page 2010
-
[73]
Conference on Learning Theory , pages=
Vc classes are adversarially robustly learnable, but only improperly , author=. Conference on Learning Theory , pages=. 2019 , organization=
work page 2019
-
[74]
Gallant, S. I. Optimal linear discriminants. Eighth International Conference on Pattern Recognition. 1986
work page 1986
-
[75]
arXiv preprint arXiv:2302.06025 , year=
Beyond UCB: Statistical Complexity and Optimal Algorithms for Non-linear Ridge Bandits , author=. arXiv preprint arXiv:2302.06025 , year=
-
[76]
Empirical Bernstein Bounds and Sample Variance Penalization , author=. stat , volume=
-
[77]
International Conference on Machine Learning , pages=
Strategic classification with unknown user manipulations , author=. International Conference on Machine Learning , pages=. 2023 , organization=
work page 2023
- [78]
- [79]
- [80]
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.