REVIEW 1 major objections 7 minor 60 references
Sample averages converge to the true objective in a Lipschitz norm under separability or NIP-definability of the random function family.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 09:54 UTC pith:VKRJBQC3
load-bearing objection A strong, credible paper that proves Lipschitzian SLLNs under separability or NIP-style definability and derives uniform subdifferential convergence; worth serious refereeing. the 1 major comments →
Lipschitzian SLLNs for random functions
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that the sample-average function E_ν f converges to the population objective E f in the Lipschitz pseudometric d_X, defined by d_X(h,g) = max{|h(0)-g(0)|, g-lip_X(h-g)}, under two structural conditions: an explicit separability assumption on the family of slices, or piecewise uniform definability in NIP expansions of the real ordered field. The definability condition holds automatically for jointly definable functions in o-minimal structures, such as semialgebraic and exponential-field-definable losses, and extends beyond them to functions with uniformly or slicewise definable restrictions. The authors prove the almost-sure limit and, under finitely many pieces a
What carries the argument
The central object is the Lipschitz extended pseudometric d_X(h,g) = max{|h(0)-g(0)|, g-lip_X(h-g)}, which makes the space of Lipschitz functions on X a Banach space under the associated Lipschitz norm. The proof reduces convergence in this metric to a Glivenko–Cantelli property of the class of difference quotients g_{x,y}(ξ) = (f(ξ,x)-f(ξ,y))/||x-y|| indexed by a countable dense subset of X. Under NIP definability, the subgraphs of these quotients form a VC-class, so uniform entropy bounds and empirical process theory apply; under separability, the slices form a Bochner-integrable random variable in a separable Banach space, reducing the claim to a classical Banach-valued strong law.
Load-bearing premise
The results hinge on Assumption 3.7, which requires that on each piece of a countable cover of the sample space, the values of every slice on a fixed countable dense set be simultaneously definable by a single formula in some NIP structure; without this uniformity the VC-class reduction and the Glivenko–Cantelli argument collapse, and the alternative separability condition excludes even simple functions like |x-ξ|.
What would settle it
Simulate the nonseparable but semialgebraic case f(ξ,x)=|x-ξ| with ξ uniform on [0,1] and X=[0,1]; the theorem predicts d_X(E_ν f, E f)→0 at rate O(ν^{-1/2}). If a direct computation or simulation shows the distance remaining bounded away from zero, the definability-based theorem is false. Conversely, construct slices that are pointwise definable but not uniformly definable on any countable dense set, violating Assumption 3.7's uniformity, and check whether convergence fails, as the authors conjecture it does.
If this is right
- Empirical (sample-average) objectives converge to the population objective in a metric that controls both function values on bounded sets and global Lipschitz constants, for broad classes of nonconvex nonsmooth stochastic programs.
- Uniform Hausdorff convergence of limiting and Clarke subdifferentials holds without interchanging expectation and subdifferentiation, yielding consistency of limiting stationary points for irregular locally Lipschitz functions.
- Under sharpness, the set of minimizers (and of stationary points) of the sample-average problem is eventually contained in the true solution set, after finitely many samples, without requiring convexity, polyhedrality, or finite support of the distribution.
- The convergence rate is O_P(ν^{-1/2}) when the definable cover has finitely many pieces and second moments exist; otherwise convergence can be arbitrarily slow, as shown by a missing-mass construction.
- Many deterministic tame classes of functions arising in modern learning, such as losses definable in the real exponential field, satisfy the Lipschitzian SLLN, ruling out previously known failure phenomena.
Where Pith is reading between the lines
- A natural extension is that the same Lipschitz pseudometric controls other first-order objects, such as metric slopes and generalized directional derivatives, so analogous SLLNs should follow with essentially the same proofs.
- The NIP-based machinery suggests a testable criterion: a family of slices satisfies the Lipschitzian SLLN as soon as its difference-quotient subgraphs form a VC-class, which one could verify computationally for parametric losses without explicit model-theoretic definability.
- The missing-mass construction implies that moment conditions alone cannot guarantee the root-n rate; rate bounds must incorporate the combinatorial complexity of the function class, paralleling the Glivenko-Cantelli versus Donsker distinction in empirical process theory.
- The paper conjectures that the countability assumptions in the slicewise definability theorem are essential; a concrete test is to seek a counterexample with an uncountable language or uncountable index set, which would sharpen the boundary of the theory.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper establishes strong laws of large numbers for sample-average approximations of random locally Lipschitz functions in the Lipschitz pseudometric d_X(h,g)=max{|h(0)-g(0)|, g-lip_X(h-g)}. Under a blanket integrability/Lipschitz assumption (Assumption 3.1), Theorem 3.3 proves d_X(E_ν f, E f) -> 0 a.s. under a separability condition on the slice family in Lip(X); Theorem 3.8 proves the same under a piecewise uniform NIP-definability condition; Theorem 3.9 adds the rate O_P(ν^{-1/2}) when there are finitely many definability pieces and second moments; Proposition 3.10 shows the rate can be arbitrarily slow with countably many semialgebraic pieces. Theorem 4.10 upgrades slicewise definability in countable-language totally Borel NIP structures, in particular o-minimal slices, to Assumption 3.7. Applications are given to uniform Hausdorff convergence of limiting and Clarke subdifferentials, stationarity consistency, finite-sample identification of minimizers and stationary points, and permanence properties.
Significance. The main positive results are significant. They supply a uniform convergence mode that controls subdifferentials of SAA objectives without interchanging expectation and subdifferentiation, and they cover broad classes of nonconvex nonsmooth objectives, including o-minimal-definable losses and many deep-learning training losses. The proof architecture is sound: Theorem 3.3 is a clean reduction to the Bochner SLLN in separable Banach spaces; Theorem 3.8 reduces the global Lipschitz modulus to Glivenko-Cantelli for VC-subgraph classes of difference quotients; Theorem 3.9 uses a Donsker argument. The delicate uniform-definability requirement in Assumption 3.7 is handled carefully, and Theorem 4.10 gives a valid mechanism for obtaining it from pointwise slicewise definability via a Borel projection argument. The negative baseline Proposition 3.6 and the slow-rate example Proposition 3.10 calibrate the assumptions. One display in the subdifferential application needs correction, but the intended conclusion is recoverable from the proof as written.
major comments (1)
- [Section 5.2, Proposition 5.2; also Eq. (4) in Section 1] The displayed chain is false as stated: d_l(∂h,∂g) for limiting subdifferentials is not generally ≤ d_l(∂̄h,∂̄g) for Clarke subdifferentials; the reverse inequality holds because taking convex hulls cannot increase Hausdorff distance. For example, h(x)=|x| and g(x)=-|x| at 0 give d_l(∂h(0),∂g(0))=1 but d_l(∂̄h(0),∂̄g(0))=0. The proof of Proposition 5.2 actually proves sup_x d_l(∂̄h,∂̄g) ≤ sup_x d_l(∂h,∂g) ≤ sup_x lip(h-g)(x) ≤ d_X(h,g). Please reverse the first inequality in Eq. (4) and in Proposition 5.2. The applications in Corollary 5.3 and the stationarity-consistency results remain valid after this correction, since both subdifferential quantities are still bounded by d_X(h,g).
minor comments (7)
- [Section 2.1] The definitions of g-lip_X h and lip h(x) use the signed quotient (h(x)-h(y))/||x-y|| without an absolute value. Since the supremum is over ordered pairs this is equivalent to the standard absolute-value modulus, but the text should say so explicitly to avoid ambiguity.
- [Section 3.3.1, proof of Theorem 3.3] The sentence 'By restricting the defining supremum to the countable set D' is terse. It is valid because each slice is L(ξ)-Lipschitz on X, so the modulus over X equals the modulus over the dense set D; adding one sentence explaining this would help the reader.
- [Section 3.3.4, Proposition 3.10] In the definition of g_k, the formula appears as ⌊2kt⌋; this should presumably be ⌊2^k t⌋. Please correct the typographical loss of the exponent.
- [Section 4.2.2, Example 4.7] The notation 'Let Z(ξ)=1_Z(ξ)' overloads the letter Z, which is also used for the set of integers. Use something like 'Let z(ξ)=𝟙_ℤ(ξ)'.
- [Section 4.3.3] The subsection title 'Missing details in Remark 4.12' is odd; it is actually a supplement to the remark. Rename it, e.g., 'Supplement to Remark 4.12'.
- [Section 5.2] In the first display of the subdifferential definitions, the Clarke subdifferential is typeset as ∂h(x)=con∂h(x); the overline or other distinguishing notation for the Clarke object should be restored so that the subsequent proposition and proof are unambiguous.
- [Proposition 3.6 and references] The failure baseline relies on the companion preprint [53], and the reader must consult that paper for the construction of f. If a published version of [53] exists, it would be helpful to cite it; otherwise consider stating the needed conclusion more explicitly.
Circularity Check
No significant circularity: central theorems derived from stated assumptions plus external results; self-citation used only as failure baseline.
full rationale
Walking the derivation chain, the positive laws (Theorems 3.3, 3.8, 3.9, and 4.10) follow from the stated assumptions (3.1, 3.2, 3.7, and 4.9) via external results: separable Banach-valued SLLNs, VC/Glivenko–Cantelli/Donsker theory, and NIP/o-minimal definability facts. Assumption 3.7 is a sufficient structural condition, not an encoding of the target convergence; Theorem 4.10 proves Assumption 3.7 from slicewise definability by constructing universally measurable pieces and value-sequence identifiability, and this argument does not presuppose the conclusion. The only self-citation, to the authors' prior negative result [53], is used to establish a failure baseline and to motivate additional assumptions (e.g., Proposition 3.6); it is not an input to the positive law, so it is not load-bearing. Minor notation issues, such as the missing absolute value in the displayed definition of g-lip, do not affect the logical derivation. Therefore, no circular step is present.
Axiom & Free-Parameter Ledger
axioms (10)
- standard math Bochner SLLN for iid Bochner-integrable random variables in separable Banach spaces (Ledoux–Talagrand).
- standard math Glivenko–Cantelli theorem for VC-subgraph classes with integrable envelope, and P-Donsker theorem for VC-subgraph classes with square-integrable envelope (van der Vaart–Wellner).
- standard math NIP formulas define VC classes, and o-minimal structures are NIP (Laskowski; Simon).
- standard math Clarke subdifferential calculus: ∂h(x)⊂∂g(x)+∂(h-g)(x) and ∂(h-g)(x)⊂lip(h-g)(x)B, plus local Lipschitz modulus bounds (Rockafellar–Wets).
- standard math Every o-minimal structure is totally Borel; R_exp is o-minimal; countable-language reducts of R_an are o-minimal (Kaiser; Wilkie).
- domain assumption The negative result in [53, Theorem 3]: under Assumption 3.1 alone, d_X(E_ν f,E_f) may fail to converge, even for convex f.
- domain assumption Assumption 3.1: A-measurability in ξ, E|f(ξ,0)|<∞, and L(ξ)-Lipschitz slices with EL<∞.
- domain assumption Assumption 3.2: {f(ξ,·)|X} lies in a separable subspace of Lip(X).
- domain assumption Assumption 3.7: piecewise uniform definability in NIP structures over a countable dense set.
- domain assumption Assumption 4.9: slicewise definability in countable-language totally Borel NIP structures.
read the original abstract
We prove strong laws of large numbers for locally Lipschitz functions in the Lipschitz pseudometric. Our results hold under either a topological or a model-theoretic condition, with the latter encompassing functions jointly definable in o-minimal structures but extending substantially beyond this class. Applications include uniform convergence of limiting and Clarke subdifferentials and finite-sample identification of solutions. Consequently, we identify broad classes of functions for which the failure phenomena revealed by our previous negative results [Tian and Royset, arXiv:2511.16568, 2025] do not occur.
Reference graph
Works this paper leans on
-
[1]
C. D. Aliprantis and K. C. Border.Infinite Dimensional Analysis: A Hitchhiker’s Guide. Springer, 3rd edition, 2006
2006
-
[2]
F. Areces, J. Duchi, and M. Sommers. Finding a stationary point of a stochastic convex problem.arXiv preprint arXiv:2607.06883, 2026
Pith/arXiv arXiv 2026
-
[3]
Artstein and R
Z. Artstein and R. J-B Wets. Consistency of minimizers and the SLLN for stochastic programs. Journal of Convex Analysis, 2(1-2):1–17, 1995
1995
-
[4]
R. J. Aumann. Integrals of set-valued functions.Journal of Mathematical Analysis and Ap- plications, 12(1):1–12, 1965. 26
1965
-
[5]
G. Bareilles, A. Gehret, J. Aspman, J. Lepˇ sov´ a, and J. Mareˇ cek. Deep learning as the disci- plined construction of tame objects.arXiv preprint arXiv:2509.18025, 2025
Pith/arXiv arXiv 2025
-
[6]
Beer.Bornologies and Lipschitz Analysis
G. Beer.Bornologies and Lipschitz Analysis. CRC Press, 2023
2023
-
[7]
Beer and M
G. Beer and M. J. Hoffman. The Lipschitz metric for real-valued continuous functions.Journal of Mathematical Analysis and Applications, 406(1):229–236, 2013
2013
-
[8]
Berend and A
D. Berend and A. Kontorovich. The missing mass problem.Statistics & Probability Letters, 82(6):1102–1110, 2012
2012
-
[9]
Bolte and E
J. Bolte and E. Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning.Mathematical Programming, 188(1):19–51, 2021
2021
-
[10]
Bolte, A
J. Bolte, A. Daniilidis, A. Lewis, and M. Shiota. Clarke subgradients of stratifiable functions. SIAM Journal on Optimization, 18(2):556–572, 2007
2007
-
[11]
Bolte, T
J. Bolte, T. Le, and E. Pauwels. Subgradient sampling for nonsmooth nonconvex minimization. SIAM Journal on Optimization, 33(4):2542–2569, 2023
2023
-
[12]
J. V. Burke and M. C. Ferris. Weak sharp minima in mathematical programming.SIAM Journal on Control and Optimization, 31(5):1340–1359, 1993
1993
-
[13]
J. V. Burke, X. Chen, and H. Sun. The subdifferential of measurable composite max integrands and smoothing approximation.Mathematical Programming, 181(2):229–264, 2020
2020
-
[14]
C. C. Chang and H. J. Keisler.Model Theory, volume 73. Elsevier, 3rd edition, 1990
1990
-
[15]
Chase and J
H. Chase and J. Freitag. Model theory and machine learning.Bulletin of Symbolic Logic, 25 (3):319–332, 2019
2019
-
[16]
F. H. Clarke.Optimization and Nonsmooth Analysis. SIAM, 1990
1990
-
[17]
D. L. Cohn.Measure Theory. Birkh¨ auser Advanced Texts Basler Lehrb¨ ucher. Springer, 2nd edition, 2013
2013
-
[18]
Coste.An Introduction to O-minimal Geometry
M. Coste.An Introduction to O-minimal Geometry. RAAG Notes. Institut de Recherche Math´ ematiques de Rennes, 1999
1999
-
[19]
Coste.An Introduction to Semialgebraic Geometry
M. Coste.An Introduction to Semialgebraic Geometry. RAAG Notes. Institut de Recherche Math´ ematiques de Rennes, 2002
2002
-
[20]
Davis and D
D. Davis and D. Drusvyatskiy. Graphical convergence of subgradients in nonconvex optimiza- tion and learning.Mathematics of Operations Research, 47(1):209–231, 2022
2022
-
[21]
Davis, D
D. Davis, D. Drusvyatskiy, S. Kakade, and J. D. Lee. Stochastic subgradient method converges on tame functions.Foundations of Computational Mathematics, 20(1):119–154, 2020
2020
-
[22]
Devale, P
T. Devale, P. Devulapalli, and S. Hanneke. Uniform convergence beyond Glivenko-Cantelli. In Matus Telgarsky and Jonathan Ullman, editors,International Conference on Algorithmic Learning Theory, volume 313 ofProceedings of Machine Learning Research, pages 1–21. PMLR, 23–26 Feb 2026. 27
2026
-
[23]
G. B. Folland.Real Analysis: Modern Techniques and Their Applications. John Wiley & Sons, 2nd edition, 1999
1999
-
[24]
D. J. Foster, A. Sekhari, and K. Sridharan. Uniform convergence of gradients for non-convex learning and optimization.Advances in Neural Information Processing Systems, 31:8759–8770, 2018
2018
-
[25]
R. M. Gray and D. L. Neuhoff. Quantization.IEEE Transactions on Information Theory, 44 (6):2325–2383, 2002
2002
-
[26]
G¨ unaydin and P
A. G¨ unaydin and P. Hieronymi. Dependent pairs.The Journal of Symbolic Logic, 76(2): 377–390, 2011
2011
-
[27]
Hieronymi
P. Hieronymi. Defining the set of integers in expansions of the real field by a closed discrete set.Proceedings of the American Mathematical Society, 138(6):2163–2168, 2010
2010
-
[28]
A. D. Ioffe. An invitation to tame optimization.SIAM Journal on Optimization, 19(4):1894– 1917, 2009
1917
-
[29]
Iusem and A
A. Iusem and A. Seeger. Distances between closed convex cones: Old and new results.Journal of Convex Analysis, 17(3-4):1033–1055, 2010
2010
-
[30]
T. Kaiser. First order tameness of measures.Annals of Pure and Applied Logic, 163(12): 1903–1927, 2012
1903
-
[31]
Karpinski and A
M. Karpinski and A. Macintyre. Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks.Journal of Computer and System Sciences, 54(1):169–176, 1997
1997
-
[32]
A. J. Kleywegt, A. Shapiro, and T. Homem-de-Mello. The sample average approximation method for stochastic discrete optimization.SIAM Journal on Optimization, 12(2):479–502, 2002
2002
-
[33]
L. S. Krapp, M. Vermeil, and L. Wirth. On tameness, measurability and the independence property.arXiv preprint arXiv:2506.08733, 2025
Pith/arXiv arXiv 2025
-
[34]
M. C. Laskowski. Vapnik-Chervonenkis classes of definable sets.Journal of the London Math- ematical Society, 2(2):377–384, 1992
1992
-
[35]
Ledoux and M
M. Ledoux and M. Talagrand.Probability in Banach Spaces: Isoperimetry and Processes, volume 23. Springer, 1991
1991
-
[36]
Livni and P
R. Livni and P. Simon. Honest compressions and their application to compression schemes. In Conference on Learning Theory, pages 77–92. PMLR, 2013
2013
-
[37]
Macintyre and E
A. Macintyre and E. D. Sontag. Finiteness results for sigmoidal “neural” networks. InACM Symposium on Theory of Computing, pages 325–334, 1993
1993
-
[38]
S. Mei, Y. Bai, and A. Montanari. The landscape of empirical risk for nonconvex losses.Annals of Statistics, 46(6A):2747–2774, 2018
2018
-
[39]
H. D. Nguyen, J. Westerhout, and X. Guo. On rates of convergence for sample average approximations without smoothness.arXiv preprint arXiv:2604.25153, 2026. 28
Pith/arXiv arXiv 2026
-
[40]
V. I. Norkin and R. J-B Wets. On strong graphical law of large numbers for random semicon- tinuous mappings.Vestnik Sankt-Peterburgskogo Universiteta, Seriya 10(3):102–111, 2013
2013
-
[41]
R. T. Rockafellar. Favorable classes of Lipschitz-continuous functions in subgradient optimiza- tion. In E. Nurminski, editor,Progress in Nondifferentiable Optimization, IIASA Collaborative Proceedings Series, pages 125–144. International Institute for Applied Systems Analysis, Lax- enburg, Austria, 1982
1982
-
[42]
R. T. Rockafellar and R. J-B Wets.Variational Analysis. Springer, 3rd printing-2009 edition, 1998
2009
-
[43]
F. Ruan. On the uniform convergence of subdifferentials in stochastic optimization and learn- ing.Mathematics of Operations Research, to appear, 2025
2025
-
[44]
Schechtman
S. Schechtman. The gradient’s limit of a definable family of functions admits a variational stratification.SIAM Journal on Optimization, 36(2):1075–1099, 2026
2026
-
[45]
Shalev-Shwartz and S
S. Shalev-Shwartz and S. Ben-David.Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014
2014
-
[46]
Shapiro and T
A. Shapiro and T. Homem-de-Mello. On the rate of convergence of optimal solutions of Monte Carlo approximations of stochastic programs.SIAM Journal on Optimization, 11(1):70–86, 2000
2000
-
[47]
Shapiro and H
A. Shapiro and H. Xu. Uniform laws of large numbers for set-valued mappings and subdif- ferentials of random functions.Journal of Mathematical Analysis and Applications, 325(2): 1390–1399, 2007
2007
-
[48]
Shapiro, T
A. Shapiro, T. Homem-de Mello, and J. Kim. Conditioning of convex piecewise linear stochastic programs.Mathematical Programming, 94(1):1–19, 2002
2002
-
[49]
Shapiro, D
A. Shapiro, D. Dentcheva, and A. Ruszczy´ nski.Lectures on Stochastic Programming: Modeling and Theory. SIAM, 3rd edition, 2021
2021
-
[50]
Simon.A Guide to NIP Theories
P. Simon.A Guide to NIP Theories. Lecture Notes in Logic. Cambridge University Press, 2015
2015
-
[51]
C. I. Steinhorn. Chapter XVI: Borel structures and measure and category logics. InModel- Theoretic Logics, volume 8, pages 579–597. Association for Symbolic Logic, 1985
1985
-
[52]
P. Ter´ an. On a uniform law of large numbers for random sets and subdifferentials of random functions.Statistics & Probability Letters, 78(1):42–49, 2008
2008
-
[53]
L. Tian and J. O. Royset. Failure of uniform laws of large numbers for subdifferentials and beyond.arXiv preprint arXiv:2511.16568, 2025
arXiv 2025
-
[54]
A. W. van der Vaart and J. A. Wellner.Weak Convergence and Empirical Processes. Springer, 2nd edition, 2023
2023
-
[55]
van den Dries.Tame Topology and O-minimal Structures, volume 248
L. van den Dries.Tame Topology and O-minimal Structures, volume 248. Cambridge University Press, 1998. 29
1998
-
[56]
van den Dries and C
L. van den Dries and C. Miller. Geometric categories and o-minimal structures.Duke Mathe- matical Journal, 84(2), 1996
1996
-
[57]
A. W. van der Vaart. New Donsker classes.Annals of Probability, 24(4):2128–2140, 1996
1996
-
[58]
A. W. van der Vaart and J. A. Wellner. Preservation theorems for Glivenko-Cantelli and uniform Glivenko-Cantelli classes. InHigh Dimensional Probability II, pages 115–133. Springer, 2000
2000
-
[59]
A. J. Wilkie. Model completeness results for expansions of the ordered field of real num- bers by restricted Pfaffian functions and the exponential function.Journal of the American Mathematical Society, 9(4):1051–1094, 1996
1996
-
[60]
H. Xu. Uniform exponential convergence of sample average random functions under general sampling with applications in stochastic programming.Journal of Mathematical Analysis and Applications, 368(2):692–710, 2010. 30
2010
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.