REVIEW 4 major objections 5 minor 1 cited by
Optimal Regularization for Performative Learning
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper shows that in performative ridge regression the optimal regularization is set by the mean performative effect, and that in the overparameterized regime performativity that reinforces existing trends can improve the optimally regu
desk verdict Candid take: solid and novel theory for performative ridge regression under a linear label-shift model, but the practical/actionability claims outrun what Assumption 1 supports. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the performative fixed point and its deterministic equivalent. In the population regime, the retraining map has a fixed point whose excess risk is expanded in the small matrix D minus the regularizer; the leading quadratic term yields the population formula for the optimal ridge. In the proportional regime, the random risk after two retraining steps is approximated by a deterministic equivalent that depends only on the covariance, the performative matrix, and the ridge parameter, with a scalar tau solving a fixed-point equation. Expanding that deterministic equivalent at small performativity and small cross-correlation between predictive and spurious features gives the sign
What would settle it
Simulate the same ridge retraining procedure with Gaussian features but replace the linear label shift with a quadratic one, such as y = x^T theta* + (x^T D theta)^2 + noise, and numerically find the population minimizer of the excess risk as a function of lambda for a fixed mean performative strength; if the minimizer does not track the predicted linear function of bbar, the linear-shift assumption rather than performativity itself is carrying the result.
Extended reading notes
Core claim
Under a linear label-shift model of performativity, repeated ridge retraining converges quickly to a fixed point, and the excess risk at that fixed point has a tractable deterministic approximation. Minimizing that approximation yields explicit formulas for the optimal ridge: in the population regime the optimal penalty is proportional to the average performative strength on predictive features, and the corresponding risk is zero only when the covariance is isotropic and the performative entries are constant. In the proportional regime where p/n exceeds 1, the optimal regularization differs from the non-performative optimum by a term linear in the mean performative strength, with a coefficie
Load-bearing premise
The load-bearing premise is that performativity is a linear label shift of the form x^T D theta with Gaussian features whose distribution does not depend on the model, theta_0 fixed with half-zero coordinates; if real performativity changes the feature distribution or is nonlinear in theta, the explicit formulas and sign results have no proven analog, and the paper's assertion that this restriction can always be enforced through pre-processing is unsupported.
Editorial extensions
If this is right
- In the population regime, a practitioner who only knows the average performative strength on predictive features and the covariance Schur complement can set the ridge penalty before deployment.
- In the overparameterized regime with low noise, the optimal ridge moves in the same direction as the performative effect on predictive features; with high noise it moves in the opposite direction.
- Whenever performativity reinforces existing trends, the optimally regularized risk in the proportional regime is lower than without performativity; the opposite holds when performativity dampens trends.
- Performative effects on spurious features enter only at second order and are typically too small to matter, which matches the paper's real-data experiments.
- Population performativity cannot be fully corrected by ridge unless the covariance is isotropic and the performative effect is constant across features; otherwise a variance-of-b residual remains.
Reading between the lines
- Editorial extension: the results suggest a practical recipe---estimate one scalar mean performative strength from a few deployments and set the ridge accordingly---but the paper does not analyze estimation error in that scalar, which a follow-up could address.
- Editorial extension: because the risk improvement holds for all noise levels in the overparameterized regime, the framework predicts a performative benefit in exactly the high-dimensional settings characteristic of deep learning; extending beyond ridge to weight decay or early stopping is a natural but untested next step.
- Editorial extension: the model assumes the feature distribution is unchanged by the deployed model; a testable extension is a performative covariance shift, for instance scaling the covariance with the current parameter, to see whether the linear-in-bbar formulas survive.
- Editorial extension: the population formula can yield a negative optimal ridge, which the paper notes but does not discuss how to implement; turning negative penalties into a practical training scheme is an open engineering question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies repeated risk minimization in performative ridge regression under a linear label-shift model (Assumption 1, Eq. (1)): features are Gaussian with fixed covariance Σ, and the performative effect enters only as x^T D θ in the label, with D=diag(b,c) and half of θ*_pop zero. In the population regime the paper derives the RRM fixed point and a first-order risk approximation, yielding the explicit optimal regularization λ*_pop = b̄ d / Tr(S1) (Corollary 2, Eq. (11)). In the overparameterized regime p/n=κ>1, it uses a two-iteration surrogate for the fixed point and a deterministic equivalent to obtain first-order expansions of the optimal ridge and the optimally regularized risk (Theorem 4, Eqs. (22)–(27)), including sign conditions such as B1≥0 at low noise and B2≤0 for all σ, κ. These predictions are illustrated on synthetic data and on Housing and LSAC with a synthetically injected label shift.
Significance. If the results hold, the paper makes a rare quantitative statement in performative learning: the optimal ridge penalty is an explicit function of the scalar performative strength b̄ and the covariance (population regime), or of b̄, c̄, ρ, σ, κ (proportional regime). The derivation is not fitted: λ* is obtained analytically, and the sign conditions (24)–(27) are falsifiable predictions. The population derivation is clean, and the deterministic equivalent for the two-iteration process is a nontrivial application of high-dimensional asymptotics. The main value is as a first-order theory for a restricted model family; the paper's practical genericity claims, however, go beyond what is actually proved or tested.
major comments (4)
- [Section 3, Eq. (1)] The statement that label-shift performativity 'can always be enforced through pre-processing' is unsupported and load-bearing. All central formulas—Corollary 2 (Eq. 11), Theorems 3–4 (Eqs. 16, 22–27)—are proved only for the family y = x^T θ*_pop + x^T Dθ + w, Gaussian x independent of θ, D=diag(b,c). If deployment changes p(x|θ) or the response is nonlinear in θ, recursions (5)/(14) are not the RRM dynamics. Pre-processing cannot enforce the assumption because D(θ) is environmental. Section 6 re-encodes the same model. Recommend removing the 'always' claim and adding out-of-family evidence or an explicit scope limitation.
- [Section 5, Theorem 3 (Eq. 15)] The deterministic equivalent is for R(Σ,θ2,θ*_pop), not for the RRM fixed point. The claim that two iterations reach the fixed point up to O(||D||²_op) is heuristic: no contractivity bound for the random recursion is given, and the O(||D||²) term appears in the theorem's error rather than being controlled by the high-probability bound. Theorem 4 differentiates at D=0, so Eqs. (22)–(27) are first-order small-performativity expansions. The abstract's unqualified 'optimal regularization scales with overall strength' should be qualified as a first-order statement.
- [Section 6] The real-data experiments do not provide an out-of-family validation. The shift is injected 'according to the theoretical model' (x^T Dθ), and Appendix D sets all b values equal. Thus the experiments test robustness to non-Gaussian x and nonlinear x–y relationships, but the performative mechanism remains Eq. (1). The conclusion that the results 'suggest their generality' is not supported. Please reframe the experiments as illustrations of the derived formulas, or add simulations with feature-shift or nonlinear performativity.
- [Section 6, Fig. 4c and Appendix D] The LSAC n=100 experiment is not clearly in the proportional overparameterized regime. In the theory p=2d and κ=p/n>1. LSAC has 22 features, so p is either 22 or 44; with n=100, p/n is 0.22 or 0.44, both below 1. The caption's phrase 'closer to the number of features d=22' does not establish the regime. The paper must state the actual p, d, κ and confirm that the half-zero spurious block is enforced; otherwise Fig. 4c cannot be used to support the large-noise prediction of Theorem 3.
minor comments (5)
- [Abstract / Introduction] 'Without knowing the exact nature of the potential shift' and 'regularization does not depend on a precise characterization of the distribution shift' are too strong; the formulas require the scalar b̄ (and Σ, σ, κ in the proportional regime). Clarify that b̄ must be known or estimated.
- [Reproducibility statement / Appendix D] The reproducibility statement says code is available at the GitHub URL, while Appendix D says code will be released upon acceptance. These statements are inconsistent and should be aligned.
- [Theorem 4, Eq. (22)] The formula σ²_B1(κ) = 1/2 − 7/(18κ) + O(κ^{-2}) is an asymptotic expansion, not a closed-form definition of the threshold. Lemma 12 proves existence and uniqueness; state the expansion as an asymptotic result rather than part of the theorem hypothesis.
- [Eq. (12)] The higher-order risk expression mixes powers of λ and traces; grouping terms by powers of λ would improve readability and make the O(||F||^4_op) truncation easier to verify.
- [Section 6, Housing] The Housing dataset has 8 features. Since the theory sets p=2d, please state how the spurious block is constructed in the real-data experiments; the same clarification is needed for LSAC.
Circularity Check
No significant circularity in the derivation; one self-definitional element in the real-data validation.
-
self definitional
[Section 6, 'Numerical Experiments', first paragraph (see also Assumption 1, Section 3)]
"These experiments allow us to test whether the theory remains predictive when (i) the data is non-Gaussian and θ*_pop is fixed by the task, and (ii) the true relationship between the feature and the target is likely to not be linear. ... Since no dataset currently provides a real performative shift, it must be encoded synthetically. ... shift the samples of the next split according to the theoretical model."
The real-world validation is presented as a test of robustness beyond the Gaussian linear label-shift model, but the performative component of the data is generated by the same label-shift mechanism y = x^Tθ* + x^TDθ + w of Assumption 1 ('according to the theoretical model'). Thus the empirical agreement with Corollary 2 and Theorem 4 is built into the data generator for the performative part; it cannot independently confirm the formulas for feature-distribution shifts or nonlinear performativity. This affects the generality claim, not the derivation of λ* itself.
full rationale
The mathematical core is self-contained: given Assumption 1, the recursion (5)/(14), the fixed point θ∞=(I+λΣ^{-1}-D)^{-1}θ* in (6), the risk expansion (8)/(12), and the deterministic equivalent (16) are derived analytically. Corollary 2 and Theorem 4 minimize the resulting approximate risk; no parameter is fitted to the quantity being predicted, and the dependence of λ* on b̄ is a derived functional relation, not an input renamed as an output. The Ildiz et al. (2025) citation shares author Mondelli, but it is used as a non-performative ridge-regression deterministic-equivalent tool whose assumptions do not include the target performative λ* results, so it is independent support rather than a circular chain. The only circular element is the Section 6 'real-world' validation, which encodes the performative shift using the paper's own theoretical model; this does not affect the mathematical results but weakens the claim that the experiments 'suggest their generality.' Separately, the assertion that the label-shift restriction 'can always be enforced through pre-processing' is unsupported and is a scope/correctness risk, not a circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Assumption 1: y = x^T θ*_pop + x^T D θ + w, x ∼ N(0,Σ) independent of θ, p=2d, θ*_pop = (a,0), D = diag(b,c), ||b||∞,||c||∞ < 1.
- domain assumption Technical regularity: κ, σ, λ ∈ (1/M,M), ||Σ||op, ||Σ^{-1}||op ≤ M, θ*_pop, θ0 ∈ B_p(R).
- standard math Black-box use of Ildiz et al. (2025) Theorem 3 and Han & Xu (2023) distributional characterization.
- ad hoc to paper Label-shift performativity can 'always be enforced through pre-processing'.
Cite this review
Pith. "Pith review of Optimal Regularization for Performative Learning." pith.science (2026). https://pith.science/paper/UIICXHNJ
@misc{pith2026251012249,
author = {Pith},
title = {Pith review of: Optimal Regularization for Performative Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/UIICXHNJ}},
note = {Machine review of arXiv:2510.12249}
}
read the original abstract
In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex dynamic than in classical supervised learning. One should thus not only optimize the model for the current data but also take into account that the model might steer the distribution in a new direction, without knowing the exact nature of the potential shift. We explore how regularization can help cope with performative effects by studying its impact in high-dimensional ridge regression. We show that, while performative effects worsen the test risk in the population setting, they can be beneficial in the over-parameterized regime where the number of features exceeds the number of samples. We show that the optimal regularization scales with the overall strength of the performative effect, making it possible to set the regularization in anticipation of this effect. We illustrate this finding through empirical evaluations of the optimal regularization parameter on both synthetic and real-world datasets.
Figures
Forward citations
Cited by 1 Pith paper
-
Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics
Under Gaussian design with n ≍ d, the empirical distribution of leave-one-out influences for convex M-estimators converges to the pushforward of a four-dimensional Gaussian through an explicit nonlinear map built from...
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[2]
Bartlett, Philip M
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117 0 (48), 2020
2020
-
[3]
Gaming helps! learning from strategic interactions in natural dynamics
Yahav Bechavod, Katrina Ligett, Steven Wu, and Juba Ziani. Gaming helps! learning from strategic interactions in natural dynamics. In Conference on Uncertainty in Artificial Intelligence (AISTATS), volume 130, 2021
2021
-
[4]
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32), 2019
2019
-
[5]
The role of learning algorithms in collective action
Omri Ben-Dov, Jake Fawkes, Samira Samadi, and Amartya Sanyal. The role of learning algorithms in collective action. In International Conference on Machine Learing (ICML), volume 235, 2024
2024
-
[6]
Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization
Simone Bombari and Marco Mondelli. Spurious correlations in high dimensional regression: The roles of regularization, simplicity bias and over-parameterization. In International Conference on Machine Learing (ICML), 2025
2025
-
[7]
Provable benefits of overparameterization in model compression: From double descent to pruning neural networks
Xiangyu Chang, Yingcong Li, Samet Oymak, and Christos Thrampoulidis. Provable benefits of overparameterization in model compression: From double descent to pruning neural networks. In Conference on Artificial Intelligence (AAAI), volume 35, 2021
2021
-
[8]
Dimension free ridge regression
Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, 52 0 (6), 2024
2024
Show all 42 references
-
[9]
Optimal classification under performative distribution shift
Edwige Cyffers, Muni Sreenivas Pydi, Jamal Atif, and Oliver Capp\' e . Optimal classification under performative distribution shift. In Conference on Neural Information Processing Systems (NeurIPS), volume 37, 2024
2024
-
[10]
A model of double descent for high-dimensional binary linear classification
Zeyu Deng, Abla Kammoun, and Christos Thrampoulidis. A model of double descent for high-dimensional binary linear classification. Information and Inference: A Journal of the IMA, 11 0 (2), 2022
2022
-
[11]
Stochastic optimization with decision-dependent distributions
Dmitriy Drusvyatskiy and Lin Xiao. Stochastic optimization with decision-dependent distributions. Mathematics of Operations Research, 48 0 (2), May 2023
2023
-
[12]
Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian
Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. Runaway feedback loops in predictive policing. In Conference on Fairness, Accountability and Transparency (FAccT), volume 81, 2018
2018
-
[13]
Analysis of classifiers’ robustness to adversarial perturbations
Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Analysis of classifiers’ robustness to adversarial perturbations. Machine learning, 107 0 (3): 0 481--508, 2018
2018
-
[14]
The distribution of ridgeless least squares interpolators
Qiyang Han and Xiaocong Xu. The distribution of ridgeless least squares interpolators. arXiv preprint arXiv:2307.02044, 2023
2023
-
[15]
Performative prediction: Past and future
Moritz Hardt and Celestine Mendler - D \" u nner. Performative prediction: Past and future. arXiv preprint arXiv:2310.16608, 2023
2023 arXiv
-
[16]
Surprises in high-dimensional ridgeless least squares interpolation
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. Annals of statistics, 50 0 (2), 2022
2022
-
[17]
High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws
Muhammed Emrullah Ildiz, Halil Alperen Gozeten, Ege Onur Taga, Marco Mondelli, and Samet Oymak. High-dimensional analysis of knowledge distillation: Weak-to-strong generalization and scaling laws. In International Conference on Learning Representations (ICLR), 2025
2025
-
[18]
How to learn when data gradually reacts to your model
Zachary Izzo, James Zou, and Lexing Ying. How to learn when data gradually reacts to your model. In Conference on Uncertainty in Artificial Intelligence (AISTATS), volume 151, 2022
2022
-
[19]
Scaling laws for learning with real and surrogate data
Ayush Jain, Andrea Montanari, and Eren Sasoglu. Scaling laws for learning with real and surrogate data. In Conference on Neural Information Processing Systems (NeurIPS), volume 37, 2024
2024
-
[20]
Towards a statistical theory of data selection under weak supervision
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a statistical theory of data selection under weak supervision. In International Conference on Learning Representations (ICLR), 2024
2024
-
[21]
Multi-agent performative prediction with greedy deployment and consensus seeking agents
Qiang Li, Chung-Yiu Yau, and Hoi-To Wai. Multi-agent performative prediction with greedy deployment and consensus seeking agents. In Conference on Neural Information Processing Systems (NeurIPS), volume 35, 2022
2022
-
[22]
Minimum-norm interpolation under covariate shift
Neil Rohit Mallinar, Austin Zane, Spencer Frei, and Bin Yu. Minimum-norm interpolation under covariate shift. In International Conference on Machine Learing (ICML), volume 235, 2024
2024
-
[23]
Perdomo, and Tijana Zrnic
John Miller, Juan C. Perdomo, and Tijana Zrnic. Outside the echo chamber: Optimizing the performative risk. In International Conference on Machine Learing (ICML), volume 139, 2021
2021
-
[24]
The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime. Annals of statistics, 53 0 (2), 2025
2025
-
[25]
Wirtschaftsprognose: Eine Untersuchung ihrer Voraussetzungen und Möglichkeiten
Oskar Morgenstern. Wirtschaftsprognose: Eine Untersuchung ihrer Voraussetzungen und Möglichkeiten. Springer, 1928. ISBN 978-3709121139
1928
-
[26]
Adhyyan Narang, Evan Faulkner, Dmitriy Drusvyatskiy, Maryam Fazel, and Lillian J. Ratliff. Multiplayer performative prediction: Learning in decision-dependent games. Journal of Machine Learning Research, 24 0 (202), 2023
2023
-
[27]
Feedback loops with language models drive in-context reward hacking
Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops with language models drive in-context reward hacking. In International Conference on Machine Learing (ICML), volume 235, 2024
2024
-
[28]
Optimal ridge regularization for out-of-distribution prediction
Pratik Patil, Jin-Hong Du, and Ryan J Tibshirani. Optimal ridge regularization for out-of-distribution prediction. In International Conference on Machine Learing (ICML), volume 235, 2024
2024
-
[29]
Performative prediction
Juan Perdomo, Tijana Zrnic, Celestine Mendler-D \"u nner, and Moritz Hardt. Performative prediction. In International Conference on Machine Learing (ICML), volume 119, 2020
2020
-
[30]
Ribeiro, Dave Zachariah, Francis Bach, and Thomas B
Antonio H. Ribeiro, Dave Zachariah, Francis Bach, and Thomas B. Sch \"o n. Regularization properties of adversarially-trained linear regression. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023
2023
-
[31]
Asymptotics of ridge (less) regression under general source condition
Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge (less) regression under general source condition. In Conference on Uncertainty in Artificial Intelligence (AISTATS), volume 130, 2021
2021
-
[32]
Generalization error of min-norm interpolators in transfer learning
Yanke Song, Sohom Bhattacharya, and Pragya Sur. Generalization error of min-norm interpolators in transfer learning. arXiv preprint arXiv:2406.13944, 2024
2024 arXiv
-
[33]
Data feedback loops: Model-driven amplification of dataset biases
Rohan Taori and Tatsunori Hashimoto. Data feedback loops: Model-driven amplification of dataset biases. In International Conference on Machine Learing (ICML), volume 202, 2023
2023
-
[34]
Benign overfitting in ridge regression
Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24 0 (123), 2023
2023
-
[35]
The power of rankings: Quantifying the effects of rankings on online consumer search and choice
Raluca Mihaela Ursu. The power of rankings: Quantifying the effects of rankings on online consumer search and choice. SSRN Electronic Journal, 2015
2015
-
[36]
Network effects in performative prediction games
Xiaolu Wang, Chung-Yiu Yau, and Hoi To Wai. Network effects in performative prediction games. In International Conference on Machine Learing (ICML), volume 202, 2023
2023
-
[37]
On the optimal weighted _2 regularization in overparameterized linear regression
Denny Wu and Ji Xu. On the optimal weighted _2 regularization in overparameterized linear regression. In Conference on Neural Information Processing Systems (NeurIPS), volume 33, 2020
2020
-
[38]
Precise high-dimensional asymptotics for quantifying heterogeneous transfers
Fan Yang, Hongyang R Zhang, Sen Wu, Christopher Re, and Weijie J Su. Precise high-dimensional asymptotics for quantifying heterogeneous transfers. Journal of Machine Learning Research, 26 0 (113), 2025
2025
-
[39]
Performativity and Prospective Fairness
Sebastian Zezulka and Konstantin Genin. Performativity and Prospective Fairness . arXiv preprint arXiv:2310.08349, 2023
2023 arXiv
-
[40]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[41]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[42]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.