REVIEW 4 major objections 5 minor 22 references
Functional Risk Minimization
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Training a network should compare functions, not just outputs, and this paper derives an objective that does exactly that.
desk verdict FRM is a genuinely new objective with a real derivation gap at its heart; worth refereeing, not accepting as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Functional Generative Model (FGM) is the central object: each datapoint $(x_i,y_i)$ is assigned its own function $f_{\theta_i}$ with $y_i=f_{\theta_i}(x_i)$ and $\theta_i \sim P(\cdot|\theta^*)$. The argument is carried by the derived FRM objective, which is then made tractable by a Laplace approximation in parameter space and a first-order Taylor expansion of the network output, $f_{\theta^*+\Delta}(x) \approx f_{\theta^*}(x) + J_{\theta^*}(x)^T \Delta$, reducing the per-point integral to a Gaussian integral over a hyperplane (regression) or half-space (classification). In the small-perturbation limit the objective becomes a weighted least-squares loss with covariance $J_i^T H^{-1}_{f,L} J_i$, which is what the experiments optimize.
What would settle it
Measure the actual per-point parameter perturbations required to fit a deep network to a real dataset (e.g., by solving $\min_{\theta_i: f_{\theta_i}(x_i)=y_i}\|\theta_i-\theta^*\|$ for each point) and compare their size with the curvature scale of the FRM Hessian. If typical perturbations are not small, the Gaussian/linearized objective (3) will diverge from the exact FGM integral (2), and the claimed equivalence and the reported gains should not hold on such data.
Extended reading notes
Core claim
The paper claims that the risk-minimization problem can be reframed in function space: instead of minimizing $L(y, f_{\theta^*}(x))$ over outputs, one minimizes a cross-entropy between the unknown data-dependent distribution over functions $P(\theta)$ and a loss-induced family $Q_{\theta^*}(\theta) \propto e^{-\mathbb{E}_x[L(f_\theta(x), f_{\theta^*}(x))]}$, penalized by the normalizer $-\log Z(\theta^*)$. The resulting FRM objective is $\arg\max_{\theta^*}\sum_i \log \int_{\theta_i: f_{\theta_i}(x_i)=y_i} e^{-\mathbb{E}_x[L(f_{\theta_i}(x), f_{\theta^*}(x))]} d\theta_i - \log Z(\theta^*)$, and the paper argues that, for common losses, ERM is a special case in which noise is restricted to output parameters. Under a small-perturbation Laplace approximation this objective becomes a weighted regression or a Gaussian-log-cdf classification loss, and the paper reports that this new objective yields better test performance than ERM in its experiments.
Load-bearing premise
Everything in the tractable FRM objective rests on the assumption that each training point can be fit by a very small perturbation of the network's weights, so that a first-order Taylor expansion of the function and a Laplace approximation of the integral are accurate; the paper itself notes this approximation does not justify accuracy over the entire integral.
Editorial extensions
If this is right
- If FRM is correct, training objectives for over-parameterized networks should include a term measuring how easily each training point can be fit by a small parameter perturbation, not just the output error.
- For common losses (MSE, L1, accuracy, cross-entropy), ERM becomes a restricted case of FRM in which only output-side parameters vary, so FRM is a strict generalization of standard practice.
- In the approximate form, FRM can be interpreted as explicitly finding the simplest hyper-model $\{\theta_1,\dots,\theta_n\}$ fitting the data, offering a concrete explanation for why over-parameterized ERM generalizes: it implicitly performs the same kind of minimization.
- The variational approximation for LLMs suggests functional noise can be modeled in adaptation space (prompts, low-rank updates), which could make FRM-style objectives scalable to large language models.
- The reported experiments imply that the benefits are largest when data variability is structured (translation, color, slope noise) rather than i.i.d. output noise, so FRM should be preferred on diverse real-world datasets.
Reading between the lines
- A direct extension the authors do not pursue: the Hessian in the FRM objective could be estimated online during training, turning FRM into a practical regularizer that does not require a second pass or full Hessian inversion.
- The framework suggests a testable criterion for when to prefer FRM over ERM: estimate whether per-point function variability (e.g., style, pose, color) dominates output noise; if it does, FRM's objective should give larger gains.
- Because FRM models each point by its own function, it may naturally handle non-stationary or out-of-distribution data where a single function cannot fit all points, connecting to test-time adaptation without an explicit fine-tuning step.
- The Laplace approximation restricts the framework to small perturbations; an empirical study measuring actual per-point perturbation sizes in trained deep networks would delimit the regime where the derived objective is a faithful surrogate for the true FGM likelihood.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Functional Risk Minimization (FRM), an alternative to empirical risk minimization in which each training point is assigned its own latent function f_{θ_i} drawn from a learned distribution over functions, and the training objective compares functions rather than outputs. The authors derive a functional-risk objective, provide a variational approximation for language models and a Taylor/Laplace approximation for over-parameterized models, prove a universal distribution theorem for FGMs, argue that common ERM losses are subsumed as special cases, and report experiments in linear regression, offline value estimation, and representation learning for MNIST variants.
Significance. If the central derivation were sound, FRM would be a broad and conceptually interesting alternative to ERM with potential implications for modeling structured noise and for understanding generalization in over-parameterized networks. The paper has real strengths: the FGM formulation is clearly described, the connections to hierarchical Bayes, meta-learning, and the NTK literature are useful, and the universal distribution theorem (Appendix A) and the loss-subsumption lemmas (Appendix B) are nontrivial. The experiments, however, are small-scale and do not compensate for the flaws in the derivation. The paper is clearly written and the approximation strategy is creative, but its load-bearing theoretical claim is not established.
major comments (4)
- [§4.1, Eqs. (1)–(2)] The derivation from expected risk minimization to the FRM objective is not valid. Equation (1) is an identity: substituting Qθ* into H(P(θ), Qθ*(θ)) − log Z(θ*) gives exactly E_{θ∼P}[E_x L(f_θ(x), f_θ*(x))], so its argmin is the expected-risk minimizer regardless of the choice of Q. Equation (2), however, is not an estimator of that expectation; it is the log marginal likelihood of the observed labels under the model θ_i ∼ Qθ* with the per-point hard constraints f_{θ_i}(x_i) = y_i. No argument is given that maximizing this misspecified likelihood converges to the minimizer of expected risk, and the replacement of the unknown P(θ) by Qθ* is an unstated modeling assumption. The claimed derivation therefore fails at its first step.
- [§4.1, Eq. (2)] The normalizer in Eq. (2) is wrong by a factor of n. If the per-point parameters θ_i are independent draws from Qθ*, the full-data likelihood is ∏_i (1/Z(θ*)) ∫_{f_{θ_i}(x_i)=y_i} e^{-E_x L(f_{θ_i}(x), f_θ*(x))} dθ_i, so the objective should end with −n log Z(θ*) rather than −log Z(θ*). Since this term is described as a label-independent regularizer and is inherited by the approximations in §4.3, the error changes the balance between the data term and the regularizer and can alter the location of the optimum.
- [§4.3] The tractable Taylor/Laplace approximation requires that the parameter perturbation Δ_i needed to fit each point be small (|Δ_i| << 1) and that f be approximately linear in Δ_i within that range, but the paper never verifies this regime for the models and datasets used in the experiments. The footnote on page 7 concedes that the small-perturbation assumption 'does not justify that this is an accurate approximation of the entire integral.' Because the regression objective (4), the classification log-cdf objective, and the interpretation in §4.4 all inherit this approximation, the empirical claims are not backed by a validated approximation.
- [§4.3, Eqs. (3)–(4)] The transition to the regression objective is incomplete. Equation (3) still contains a factor Z(θ*) inside the integrand while the outer −log Z(θ*) term is dropped, and the Gaussian density used to obtain Eq. (4) also carries a θ*-dependent normalizer (proportional to |H_{f,L}|^{1/2}) that is omitted. Since the paper explicitly states that both H_{f,L} and J_i depend on θ*, these omissions are not harmless constants and can change the optimum. The regression objective as written is therefore not a faithful consequence of the Laplace approximation.
minor comments (5)
- [Abstract and §5] The abstract claims 'better performance in supervised, unsupervised, and RL experiments,' but the experiments are limited to linear regression, mountain car value estimation, and MNIST-based VAE representations; the RL experiment is offline value estimation rather than policy learning, so the claim is stronger than the evidence.
- [§3.1 and references] There are typographical errors, including 'Altough' in §3.1 and an inconsistent citation to 'LeCun (1998)' versus 'LeCun et al. (1998)' in the references and figure captions.
- [§4.2] The variational approximation for LLMs is described only at a high level; the distributions P_z and G in the expression for θ are not defined, and the claim that the prompt-based adaptation is 'closely connected to classic VAEs' would benefit from an explicit derivation.
- [Appendix B] The lemmas showing that MSE, L1, accuracy, and cross-entropy are subcases of functional losses use unnormalized densities (e.g., F ∝ e^{-|β−β_i|^2}); the role of the normalizer, which is central to the FRM objective, is not addressed, so the claimed subsumption is only shown for an informal version of the functional-loss objective.
- [§5.1] Figure 7 lacks details about dataset sizes, the precise noise distributions used, and whether the reported ratios are averages over many trials; without this information the magnitude of the reported gains is difficult to assess.
Circularity Check
Eq. (2) is a definition of the FRM objective under the FGM assumption, not a consequence of Eq. (1); the paper's claimed derivation from expected-risk minimization is therefore self-definitional at the central step.
-
renaming known result
[Section 4.1, Eq. (1)]
"arg min θ∗ E_x,y[L(y,fθ∗(x))] = arg min θ∗ H(P(θ),Qθ∗(θ)) − log(Z(θ∗)). (1) with H(P,Q) being the H cross-entropy operator and Qθ∗(θ) = e^{−E_x L(fθ(x),fθ∗(x))}/Z(θ∗), Z(θ∗) = ∫θ e^{−E_x L(fθ(x),fθ∗(x))} dθ being a class of probability distributions and their normalizers."
Substituting the definitions of Q and Z makes Eq. (1) an identity: H(P,Qθ*) − log Z(θ*) = E_{θ∼P} E_x[L(fθ(x), fθ*(x))], which is exactly the expected risk when y = fθ(x). The equation therefore adds no variational or distributional content; it merely rewrites the risk as a cross-entropy minus a normalizer. The subsequent interpretation that the first term encourages matching two distributions in function space is a relabeling of the original risk, not a new derivation, and the choice of Q is irrelevant to the argmin.
-
self definitional
[Section 4.1, Eq. (2)]
"We approximate this by instead computing the cross-entropy on P(y|x) following the functional generative model. Thus, for a given dataset Dtrain = ((xi,yi))n i=1 the FRM objective is: arg max θ∗ ∑(xi,yi) log ∫θi:fθi(xi)=yi e^{−E_x[L(fθi(x),fθ∗(x)]} dθi − logZ(θ∗) (2)"
Eq. (2) is introduced as a replacement for the uncomputable cross-entropy H(P(θ),Qθ*) but it is not an estimator of that quantity. It is the per-point log marginal likelihood of the labels under the FGM prior θi ∼ Qθ*, i.e., exactly the FGM generative model from which the framework is named. The paper defines this expression as 'the FRM objective' and then presents it as derived from risk minimization; the derivation reduces to stipulating the FGM likelihood. The normalizer is also stated as −log Z(θ*) instead of −n log Z(θ*), so as written Eq. (2) is not even the correct n-point likelihood.
full rationale
The paper contains substantial independent content: Theorem 1 has a constructive proof, Appendix B gives valid equivalence proofs for ERM losses as output-offset FGMs, and the Taylor/Laplace approximation in Section 4.3 is a self-contained tractability device. There is no load-bearing self-citation: the cited prior work by the authors is contextual, and no uniqueness theorem is imported. However, the central claim that FRM is derived from expected-risk minimization fails at the first transition. Eq. (1) is an identity for every choice of Q, so it carries no functional-risk content. Eq. (2) is a new definition of the FRM objective as FGM maximum likelihood, not a consequence of Eq. (1); it replaces the unknown P(θ) with per-point constraints by stipulation. The 'better performance' experiments then test this stipulated objective against ERM on data whose noise is generated to match the FGM assumption, which is a sanity check rather than an independent prediction. The included footnote on the Laplace approximation ('does not justify that this is an accurate approximation of the entire integral') is an honest limitation but does not repair the definitional character of Eq. (2). Overall, the framework is a legitimate proposal, but its central derivation reduces to a definition, giving partial circularity.
Assumptions & free parameters
free parameters (1)
- Functional noise prior parameters (variational LLM approximation) =
not reported
assumptions (4)
- domain assumption Each data point (x_i,y_i) is generated from its own function f_{θ_i} with y_i = f_{θ_i}(x_i) and θ_i ~ P(θ) (FGM assumption).
- ad hoc to paper The parameter perturbation Δ_i required to fit each point is small, and f is approximately linear in parameters within that range (equation 3).
- standard math ERM with common losses is equivalent to maximum likelihood with an output-space noise model (Sec. 2.1, footnote).
- standard math q has continuity conditions in Theorem 1 (x ↦ q(Y∈by|X=x) and α ↦ q((1+α)Y∈by|X=x) continuous).
invented entities (2)
-
Per-data-point latent function θ_i
-
Functional Generative Model (FGM) probability space
Cite this review
Pith. "Pith review of Functional Risk Minimization." pith.science (2026). https://pith.science/paper/6V2PXPY6
@misc{pith2026241221149,
author = {Pith},
title = {Pith review of: Functional Risk Minimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/6V2PXPY6}},
note = {Machine review of arXiv:2412.21149}
}
abstract
The field of Machine Learning has changed significantly since the 1970s. However, its most basic principle, Empirical Risk Minimization (ERM), remains unchanged. We propose Functional Risk Minimization~(FRM), a general framework where losses compare functions rather than outputs. This results in better performance in supervised, unsupervised, and RL experiments. In the FRM paradigm, for each data point $(x_i,y_i)$ there is function $f_{\theta_i}$ that fits it: $y_i = f_{\theta_i}(x_i)$. This allows FRM to subsume ERM for many common loss functions and to capture more realistic noise processes. We also show that FRM provides an avenue towards understanding generalization in the modern over-parameterized regime, as its objective can be framed as finding the simplest model that fits the training data.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[3]
Proof As shown in (Jang et al., 2016; Maddison et al.,
For any arbitrary function classfθ,β(x) expressible asfθ,β(x) =fθ(x) +β, β∈ RC, there exists a functional loss restricted to functional adaptationsθi =θ that only changeβ→βi which is equivalent to the cross-entropy loss. Proof As shown in (Jang et al., 2016; Maddison et al.,
work page 2016
-
[4]
Efficient and modular implicit differentiation.arXiv preprint arXiv:2105.15183,
Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe Llinares-López, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation.arXiv preprint arXiv:2105.15183,
-
[9]
Recasting gradient-based meta-learning as hierarchical bayes.arXiv preprint arXiv:1801.08930,
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes.arXiv preprint arXiv:1801.08930,
-
[11]
Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,
-
[12]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. InAdvances in neural information processing systems, pp. 2017–2025,
work page 2017
-
[14]
Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144,
-
[15]
Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114,
-
[16]
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712,
Show all 22 references
-
[17]
Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003,
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt.Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003,
2021
-
[19]
12 Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar
URL http://arxiv.org/abs/1803.02999. 12 Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning iii: explaining the non-overfitting puzzle.arXiv preprint arXiv:1801.00173,
-
[22]
Adaptive risk minimization: A meta-learning approach for tackling group shift.arXiv preprint arXiv:2007.02931, 8:9,
Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift.arXiv preprint arXiv:2007.02931, 8:9,
2007 arXiv
-
[1969]
Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization.arXiv preprint arXiv:2006.10726,
2006 arXiv
-
[1994]
Test-time training for out-of-distribution generalization
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. arXiv preprint arXiv:1909.13231,
1909 arXiv
-
[1997]
Meta-learning in neural networks: A survey.arXiv preprint arXiv:2004.05439,
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey.arXiv preprint arXiv:2004.05439,
2004 arXiv
-
[2015]
steerability
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks.arXiv preprint arXiv:1907.07171,
1907 arXiv
-
[2016]
Invariant risk minimization.arXiv preprint arXiv:1907.02893,
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization.arXiv preprint arXiv:1907.02893,
1907 arXiv
-
[2017]
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635,
-
[2018]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al
URL http://github.com/google/jax. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing s...
1901
-
[2019]
Deep pnml: Predictive normalized maximum likelihood for deep neural networks.arXiv preprint arXiv:1904.12286,
Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pnml: Predictive normalized maximum likelihood for deep neural networks.arXiv preprint arXiv:1904.12286,
1904 arXiv
-
[2020]
Noisy networks for exploration.arXiv preprint arXiv:1706.10295,
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration.arXiv preprint arXiv:1706.10295,
-
[2021]
Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412,
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization.arXiv preprint arXiv:2010.01412,
2010 arXiv
-
[2022]
Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
Ferran Alet, Maria Bauza, Kenji Kawaguchi, Nurullah Giray Kuru, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time. arXiv preprint arXiv:2009.10623,
2009 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.