Pith. sign in

REVIEW 4 major objections 3 minor 2 references

Gradient Descent as Implicit EM in Distance-Based Neural Models

T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For log-sum-exp objectives, the gradient with respect to each distance is exactly the negative posterior responsibility of that component—an algebraic identity, not an approximation. Consequently, gradient descent on such objectives perform

desk verdict The algebra is right and the limits section is honest, but the paper's headline claim — that gradient descent on log-sum-exp objectives is EM — is a redefinition, not a derivation. read the letter →

arxiv 2512.24780 v2 pith:5WMFE4JD submitted 2025-12-31 cs.LG

Alan Oursland This is my paper · ORCID
classification cs.LG
keywords log-sum-expexpectation-maximizationgradientdescentsoftmaxposteriorresponsibilitiesattentionmechanismscross-entropyclassificationmixturemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that standard neural-network training objectives—when read as log-sum-exp functions over distances or energies—automatically perform expectation-maximization. The key is a simple identity: the gradient of the loss with respect to a component's distance equals the negative of that component's softmax responsibility. Because backpropagation delivers responsibility-weighted updates, the authors argue that any model trained with these objectives is implicitly running EM, with no separate inference step. A sympathetic reader would care because this would explain why attention heads specialize, why classification networks behave like mixture models, and why transformers can reproduce Bayesian posteriors—without invoking emergence or architectural accident. The paper presents this as a unification of three learning regimes under a single geometric mechanism.

What carries the argument

The central object is the log-sum-exp (LSE) objective L = log Σ_j exp(−d_j), where d_j are distances or energies. The key identity is ∂L/∂d_j = −r_j, with r_j the softmax responsibility. This identity is the machinery: it converts a purely algebraic property of the softmax into a statement about learning dynamics. The paper interprets neural outputs as distances (prior work on Mahalanobis distance), so exp(−d_j) behaves as an unnormalized likelihood, and r_j becomes a posterior probability. The identity is then sufficient to claim that every gradient step is a responsibility-weighted update, which is the defining behavior of EM's M-step. The paper also relies on a semantic shift: probabiliti

What would settle it

Train a transformer on a non-normalized objective (e.g., independent per-token sigmoid loss) in the same Bayesian wind tunnel setting; if it still reproduces the analytic posterior, then the claim that LSE geometry is necessary for Bayesian structure is refuted. Alternatively, compare the parameter-update trajectory of gradient descent on an LSE mixture objective with the explicit EM update trajectory; if they diverge in a way not explainable by learning rate, the 'implicit EM' equivalence is not exact.

Watch

Extended reading notes

Core claim

The paper's central claim is that for any objective of the form L = log Σ_j exp(−d_j), the gradient with respect to the j-th distance is exactly −r_j, where r_j = exp(−d_j) / Σ_k exp(−d_k) is the posterior responsibility of component j. This identity is derived by one application of the chain rule and holds whenever distances are differentiable. The authors then assert that gradient descent on such objectives is generalized expectation-maximization performed continuously: the forward pass computes unnormalized likelihoods and implicitly normalizes them (the E-step), while the backward pass propagates responsibility-weighted gradients to parameters (the M-step). They argue that this mechanism

Load-bearing premise

The load-bearing premise is that neural outputs should be read as distances or energies to learned prototypes, rather than as confidences or scores; if that semantic interpretation fails, the 'posterior responsibility' label is just a softmax weight and the EM claim loses its probabilistic force.

Editorial extensions

If this is right

  • If correct, training any model with a softmax-based loss is doing probabilistic inference, which reframes interpretability: the assignments a network makes are literally present in the gradient signal.
  • The unification predicts that attention, mixture models, and classification heads share identical responsibility-weighted dynamics, so insights from one domain transfer directly to the others.
  • The framework explains why removing normalization (e.g., independent sigmoids) destroys soft assignment, and why objectives like maximum correntropy behave differently: they abandon the assignment structure entirely.
  • The missing volume term in neural objectives explains collapse as an inherent risk of the implicit-EM dynamics, pointing to where regularization or architectural constraints intervene.
  • The closed-world assumption of softmax—every input must be assigned—is a direct consequence of the objective geometry, not an arbitrary design choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension follows from the strong claim: if gradient descent on LSE objectives is truly EM, then the fixed points of training on a Gaussian mixture should match the maximum-likelihood estimates of EM; deviations would reveal where the analogy breaks.
  • The identity suggests that any softmax-normalized layer, including mixture-of-experts gates, is implicitly performing inference; this could lead to new architectures that explicitly control responsibilities rather than leaving them implicit.
  • The paper's own framing implies that the 'Bayesian geometry' seen in transformers could be deliberately turned on or off by choosing normalized versus unnormalized objectives, offering a design lever for uncertainty-aware models.
  • Because the derivation is algebraic, the same identity should hold for any differentiable distance function, including learned metrics or kernel distances—an extension the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper claims to explain EM-like and Bayesian behavior in neural networks by proving a gradient identity: for any objective L = log Σ_j exp(-d_j), the derivative with respect to the j-th distance is the negative softmax responsibility -r_j (Theorem 1, Eq. 5). On this basis it argues that gradient descent on such objectives is exactly/implicitly EM, that attention mechanisms and cross-entropy classification are the same mechanism under different constraints, and that Bayesian structure in trained transformers is a necessary consequence of the objective geometry. The proof of the identity is a correct application of the chain rule, but the paper's central interpretive claims are not supported by the derivation.

Significance. The algebraic identity in Theorem 1 is correct, and the paper is clearly written and candid about some of its limitations (e.g., §6.2, §6.4). If the paper had restricted itself to observing that LSE objectives yield responsibility-weighted gradients, it would be a modest, acceptable observation already present in the energy-based and mixture-modeling literature. As submitted, however, the paper makes much stronger claims: that gradient descent 'is EM,' that optimization and inference 'are the same process,' and that Bayesian structure in transformers is a 'necessary consequence.' These claims are not established. The manuscript therefore does not make a sound case for its headline contributions, despite the correctness of the elementary derivative.

major comments (4)
  1. [§3.2, §3.3, Eq. (5)] The theorem is a textbook softmax derivative, but the inference from the identity to 'gradient descent ... is exactly EM' is not valid. Classical EM, as defined in §2.3, requires an M-step that maximizes the expected complete-data log-likelihood: θ^{t+1} = argmax_θ Σ_i Σ_j r_ij^t log p_j(x_i|θ_j). A gradient descent step is θ^{t+1} = θ^t + η Σ_i Σ_j r_ij^t ∇_θ log p_j(x_i|θ^t). These coincide only for specially chosen step sizes and parameterizations. Section 3.3 weakens 'implicit EM' to mean 'responsibility-weighted parameter updates,' which is already contained in Eq. (5). The abstract, §1.3, and §8 assert a stronger equivalence that is never derived.
  2. [§4.2] Attention mechanisms do not instantiate Theorem 1 as stated. The training objective of a transformer is not L = log Σ_j exp(s_ij) over keys; softmax attention is a network layer, not a loss. The gradient of the task loss with respect to W_V or the attention scores is propagated through o_i = Σ_j α_ij v_j and downstream layers; it is not generally equal to a responsibility-weighted update of distance parameters. The claim in §4.2 that 'each backward pass updates those parameters in proportion to the responsibilities' requires specifying which objective L is being used and deriving the gradient. That derivation is missing, so the 'conditional regime' is not actually covered by Eq. (5).
  3. [Abstract, §5.2, §6.2] The conclusion that Bayesian structure in trained transformers is 'a necessary consequence of the objective geometry' is unsupported. The gradient identity constrains per-step dynamics; it does not by itself imply that fixed points are Bayesian posteriors, that attention weights converge to posterior probabilities, or that inference-time behavior matches the analytic posterior. The paper's own §6.2 notes that neural objectives omit the log-determinant volume term that is essential to Gaussian mixture likelihoods, implying the dynamics are not identical to full Bayesian mixture inference. No argument connects responsibility-weighted gradients to the sub-bit posterior agreement reported in Aggarwal et al. This is a central overclaim, not a minor caveat.
  4. [§4.1, Eq. (6)] There is a sign error in the unsupervised regime. For L = -log Σ_j exp(-d_j), the derivative is ∂L/∂d_j = +r_j, not -r_j. The text says the sign flip reflects switching from maximizing log-likelihood to minimizing its negative, but the displayed equation does not implement that flip: the derivative of -log Z is +exp(-d_j)/Z. As written, gradient descent would increase d_j for high-responsibility components, contradicting the following sentence that prototypes 'move closer.' The correct sign is needed for the claimed M-step dynamics.
minor comments (3)
  1. [§4.1] Equation (6) is called the negative log marginal likelihood, but the LSE objective over distances without a volume term is not the likelihood of a full Gaussian mixture. The acknowledgment in §6.2 should be moved or previewed here to avoid implying a proper mixture model.
  2. [§2.2] The phrase 'This structure appears throughout deep learning... in attention mechanisms' is misleading in a paper whose theorem concerns objectives, not network layers. Attention weights are responsibilities appearing in a forward pass, not necessarily responsibilities arising as gradients of an LSE loss.
  3. [§5.1] The geometric substrate in §2.1 is imported from a self-citation (Oursland 2024) and is described as 'not a modeling choice' but 'a mathematical property.' Since the semantic interpretation is load-bearing for calling r_j a posterior responsibility, the prior work should be summarized sufficiently that this paper is understandable without accessing the unpublished preprint.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 1 is a correct but definitional softmax identity; the central 'implicit EM' conclusion holds only under the paper's own stipulative definition, and the distance semantics are imported from the author's prior work.

  1. self definitional [Section 3.1, Eq. (5); Section 2.2, Eq. (3); Section 3.2]
    "The right-hand side is the negative of the normalized exponential—precisely the responsibility r_i as defined in Section 2.2. This yields the main result. Theorem 1. For any objective of the form L = log Σ_j exp(−d_j), the gradient with respect to the j-th distance is the negative responsibility of component j: ∂L/∂d_j = −r_j where r_j = exp(−d_j)/Σ_k exp(−d_k). ... Throughout this paper, 'implicit EM' refers to the emergence of responsibility-weighted parameter updates under gradient descent on log-sum-exp objectives—not to coordinate-ascent EM or guarantees about convergence."

    Theorem 1 is not an independent prediction: r_j was defined in Eq. (3) as exp(−d_j)/Σ_k exp(−d_k), and L was defined as log Σ_j exp(−d_j). The derivative of log Z with respect to d_j is the normalized exponential by the chain rule, so Eq. (5) restates the softmax definition. The further step, calling the backward pass 'the M-step,' identifies any responsibility-weighted parameter update with EM's M-step; this identification is the paper's own stipulated definition of 'implicit EM,' not a property of classical EM's argmax M-step. Thus the central 'gradient descent is EM' claim is forced by the chosen definition rather than derived from an independent equivalence.

  2. self citation load bearing [Section 2.1 and Section 5.1]
    "In prior work [Oursland, 2024], we established a different interpretation: the outputs of standard neural networks are better understood as distances or energies relative to learned prototypes. ... The results that follow—particularly the identification of gradients with responsibilities—depend on this geometric framing. ... Without the distance interpretation, the identification of gradients with responsibilities is a formal curiosity."

    The semantic premise that makes r_j a 'posterior responsibility' and the update an 'EM M-step' is the distance/energy reading of neural outputs. That premise is imported from the author's own prior work and is not re-derived or independently verified in this paper. The paper explicitly says the gradient identification is only a 'formal curiosity' without this substrate, so the substrate is load-bearing. To the extent that the prior work is itself an interpretive ansatz rather than an externally established fact, the paper's Bayesian/EM interpretation is supported by a self-citation chain.

full rationale

The gradient identity ∂L/∂d_j = −r_j is mathematically correct and self-contained—it is the standard derivative of the log-sum-exp function. The paper is transparent that this is a 'textbook calculation.' However, the paper's advertised consequence, that gradient descent 'is EM' and that 'optimization and inference are the same process,' does not follow from that identity alone. It follows only because the paper defines 'implicit EM' as 'responsibility-weighted parameter updates,' which is exactly what Theorem 1 already gives. Any stronger equivalence to classical EM—whose M-step is an argmax over the expected complete-data log-likelihood, not a generic gradient step—is asserted rather than derived. The distance/prototype semantics needed to call r_j a Bayesian posterior are imported from Oursland 2024, a self-citation, and the paper itself states that without that substrate the result is 'a formal curiosity.' This is partial circularity: the algebra is checkable and the external empirical evidence from Aggarwal et al. is independent, but the central conceptual claim reduces to a definition plus a self-cited interpretive premise.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters or new entities. The framework relies on the distance-based semantics of neural outputs, a uniform-prior convention, differentiability, and gradient-based optimization. The attention mapping requires an additional assumption that value projections, not values, are the persistent prototypes.

assumptions (5)
  • domain assumption Neural outputs are interpreted as distances/energies to learned prototypes.
    Section 2.1 imports this from prior self-cited work [Oursland 2024]; the probabilistic reading of r_j depends on it.
  • domain assumption Uniform prior over components in r_j = exp(-d_j)/Σ exp(-d_k).
    Section 2.2 Eq. (3) defines responsibilities as posterior under a uniform prior; not justified for attention/classification.
  • standard math Distances d_j are differentiable.
    Theorem 1 requires differentiability for the chain rule.
  • domain assumption Parameters are updated by gradient descent.
    Section 3.3 states the implicit-EM dynamics claim only holds for gradient-based optimizers.
  • domain assumption Attention value projections WV are persistent prototypes.
    Section 4.2 posits this to map attention to EM despite transient value vectors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient Descent as Implicit EM in Distance-Based Neural Models." pith.science (2026). https://pith.science/paper/5WMFE4JD

@misc{pith2026251224780,
  author       = {Pith},
  title        = {Pith review of: Gradient Descent as Implicit EM in Distance-Based Neural Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WMFE4JD}},
  note         = {Machine review of arXiv:2512.24780}
}
abstract

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in attention mechanisms, classification heads, and energy-based models -- yet existing explanations often rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct explanation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component: $\partial L / \partial d_j = -r_j$. The identity is algebraic, requiring only differentiability; it is a specialization of Fisher's identity, and its significance here is its address: standard neural objectives instantiate it without modification. The consequence is that gradient descent on such objectives performs generalized expectation-maximization implicitly, with responsibilities arising as gradients to be applied rather than auxiliary variables to be computed. This result unifies three regimes of learning: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. Our claims live at training time: the responsibility-weighted gradient dynamics recently documented in transformers follow from the objective's geometry. The in-context Bayesian computation that trained transformers perform at inference time is the endpoint of these dynamics, not their per-step content.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

  1. [1]

    Dalal, and Vishal Misra

    Naman Aggarwal, Siddhartha R. Dalal, and Vishal Misra. The Bayesian geometry of transformer attention.arXiv preprint arXiv:2512.22471, 2025a. Naman Aggarwal, Siddhartha R. Dalal, and Vishal Misra. Gradient dynamics of attention: How cross-entropy sculpts Bayesian manifolds.arXiv preprint arXiv:2512.22473, 2025b. Arthur P. Dempster, Nan M. Laird, and Donal...

    arXiv

  2. [2006]

    Interpreting neural networks through Mahalanobis distance.arXiv preprint arXiv:2410.19352,

    Alan Oursland. Interpreting neural networks through Mahalanobis distance.arXiv preprint arXiv:2410.19352,

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.