Pith. sign in

REVIEW 2 major objections 5 minor 30 references

A physics-structured deep equilibrium model solves to its own start point and contributes zero accuracy on two reasoning tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 06:56 UTC pith:V5LEIGF2

load-bearing objection Clean, reproducible negative result: the DEQ is a no-op identity map on two tasks, the anchor story is falsified, and the substitution diagnostic is the real takeaway. the 2 major comments →

arxiv 2607.11116 v1 pith:V5LEIGF2 submitted 2026-07-13 cs.LG cs.AI

The Equilibrium Is the Initialization: Lazy Identity Collapse in Physics-Structured Deep Equilibrium Reasoning

Joyjeet Singh This is my paper
classification cs.LG cs.AI
keywords deep equilibrium modelsport-Hamiltonian networksimplicit layersgradient starvationlazy identity collapsesubstitution testreasoning benchmarksdiagnostic protocol
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep equilibrium models are sold as input-adaptive reasoners: harder problems should take more solver steps, and the final fixed point should be the product of genuine iteration. This paper shows a carefully engineered port-Hamiltonian version of that idea doing none of that work. On ProofWriter entailment and a BFS-verified graph-reachability task, the solved equilibrium equals the learned initialization to numerical precision; skipping the solver entirely changes test accuracy by zero points in eighteen of nineteen runs. The tempting culprit—an anchoring term that both starts and attracts the dynamics—is ruled out by controlled retraining. The real mechanism is gradient starvation: both tasks can already be solved by a direct context pathway, so “compute nothing” is a stable optimum the optimizer reaches by two different routes. Iteration counts ignore ground-truth difficulty, the full apparatus never beats a two-layer MLP, and the standard zeroing ablation is shown to be confounded. The authors turn the failure into a cheap four-test audit that any claimed implicit or recurrent reasoner can run in minutes.

Core claim

Across two reasoning tasks, multiple seeds, and controlled ablation arms, a port-Hamiltonian deep equilibrium model with learned initialization collapses to the identity: the solved equilibrium equals the solver’s start point, and bypassing the solver changes test accuracy by +0.00 percentage points in 18 of 19 runs. The single escape is divergence into decoder-adapted noise. The collapse is caused by gradient starvation along two distinct routes, not by the anchoring term, and the standard zeroing ablation confounds pathway with computation.

What carries the argument

The substitution test: feed the solver’s start point directly to the decoder, skipping the fixed-point solve. It isolates computation from pathway and returns a stable zero where the usual zeroing ablation produces seed-dependent swings as large as +44 pp.

Load-bearing premise

The two chosen tasks can already be solved by a shallow feed-forward read-out of the same frozen context, so the loss never has to reward genuine iteration.

What would settle it

Retrain the same architecture on a task whose direct context pathway is provably insufficient (for example, multi-hop reachability that cannot be read off the frozen embeddings) and check whether the substitution gap remains zero and whether iteration count begins to track ground-truth difficulty.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper reports a carefully controlled empirical study of a port-Hamiltonian deep equilibrium model (DEQ) with learned, goal-conditioned initialization on two binary reasoning tasks (ProofWriter entailment over frozen DeBERTa embeddings and BFS-labeled k-hop graph reachability). Across seeds, an anchor-strength sweep, and three intervention arms (coupled start=anchor, noise-decoupled retrain, and anchor removed), the solved equilibrium equals the solver start point to numerical precision (≈10⁻⁶–10⁻⁷), the substitution test that bypasses the solver entirely yields a +0.00 pp accuracy gap in 18 of 19 runs, iteration counts are uncorrelated with ground-truth BFS distance (r=0.009), and the full apparatus never outperforms a fresh two-layer MLP (or context-only) baseline. The authors falsify the natural “anchor causes collapse” story, identify two gradient-starvation routes, document a minority solver-blow-up regime that produces co-adapted noise, show that zeroing ablations are confounded while substitution is not, and distill a four-test diagnostic protocol (T1–T4). All results are regenerated from released raw logs on a free Colab GPU.

Significance. If the reported measurements hold—which the multi-seed tables, controlled retrain arms, substitution-versus-zeroing comparison, and positive MLP controls make highly plausible under the stated scope—this is a high-value negative result and methodological contribution for the DEQ / looped / recurrent-depth literature. The work explicitly tests and rejects the most tempting mechanistic explanation, distinguishes two degenerate regimes that pooled statistics would hide, and supplies a cheap, transferable audit (especially T1 substitution) that can be run in minutes. Full release of code, configs, seeds, raw logs, and analysis scripts that regenerate every table and figure is a genuine strength and raises the bar for similar cautionary studies. The result is scoped (tasks admit a direct pathway; one architecture family), yet the diagnostic protocol and the demonstration that “compute nothing” is a stable optimum under gradient starvation remain broadly useful.

major comments (2)
  1. Section 5.3 and Limitations: the two pipelines differ in backward-pass coverage of z0 (ProofWriter differentiates through the initializer; the graph pipeline’s custom VJP zeros those cotangents, leaving the initializer untrained). The paper correctly presents this as two routes to the same no-op, but the fully-differentiated “trained initializer + identity solve” configuration is therefore demonstrated on only one of the two tasks. A short additional experiment or clearer quantification of how much of the graph-task collapse is attributable to the untrained start would strengthen the claim that lazy identity collapse is the generic attractor rather than an artifact of gradient isolation.
  2. Section 9 and §§4.5/5.3: both tasks are solvable to high accuracy by a shallow feed-forward readout of the identical context (MLP baseline matches or beats the full apparatus). The paper acknowledges that tasks where the direct pathway cannot solve the problem “may exert the gradient pressure our setup lacks,” yet the cautionary force for the broader “implicit reasoning” literature rests on this point. Adding at least one controlled task on which the MLP baseline fails substantially (or an explicit negative-control experiment that removes the direct context channels) would make the gradient-starvation diagnosis more decisive rather than expected.
minor comments (5)
  1. Figure 1 caption and surrounding text: the y-axis label “mean h* start (train)” is slightly ambiguous; “mean ∥h*−start∥” would match the body text more cleanly.
  2. Table 2: the blow-up seed (seed 3) is correctly highlighted, but a one-sentence note in the caption that zeroing improves accuracy only in that regime would help readers who glance only at the table.
  3. Section 3.1, Eqs. (1)–(2): the spectral-norm clamp ≤0.95 is stated after the equations; moving the clamp description into the equation block or an immediately following sentence would reduce the chance a reader misses a load-bearing implementation detail.
  4. Abstract and §4.1: the phrase “+0 .00 percentage points” contains an extraneous space; cosmetic but repeated.
  5. Appendix A: the two-body energy-drift comparison is useful background; a single sentence in the main text pointing to it would help readers who skip the appendix understand why the port-Hamiltonian structure was chosen despite the eventual no-op result.

Circularity Check

0 steps flagged

No significant circularity: empirical measurements of no-op equilibria and substitution gaps against independent external baselines (BFS distance, fresh MLP, majority floors).

full rationale

The paper is a controlled empirical negative result, not a first-principles derivation. Its central claims (equilibrium equals start point to numerical precision; substitution gap of +0.00 pp in 18/19 runs; apparatus never beats a two-layer MLP; iteration-difficulty correlation r=0.009) are direct measurements on trained models, evaluated against independently defined quantities: BFS-verified distances, a freshly trained MLP baseline on identical frozen embeddings, majority-class floors, and the solver's own start point fed to the decoder. Controlled arms (anchor removal, noise-decoupled retrain, forced displacement) and the two gradient-starvation routes are likewise experimental interventions whose outcomes are reported, not assumed. Citations supply standard background methods (DEQs, Anderson acceleration, port-Hamiltonian structure) and do not load-bear uniqueness theorems or smuggle ansätze that force the no-op conclusion. No parameter is fitted to data and then re-presented as a prediction; no quantity is defined in terms of the result it purports to derive. The four-test diagnostic protocol is distilled post hoc from the observed failures and is falsifiable on other models. The Limitations section itself scopes the claim to the studied design family and tasks, confirming the result is self-contained rather than circular.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The central negative claim rests on standard fixed-point and automatic-differentiation machinery plus a concrete experimental design. Free parameters are ordinary architectural and solver hyperparameters; none are fitted to produce the identity result. The only non-standard entities are the named failure regimes and the diagnostic tests themselves, both of which are operationally defined by the experiments.

free parameters (5)
  • step size epsilon = 0.05
    Fixed at 0.05 in the symplectic-Euler-style update; chosen by hand, not fitted to the collapse result.
  • anchor strength ra = 0.02 (default)
    Default 0.02; swept over {0.005,0.02,0.05,0.1,0.2} and set to 0 in arm C; collapse persists at every value.
  • spectral-norm clamp = 0.95
    Weight matrices of K and U clamped to norm ≤0.95 after every update; a stability hyperparameter.
  • Anderson memory m and iteration cap = m=5, cap=300
    m=5, cap 300; standard solver settings that do not force identity convergence.
  • noise scale sigma in arm B = 2
    sigma=2 used to decouple start from anchor; chosen to produce a measurable displacement norm ≈7.87.
axioms (4)
  • domain assumption A deep equilibrium layer’s output is the fixed point of a learned map, differentiated via the implicit-function theorem / adjoint solve.
    Standard DEQ construction (Bai et al. 2019) used throughout §§3–5; not re-derived.
  • domain assumption Anderson acceleration with the stated mixing and clipping finds a fixed point when one exists within the iteration budget.
    Solver correctness is assumed so that residual and displacement measurements are meaningful (§3.1, Appendix C).
  • domain assumption The port-Hamiltonian structure–dissipation split with learned K, U and context-conditioned damping yields a well-defined continuous-time latent dynamics.
    Taken from the port-Hamiltonian neural-network literature (Desai et al. 2021) and used as the concrete dynamics in Eqs. (1)–(2).
  • ad hoc to paper BFS distance on the synthetic graphs is a valid ground-truth difficulty signal for iteration-count correlation.
    Introduced by the experimental design (§3.2, §4.3); reasonable but specific to this benchmark.
invented entities (3)
  • lazy identity collapse regime independent evidence
    purpose: Name the observed failure mode in which the solved equilibrium equals the solver start to numerical precision and contributes zero accuracy.
    Operationally defined by displacement ≤10^{-6} and substitution gap = 0.00 pp; measured directly, not postulated a priori.
  • solver blow-up regime independent evidence
    purpose: Name the minority failure mode in which the solver diverges and the decoder co-adapts to the resulting noise channel.
    Observed in one of five ProofWriter seeds (displacement 171) and in an earlier gate-variant run; signature is below-chance substitution and accuracy improvement upon zeroing.
  • four-test diagnostic protocol (T1–T4) independent evidence
    purpose: Provide a cheap, architecture-agnostic audit for claimed implicit or recurrent computation.
    Distilled from the experimental analysis (§7); each test is a concrete forward-pass or probe procedure that any lab can run.

pith-pipeline@v1.1.0-grok45 · 16622 in / 3247 out tokens · 42403 ms · 2026-07-14T06:56:22.317389+00:00 · methodology

0 comments
read the original abstract

Deep equilibrium models promise input-adaptive implicit computation: harder problems should demand more solver iterations, and the solved equilibrium should encode the result of genuine iterative inference. We report a cautionary study of a port-Hamiltonian DEQ with a learned initialization on two reasoning tasks -- ProofWriter entailment over frozen DeBERTa embeddings and a BFS-verified graph-reachability benchmark -- in which the implicit computation is a silent no-op. Across tasks, seeds, and controlled ablation arms, the solved equilibrium equals the solver's start point to numerical precision, and bypassing the solver entirely changes test accuracy by +0.00 percentage points in 18 of 19 training runs. Controlled interventions falsify the tempting explanation: removing the anchoring term reproduces every result, and retraining with noise-decoupled starts yields a solver that converges to the noisy start while the decoder learns to ignore it. The single escaping run diverges instead ($\|h^{*}-z_0\|=171$), producing a co-adapted noise channel whose removal improves accuracy. Iteration counts are uncorrelated with ground-truth difficulty ($r=0.009$), and the full apparatus never outperforms a two-layer MLP on either task. We trace the mechanism to gradient starvation along two distinct routes, show that the standard zeroing ablation is confounded and gives wildly seed-dependent answers where the correct substitution test gives a stable zero, and distill a four-test diagnostic protocol for auditing claimed implicit computation. All experiments run on a single free Colab GPU; code, raw logs, and analysis scripts are released.

Figures

Figures reproduced from arXiv: 2607.11116 by Joyjeet Singh.

Figure 1
Figure 1. Figure 1: Training-time displacement between the solved equilibrium and the solver’s start point, [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Substitution and ablation accuracies. Left: graph task (mean [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: iteration histograms (arm B’s bimodality reflects noisy starts that occasionally [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ProofWriter seeds in the (displacement, substitution-gap) plane. Four seeds collapse to [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training curves, graph task, all arms and seeds. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 10 linked inside Pith

  1. [1]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. InAdvances in Neural Information Processing Systems, 2019

    2019

  2. [2]

    Uni- versal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. InInternational Conference on Learning Representations, 2019

    2019

  3. [3]

    Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach.arXiv preprint arXiv:2502.05171, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    Alex Graves. Adaptive computation time for recurrent neural networks.arXiv preprint arXiv:1603.08983, 2016

    Pith/arXiv arXiv 2016

  5. [5]

    Hamiltonian neural networks

    Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, 2019

    2019

  6. [6]

    SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems.Neural Networks, 132:166–179, 2020

    Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems.Neural Networks, 132:166–179, 2020

    2020

  7. [7]

    Desai, Marios Mattheakis, David Sondak, Pavlos Protopapas, and Stephen J

    Shaan A. Desai, Marios Mattheakis, David Sondak, Pavlos Protopapas, and Stephen J. Roberts. Port-Hamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E, 104(3):034312, 2021

    2021

  8. [8]

    Zico Kolter

    Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Multiscale deep equilibrium models. In Advances in Neural Information Processing Systems, 2020

    2020

  9. [9]

    Implicit deep learning.SIAM Journal on Mathematics of Data Science, 3(3):930–958, 2021

    Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning.SIAM Journal on Mathematics of Data Science, 3(3):930–958, 2021

    2021

  10. [10]

    Zico Kolter

    Ezra Winston and J. Zico Kolter. Monotone operator equilibrium networks. InAdvances in Neural Information Processing Systems, 2020. 11

    2020

  11. [11]

    Anderson

    Donald G. Anderson. Iterative procedures for nonlinear integral equations.Journal of the ACM, 12(4):547–560, 1965

    1965

  12. [12]

    Zico Kolter

    Shaojie Bai, Vladlen Koltun, and J. Zico Kolter. Stabilizing equilibrium models by Jacobian regularization. InInternational Conference on Machine Learning, 2021

    2021

  13. [13]

    JFB: Jacobian-free backpropagation for implicit networks

    Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin. JFB: Jacobian-free backpropagation for implicit networks. InProceedings of the AAAI Conference on Artificial Intelligence, 2022. arXiv:2103.12803

    Pith/arXiv arXiv 2022

  14. [14]

    Zico Kolter, and Roger B

    Cem Anil, Ashwini Pokle, Kaiqu Liang, Johannes Treutlein, Yuhuai Wu, Shaojie Bai, J. Zico Kolter, and Roger B. Grosse. Path independent equilibrium models can better exploit test-time computation. InAdvances in Neural Information Processing Systems, 2022

    2022

  15. [15]

    Lagrangian neural networks.arXiv preprint arXiv:2003.04630, 2020

    Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks.arXiv preprint arXiv:2003.04630, 2020

    Pith/arXiv arXiv 2003

  16. [16]

    Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural or- dinary differential equations. InAdvances in Neural Information Processing Systems, 2018. arXiv:1806.07366

    Pith/arXiv arXiv 2018

  17. [17]

    PonderNet: Learning to ponder.arXiv preprint arXiv:2107.05407, 2021

    Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to ponder.arXiv preprint arXiv:2107.05407, 2021

    Pith/arXiv arXiv 2021

  18. [18]

    Can you learn an algorithm? Generalizing from easy to hard problems with recurrent networks

    Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? Generalizing from easy to hard problems with recurrent networks. InAdvances in Neural Information Processing Systems, 2021

    2021

  19. [19]

    End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking

    Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. InAdvances in Neural Information Processing Systems, 2022

    2022

  20. [20]

    Nowak, and Dimitris Papailiopoulos

    Liu Yang, Kangwook Lee, Robert D. Nowak, and Dimitris Papailiopoulos. Looped trans- formers are better at learning learning algorithms. InInternational Conference on Learning Representations, 2024

    2024

  21. [21]

    Hierarchical reasoning model.arXiv preprint arXiv:2506.21734, 2025

    Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi Yadkori. Hierarchical reasoning model.arXiv preprint arXiv:2506.21734, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Less is more: Recursive reasoning with tiny networks.arXiv preprint arXiv:2510.04871, 2025

    Alexia Jolicoeur-Martineau. Less is more: Recursive reasoning with tiny networks.arXiv preprint arXiv:2510.04871, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    The CLRS algorithmic reasoning benchmark

    Petar Veliˇ ckovi´ c, Adri` a Puigdom` enech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The CLRS algorithmic reasoning benchmark. InInternational Conference on Machine Learning, 2022

    2022

  24. [24]

    Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644, 2016

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644, 2016. v1 October 2016

    Pith/arXiv arXiv 2016

  25. [25]

    Designing and interpreting probes with control tasks

    John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of EMNLP-IJCNLP, 2019. 12

    2019

  26. [26]

    Gradient starvation: A learning proclivity in neural networks

    Mohammad Pezeshki, S´ ekou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. InAdvances in Neural Information Processing Systems, 2021

    2021

  27. [27]

    Wichmann

    Robert Geirhos, J¨ orn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020. arXiv:2004.07780

    Pith/arXiv arXiv 2020

  28. [28]

    ProofWriter: Generating implications, proofs, and abductive statements over natural language

    Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. InFindings of the Association for Computational Linguistics: ACL-IJCNLP, 2021

    2021

  29. [29]

    DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. InInternational Conference on Learning Representations, 2023. A Why Physics Structure Seemed Worth Trying The port-Hamiltonian parameterization was not chosen arbitrarily. In a standalone rollout experime...

    2023

  30. [30]

    — roughly twenty orders of magnitude of improvement in energy control. This is the standard, genuine benefit of the inductive bias, and it is precisely why the main text’s finding is worth reporting: stability of the dynamics and usefulness of the equilibrium are different properties, and the first does not purchase the second. B Training Curves and Insta...