REVIEW 4 major objections 3 minor 3 references
Momentum Point-Perplexity Mechanics in Large Language Models
T0 review · 4 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that transformer hidden states obey a near-conservation law analogous to energy, and that this invariant yields a minimal-perturbation steering method for guiding generation.
desk verdict The submitted full text is an unrelated CVRP paper, so the LLM energy-conservation claim is unverifiable; the abstract alone is too underspecified to evaluate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a per-token energy $\mathcal{E}(t)$ defined as a scalar function of the hidden-state change rate and the model's next-token certainty, analogous to kinetic-plus-potential energy in physics. The paper views token generation as a trajectory through hidden-state space and writes it in a 'log-Lagrangian' form, where the model's log-likelihood plays the role of a potential. Jacobian steering uses the Jacobian of the logits with respect to the hidden state to find the minimal perturbation direction that raises a target token's probability while keeping $\mathcal{E}$ nearly constant.
What would settle it
Ask the authors for the exact definition of the energy. With that formula in hand, compute it on a held-out transformer and a fixed set of prompts, with the formula chosen before seeing the outputs. If the per-token variance is large, the conservation claim is refuted. If no such fixed formula exists, the claim is not yet a testable one.
Extended reading notes
Core claim
During autoregressive inference, each token step moves the transformer's hidden state. The paper reports that a scalar quantity built from the size of that move and the model's confidence in the next token—called the 'energy'—remains nearly constant across generation. This holds across 20 open-source transformer models spanning 135M to 3B parameters. Random-weight models conserve energy more tightly, while training shifts models into a faster, more decisive regime with larger fluctuation. From a 'log-Lagrangian' formulation of these dynamics, the authors derive Jacobian steering, which perturbs hidden states by the minimal amount needed to push the model's output toward a target token; in tw
Load-bearing premise
The load-bearing premise is that the 'energy' is a genuine, pre-specified invariant rather than a quantity selected after the fact because its variance happened to be low; because the abstract never states the functional form, a reader cannot rule out post-hoc fitting.
Editorial extensions
If this is right
- If the energy is conserved across models and prompts, deviations from conservation become a candidate early-warning signal for unusual or unsafe generation.
- Jacobian steering offers a minimal-intervention way to bias a model's continuation toward a chosen token, with smaller hidden-state changes than typical activation steering.
- The observed training shift—from tight conservation to faster, more variable dynamics—could be used as a diagnostic of how pretraining changes internal processing.
- A conserved scalar for transformer dynamics gives interpretability and anomaly detection a shared currency: one number to monitor per token.
Reading between the lines
- One testable consequence the paper does not explore: if the energy is truly conserved, unusual or adversarial inputs should break conservation more often than ordinary text, making it a possible detector for out-of-distribution or hallucinated tokens. This is our inference, not the paper's claim.
- The mechanics analogy suggests that other conserved quantities (e.g., a 'momentum' vector) may exist; deriving them could yield alternative steering directions beyond the Jacobian of the logits. We are speculating beyond the paper.
- A useful extension left implicit in the paper: apply the energy monitor to a chat or instruction-tuned model and check whether instruction-following changes the conservation profile compared to base pretrained models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The submission is titled 'Momentum Point-Perplexity Mechanics in Large Language Models' and its abstract claims that a scalar 'energy' combining hidden-state rate of change and next-token certainty is nearly conserved across 20 open-source transformer models, that random-weight models conserve this energy more tightly than pretrained models, and that a derived 'Jacobian steering' method can perturb hidden states to favor target tokens while preserving the energy and improving semantic quality. The supplied full text, however, is an unrelated paper on the POMO+ method for the Capacitated Vehicle Routing Problem. No equations, model identifiers, experimental protocols, thresholds, baselines, or statistical tests for the abstract's claims are present in the submission.
Significance. If the energy invariant were genuinely discovered with a prespecified functional form and validated across models, the result could be of interest to the interpretability and steering communities. The claimed connection between a physics-style conserved quantity and a minimal-perturbation control method is conceptually appealing. However, the manuscript as submitted provides no way to evaluate any of these claims: the central equation is absent, the conservation criterion is unspecified, the steering experiments are not described, and the attached full text is a different paper. The submission therefore has no scientifically checkable content, and its potential significance cannot be assessed.
major comments (4)
- [Full text (title/abstract)] The full text supplied with the submission is not the paper described in the abstract. It is titled 'POMO+: Leveraging starting nodes in POMO for solving Capacitated Vehicle Routing Problem' and contains no discussion of hidden states, perplexity, energy conservation, or steering. Thus none of the abstract's central claims are supported by any derivations, experimental data, or code in the manuscript.
- [Abstract, first sentence] The energy functional is never specified. The claim that 'a quantity combining the rate of change in hidden states and the model's next-token certainty' remains nearly constant is untestable unless the functional form, the coefficient values, and the numerical tolerance for 'nearly constant' are given. Without a prespecified functional form, the conservation claim could be a post hoc fit, making the central finding unfalsifiable as presented.
- [Abstract, steering claim] The 'Jacobian steering' method is stated to 'maintained near-constant energy in two tested models and produced continuations rated higher in semantic quality than the models' natural outputs.' No details are given of the two models, the perturbation procedure, the evaluation set, the rating protocol, the baselines, or any statistical uncertainty. As a result, the reported quality improvement is not distinguishable from noise or from a favorable but unspecified evaluation setup.
- [Abstract, random-weight observation] The observation that random-weight models conserve energy more tightly than pretrained models is used to support the invariant interpretation, but without a formula for the energy, this observation could be an artifact of contracting hidden-state dynamics in randomly initialized transformers. A concrete test would be to report the energy functional, show that its parameters are fixed before measuring conservation, and compare the variance of that fixed functional across random and pretrained models.
minor comments (3)
- [Abstract] The abstract uses undefined terms such as 'log-Lagrangian view' and 'momentum point-perplexity energy' without defining them or pointing to a section where they are defined.
- [Full text, general presentation] The unrelated POMO+ manuscript contains typos and stylistic issues ('necesserily', 'strenghten', 'over the past years'). These are not relevant to the stated topic but further indicate that the submission was not prepared for review of the abstract's claims.
- [References] The submitted full text has references only for the CVRP paper; none of the 20 transformer models, the conservation law, or the steering method are cited or described in a reference list.
Circularity Check
No demonstrated circularity: the abstract gives no functional form for the energy and the supplied full text is an unrelated CVRP paper, so no circular reduction can be exhibited.
full rationale
The submitted text does not contain enough formal content to exhibit a circular reduction. The abstract's central energy quantity is never given a functional form ('a quantity combining the rate of change in hidden states and the model's next-token certainty'), so it is impossible to determine whether the constancy claim is a pre-specified physical law or a post-hoc selected statistic. That is an evidence/verifiability gap, not a demonstrated circularity, and hard rule 1 prohibits speculation about post-hoc fitting. The Jacobian steering method is described only verbally; the statement that it 'maintained near-constant energy' could in principle be an internal consistency check, but the abstract does not show that energy preservation was imposed as a constraint in the derivation, so the validation cannot be reduced to the construction. There are no self-citations in the abstract to evaluate. The supplied 'full text' is a different manuscript (POMO+ for CVRP) and therefore contains no derivations for the LLM claims; I flag this as an omitted proof / missing support, but an omitted proof is not circularity. Under the hard rules, no circular step can be quoted, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (1)
- coefficients in the energy functional =
unknown
assumptions (2)
- domain assumption Hidden states of transformers admit meaningful velocity and Jacobian fields in activation space
- ad hoc to paper Physics analogies (energy, Lagrangian) apply to language model inference
invented entities (2)
-
Momentum point-perplexity energy
-
Log-Lagrangian view
Cite this review
Pith. "Pith review of Momentum Point-Perplexity Mechanics in Large Language Models." pith.science (2026). https://pith.science/paper/J777IBRF
@misc{pith2026250808492,
author = {Pith},
title = {Pith review of: Momentum Point-Perplexity Mechanics in Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/J777IBRF}},
note = {Machine review of arXiv:2508.08492}
}
read the original abstract
We take a physics-based approach to studying how the internal hidden states of large language models change from token to token during inference. Across 20 open-source transformer models (135M-3B parameters), we find that a quantity combining the rate of change in hidden states and the model's next-token certainty, analogous to energy in physics, remains nearly constant. Random-weight models conserve this "energy" more tightly than pre-trained ones, while training shifts models into a faster, more decisive regime with greater variability. Using this "log-Lagrangian" view, we derive a control method called Jacobian steering, which perturbs hidden states in the minimal way needed to favor a target token. This approach maintained near-constant energy in two tested models and produced continuations rated higher in semantic quality than the models' natural outputs. Viewing transformers through this mechanics lens offers a principled basis for interpretability, anomaly detection, and low-risk steering. This could help make powerful models more predictable and aligned with human intent.
Reference graph
Works this paper leans on
-
[1]
POMO+: Leveraging starting nodes in POMO for solving Capacitated Vehicle Routing Problem Szymon Jakubicz University of Warsaw, Poland s.jakubicz@student.uw.edu.pl Karol Ku´ zniak University of Warsaw, Poland km.kuzniak@student.uw.edu.pl Jan Wawszczak University of Warsaw, Poland janekw23@gmail.com Paweł Gora Fundacja Quantum AI pawel.gora@qaif.org Abstrac...
work page 1959
-
[2020]
now set the bar for neural VRP heuristics. Guided by these advances, our work reviews state-of-the-art RL solvers, selects the most promis- ing one to improve (POMO), and enhances it with a lightweight auxiliary agent that learns to choose the best starting nodes, which is an open sugges- tion from the authors of the original model. We called our method P...
work page Pith review arXiv 2017
-
[2024]
Leader Reward for POMO-Based Neural Combinatorial Optimization
Leader reward for pomo- based neural combinatorial optimization. Preprint, arXiv:2405.13947
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.