REVIEW 3 major objections 5 minor 1 cited by
LCA: Loss Change Allocation for Neural Network Training
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper introduces Loss Change Allocation, a per-parameter decomposition of each training iteration's loss change, and uses it to claim that just over half of parameters help at any step, some layers consistently hurt, and the biggest…
desk verdict A useful, well-grounded diagnostic for per-parameter credit during training; the per-parameter claims rest on a convention, but the tool and the observations are worth engaging. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the path-integral identity $L(\theta_T)-L(\theta_0)=\int_C \langle \nabla_\theta L(\theta), d\theta\rangle$, approximated one step at a time by first-order Taylor terms $A_{t,i}$. To keep the approximation accurate on curved loss surfaces, the gradient is averaged at the initial, midpoint, and endpoint of each step using the fourth-order Runge-Kutta (RK4) rule, equivalently Simpson's rule, halving the step until the per-iteration error is below 0.001. This machinery converts an otherwise opaque scalar loss curve into an additive, signed, per-parameter credit map, which is what makes the paper's per-weight, per-layer, and per-class claims measurable.
What would settle it
Compute, for one curved training step of a small network, both the LCA value of each parameter and the exact loss difference caused by moving only that parameter along its update while all other parameters are fixed; if the signs disagree for a substantial fraction of parameters, the per-parameter claims fail. Alternatively, integrate the path integral along the same step with very fine quadrature and compare per-component values.
Extended reading notes
Core claim
The central claim is that the change in training loss from one iteration to the next can be partitioned, parameter by parameter, by the product of that parameter's component of the full-training-set gradient and its movement in that step: $A_{t,i} = (\nabla_\theta L(\theta_t))_i(\theta_{t+1}-\theta_t)_i$, with the gradient evaluated at a Runge-Kutta-averaged point so the components sum to the true loss change to within a small error. LCA is signed and additive, so it can be summed over iterations, neurons, channels, or layers. On this basis the paper reports three empirical findings: in a typical iteration only slightly more than half of parameters have negative LCA and thus 'help'; some whole layers, in particular the first and last layers of a CIFAR-ResNet trained with SGD, accumulate positive LCA over training, which the paper attributes to their being phase-lagged relative to other layers; and the top moments of per-layer learning occur at the same iterations for all layers far more often than chance.
Load-bearing premise
The measurement assumes that the per-weight split of the loss change is accurate, but the paper only checks that the components sum to the true loss change; on sharply curved steps, a weight labeled 'helping' could actually be hurting.
Editorial extensions
If this is right
- Because LCA is additive over time and over parameter groups, practitioners can produce a complete per-layer credit map of a training run and use it to decide which layers to freeze, shrink, or leave at their initialization.
- The near-50% helping fraction implies that stochastic-gradient training is far noisier at the parameter level than scalar loss curves suggest; the paper's oscillation counts quantify this, with weight directions changing on average every 6.7 iterations for CIFAR-ResNet with SGD.
- Freezing the last layer of CIFAR-ResNet with SGD, or cutting its learning rate tenfold, turns that layer's LCA from positive to negative and improves both train and test loss, so layer-hurting is not merely a curiosity but a lever on performance.
- The synchronization of peak learning across layers implies that training advances in coherent global pulses; this is measurable from LCA data and is significant against a randomly shifted baseline, with 9.4 aligned peaks in the actual MNIST-FC network versus 0.4 in the baseline.
- The heavy-tailed distribution of LCA values indicates that a Wiener-process picture of learning should be refined to a heavy-tailed process.
Reading between the lines
- If per-parameter LCA signs are accurate, the near-50% helping rate implies the effective signal-to-noise ratio of gradient descent is far lower than commonly assumed; a testable extension is to mask out the hurting half of the weights each step and see how convergence changes.
- The linear relationship between last-layer delay and its LCA suggests a control-theoretic optimization rule: per-layer momentum or update lag could be tuned automatically to keep every layer's LCA negative, something the paper does not attempt.
- Because the method's cost scales with full-training-set gradient evaluations, a natural extension the paper mentions is approximating LCA with minibatch gradients; an editor's testable version is to measure how the per-layer conclusions degrade as the gradient batch shrinks.
- The synchronized-peak result implies that the global loss curve hides sharp coincident events; a testable consequence is that learning-rate schedules triggered by these coincident peaks could be more sample-efficient than fixed schedules.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Loss Change Allocation (LCA), a per-parameter, per-iteration decomposition of the change in training loss during neural network training. Starting from the path integral identity in Eq. (1), the authors approximate the integral along each optimizer step using an RK4 quadrature and allocate the resulting signed contributions to individual parameters. They release code and use LCA to report three empirical observations: only slightly over half of parameters help on a typical iteration; certain layers (e.g., first and last in CIFAR-ResNet with SGD) consistently hurt overall, proposed to be due to phase lag; and learning increments are synchronized across layers. The paper also includes control experiments (freezing layers, varying momentum/learning rate) and a permutation-style baseline for synchronization.
Significance. The core mathematical identity is correct and the aggregate validation in Table S1 is strong: the RK4 sum matches the total loss change to within 0.14% in all configurations. The paper's strengths include a clear, easily implementable measurement, publicly released code, and falsifiable baseline comparisons for the synchronization claim. If the per-parameter accuracy of LCA is established, the tool would be a useful diagnostic for training dynamics, and the reported phenomena—noisy per-parameter credit, hurting layers, and synchronization—would be interesting and publishable. However, the headline insights operate at a finer granularity than the validation, and the missing per-parameter check currently limits the strength of all three conclusions.
major comments (3)
- [Section 2, Eq. (3) and Table S1] The paper's validation of LCA only constrains the sum over parameters: Table S1 reports total error under 0.14% and average iteration error, but the claims in Sections 3-5 are about the sign and magnitude of individual A_t,i (percent of parameters helping, per-layer sums, and peak detection). An aggregate error bound does not bound per-coordinate errors: with hundreds of thousands of parameters, per-parameter quadrature errors much larger than the total can cancel in the sum. For example, the RK4 approximation of a component integral can have the wrong sign whenever the gradient component changes sign along the step and the (1,4,1) quadrature misweights the midpoint. I request a per-parameter validation on a small model: compute the true per-coordinate line integrals along the straight segment by fine subdivision (or exact integration on a quadratic surrogate) and report sign agreement rates, per-layer relative errors, and the fraction of parameters whose LCA sign flips between the RK4 estimate and the high-resolution reference. Without such a check, the three empirical insights are not established as properties of the training process; they may be properties of the quadrature approximation.
- [Section 4 and Figure 5] The phase-lag hypothesis is tested indirectly by varying the last layer's momentum, but changing momentum alters the effective learning rate, the variance of the updates, and the coupling with other layers, not just the information delay. The near-linear relationship between delay and last-layer LCA is suggestive, but it does not establish that phase lag is the mechanism behind the hurting last layer. To support the claim, the paper should directly measure the phase relationship between layer-wise LCA oscillations (e.g., cross-correlation or Hilbert phase) and show that the hurting layer's phase lags the others by a consistent amount. As written, the conclusion is presented as a hypothesis, which is acceptable for an exploratory paper, but given that this is one of the three central contributions, the level of support is currently insufficient.
- [Section 5 and Section S5] The synchronization analysis is not fully specified. The main text says 'p-value < 1e-6' (written as '1−6' in the manuscript), but the supplementary baseline description ('shifting each layer in each class randomly by -2, -1, 0, 1, 2') does not state the number of random replicates, whether shifts are applied per layer-class independently, how overlapping shifts are handled, or how the CIFAR thresholds (25-50% vs 50-100% synchronization) map to the reported statistics. Without these details, the test is not reproducible from the paper text. Please provide the exact procedure, including the null-distribution generation and the number of samples, and ideally release the analysis script.
minor comments (5)
- [Section 2] The text says the midpoint gradient 'doubles computation', but the RK4 rule in Eq. (2) requires three full-batch gradient evaluations per iteration (at theta_t, the midpoint, and theta_{t+1}), not two. Please correct this description.
- [Section 5] The string 'p-value < 1−6' appears to be a typo for 'p-value < 10^-6' (or '< 1e-6'). Please fix.
- [Figure 6 caption and Section S5] There are typographical errors: 'seperated' should be 'separated', 'Eeah row' should be 'Each row', and 'orderred' should be 'ordered'. Please proofread the supplementary text.
- [Section 4] The p-values for layer-level LCA (e.g., p < 10^-4 for the first and last layers) are reported without stating the statistical test used. Please specify whether these are t-tests, permutation tests, or another procedure, and describe the sample size per test.
- [Eq. (1)-(3)] The notation A_{t,i} is defined only after Eq. (3); consider defining it in the text preceding Eq. (2) to improve readability.
Circularity Check
No significant circularity: LCA is a direct first-order path-integral decomposition of observed gradients and parameter updates, and the empirical insights are measurements rather than predictions fitted to the same inputs.
full rationale
The paper's central construction (Eq. 1–3) defines each A_{t,i} = (∂L/∂θ_i)(θ_{t+1,i} − θ_{t,i}), approximately summing to the per-iteration loss change. This is a numerical quadrature of an actual training trajectory, not a model with fitted parameters; Table S1 reports the approximation error against the true loss change. The claims in Sections 3–5 (near-50% helping parameters, hurting layers, cross-layer synchronization) are descriptive statistics of these computed LCA values, compared where appropriate against randomized baselines or interventions such as freezing or changing the last-layer momentum. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing assumption is imported from the authors' own prior work; the self-citations to Refs. [19], [22], [33], and [36] appear as related work or future directions and do not carry the derivation. Concerns about per-parameter attribution accuracy or path-dependence of the coordinate-wise allocation are validity questions about the measurement, not instances where a claimed result reduces by construction to its own input.
Assumptions & free parameters
free parameters (1)
- peak_threshold_k =
20
assumptions (4)
- standard math Fundamental theorem of calculus: change in loss along any path equals the integral of gradient dot motion (Eq. 1)
- domain assumption The loss surface along the training trajectory is smooth enough that the fourth-order Runge-Kutta approximation yields accurate per-iteration loss changes (validated only in aggregate, Table S1)
- domain assumption Full training-set loss L(theta) is the right scalar for measuring training progress
- domain assumption Momentum m corresponds to an information delay of m/(1-m) steps (Section 4)
Cite this review
Pith. "Pith review of LCA: Loss Change Allocation for Neural Network Training." pith.science (2026). https://pith.science/paper/E2J3QLED
@misc{pith2026190901440,
author = {Pith},
title = {Pith review of: LCA: Loss Change Allocation for Neural Network Training},
year = {2026},
howpublished = {\url{https://pith.science/paper/E2J3QLED}},
note = {Machine review of arXiv:1909.01440}
}
read the original abstract
Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is limited, with a single scalar loss being the most common viewport into this high-dimensional, dynamic process. We propose a new window into training called Loss Change Allocation (LCA), in which credit for changes to the network loss is conservatively partitioned to the parameters. This measurement is accomplished by decomposing the components of an approximate path integral along the training trajectory using a Runge-Kutta integrator. This rich view shows which parameters are responsible for decreasing or increasing the loss during training, or which parameters "help" or "hurt" the network's learning, respectively. LCA may be summed over training iterations and/or over neurons, channels, or layers for increasingly coarse views. This new measurement device produces several insights into training. (1) We find that barely over 50% of parameters help during any given iteration. (2) Some entire layers hurt overall, moving on average against the training gradient, a phenomenon we hypothesize may be due to phase lag in an oscillatory training process. (3) Finally, increments in learning proceed in a synchronized manner across layers, often peaking on identical iterations.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Partitioned integrators for thermodynamic parameterization of neural networks
Layer-partitioned Langevin integrators (LOL and AdLaLa) train single hidden layer perceptrons faster, more accurately, and more robustly than SGD/Adam on hard spiral and trigonometric classification problems.
Reference graph
Works this paper leans on
-
[1]
Critical learning periods in deep neural networks
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. CoRR, abs/1711.08856, 2017. URL http://arxiv.org/abs/1711.08856
arXiv 2017
-
[2]
G. Alain and Y . Bengio. Understanding intermediate layers using linear classifier probes. ArXiv e-prints, October 2016
work page 2016
-
[3]
Optimization methods for large-scale machine learning
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223–311, 2018
work page 2018
-
[4]
The loss surfaces of multilayer networks
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, pages 192–204, 2015
work page 2015
-
[5]
Identifying and attacking the saddle point problem in high-dimensional non- convex optimization
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non- convex optimization. In Advances in neural information processing systems, pages 2933–2941, 2014
work page 2014
-
[6]
The lottery ticket hypothesis: Finding sparse, trainable neural networks
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), volume abs/1803.03635, 2019. URL http://arxiv.org/abs/1803.03635
arXiv 2019
-
[7]
Qualitatively characterizing neural network optimization problems
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544, 2014
arXiv 2014
-
[8]
A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=r14EOsCqKX
work page 2019
Show all 36 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385
2015 arXiv
-
[10]
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015
2015
-
[11]
Improving neural networks by preventing co-adaptation of feature detectors
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012
2012 arXiv
-
[12]
Fix your classifier: the marginal value of training the last weight layer
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. CoRR, abs/1801.04540, 2018. URL http://arxiv.org/abs/1801. 04540
2018 arXiv
-
[13]
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL http://arxiv.org/ abs/1502.03167
2015 arXiv
-
[14]
On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
Stanisław Jastrz˛ ebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length. In International Conference on Learning Representations (ICLR) , page arXiv:1807.05031, Jul 2019
2019 arXiv
-
[15]
On large-batch training for deep learning: Generalization gap and sharp minima
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016
2016 arXiv
-
[16]
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting...
2017 doi
-
[17]
Beitrag zur näherungweisen integration totaler differentialgleichungen
Wilhelm Kutta. Beitrag zur näherungweisen integration totaler differentialgleichungen. 1901
1901
-
[18]
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998
1998
-
[19]
Measuring the Intrinsic Dimension of Objective Landscapes
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Dimension of Objective Landscapes. In International Conference on Learning Representations, April 2018
2018
-
[20]
Visualizing the loss landscape of neural nets
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pages 6389–6399, 2018
2018
-
[21]
The loss surface of deep and wide neural networks
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 2603–2612. JMLR. org, 2017
2017
-
[22]
Raghu, J
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. ArXiv e-prints, June 2017
2017
-
[23]
Über die numerische auflösung von differentialgleichungen
Carl Runge. Über die numerische auflösung von differentialgleichungen. Mathematische Annalen, 46(2):167–178, 1895
-
[24]
On the quality of the initial basin in overspecified neural networks
Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks. In International Conference on Machine Learning, pages 774–782, 2016
2016
-
[25]
Opening the black box of deep neural networks via information
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, abs/1703.00810, 2017. URL http://arxiv.org/abs/1703.00810
2017 arXiv
-
[26]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014. URL http://arxiv.org/abs/1409.1556
2014 arXiv
-
[27]
Deep inside convolutional networks: Visualising image classification models and saliency maps
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, presented at ICLR Workshop 2014, 2013
2014 arXiv
-
[28]
No bad local minima: Data independent training error guarantees for multilayer neural networks
Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. arXiv preprint arXiv:1605.08361, 2016
2016 arXiv
-
[29]
Riedmiller
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. CoRR, abs/1412.6806, 2014. URL http://arxiv. org/abs/1412.6806
2014 arXiv
-
[30]
On the importance of initialization and momentum in deep learning
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147, 2013
2013
-
[31]
Simpson’s rule
Eric W Weisstein. Simpson’s rule. 2003
2003
-
[32]
A walk with sgd
Chen Xing, Devansh Arpit, Christos Tsirigotis, and Yoshua Bengio. A walk with sgd. 2018
2018
-
[33]
Yosinski, J
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding Neural Networks Through Deep Visualization. ArXiv e-prints, June 2015
2015
-
[34]
Improved multitask learning through synaptic intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli. Improved multitask learning through synaptic intelligence. CoRR, abs/1703.04200, 2017. URL http://arxiv.org/abs/1703. 04200
2017 arXiv
-
[35]
Are all layers created equal? arXiv preprint arXiv:1902.01996, 2019
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint arXiv:1902.01996, 2019
1902 arXiv
-
[36]
Total error
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. arXiv preprint arXiv:1905.01067, 2019. 11 Supplementary Information for: LCA: Loss Change Allocation for Neural Network Training S1 Supplementary results:...
1905 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.