{"id":"643c4c0a-7d97-4452-9e4e-a19dbd0f8120","arxiv_id":"1909.01440","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A per-parameter decomposition of training loss change shows that learning is noisy, with only about half of parameters helping per step, some layers hurting overall, and learning spikes synchronized across layers.","lead":"This paper introduces Loss Change Allocation (LCA), a method that attributes each step of a network's training loss change to individual parameters. It reveals that only about half of a network's parameters help at any given step, that some layers consistently hurt, and that learning spikes happen in sync across layers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-parameter LCA values are not validated: aggregate error (Table S1) does not constrain individual A_{t,i}, and the coordinate-wise line integral is path-dependent, so 'helping'/'hurting' labels may be artifacts of the straight-segment convention.","rationale":"The reader's weakest assumption is that per-parameter LCA accuracy is unvalidated because the paper only checks the aggregate sum. My stress-test agrees with that identification and sharpens it: the problem is not only numerical quadrature error but also the fundamental path-dependence of any coordinate-wise decomposition of a conservative vector field. Since the empirical insights are about per-parameter and per-layer signs, the lack of validation is genuinely load-bearing. The paper's method is still a mathematically well-defined diagnostic, and the aggregate grounding is a real strength, so the appropriate verdict remains conditional: the tool and observations are promising, but the per-parameter claims should be tested against an alternative path before being treated as established. The proposed concrete test is feasible on the smallest model and would directly settle whether the concern lands.","tokens_in":18162,"tokens_out":8596,"duration_ms":100277,"concrete_test":"Using the released code on MNIST-FC (84,060 parameters, 880 iterations), recompute per-parameter LCA for a sample of 100 SGD steps with (a) the paper's straight-segment RK4 rule and (b) a piecewise-linear path that moves one entire layer at a time from old to new values, in both forward and reverse layer order. Each of these paths has exactly the same total loss change by the fundamental theorem of calculus. Count the fraction of parameters whose helping/hurting sign flips between the straight-segment result and either layer-sequential ordering, and check whether the layer-summed LCA signs (especially the first and last layers) and the global '~50% helping' statistic change materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central load-bearing claim is that the per-parameter decomposition in Eq. 3, A_{t,i} = (RK4-averaged gradient component) * (parameter displacement), measures whether each parameter helps or hurts training. The paper validates only the sum over i against the true loss change (Table S1, total error under 0.14%), not the individual A_{t,i}. Aggregate accuracy does not imply per-parameter accuracy: quadrature error can cancel across coordinates, and more fundamentally the coordinate-wise allocation is path-dependent. Because the full-batch gradient is conservative, every path from θ_t to θ_{t+1} gives the same total loss change but different per-coordinate integrals ∫ ∂L/∂θ_i dθ_i. The straight-line path used in Eq. 2–3 is convenient, but nothing in the paper shows it is the correct attribution rule. A parameter can be labeled 'helping' along the straight segment and 'hurting' along a coordinate-sequential path, or vice versa. Claims (1) 'barely over 50% of parameters help', (2) 'first/last layers hurt', and (3) 'synchronized learning' are all statements about signs and layer-sums of these path-dependent values. The freezing experiments in Section 4 provide indirect support for the last-layer result, but they are consistent with other mechanisms and do not validate per-parameter signs. Without a check of per-parameter or per-layer stability under alternative valid paths, the empirical insights are not established as properties of the training process; they may be properties of the chosen integration path.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Loss Change Allocation (LCA), a per-parameter, per-iteration decomposition of the change in training loss during neural network training. Starting from the path integral identity in Eq. (1), the authors approximate the integral along each optimizer step using an RK4 quadrature and allocate the resulting signed contributions to individual parameters. They release code and use LCA to report three empirical observations: only slightly over half of parameters help on a typical iteration; certain layers (e.g., first and last in CIFAR-ResNet with SGD) consistently hurt overall, proposed to be due to phase lag; and learning increments are synchronized across layers. The paper also includes control experiments (freezing layers, varying momentum/learning rate) and a permutation-style baseline for synchronization.","tokens_in":18415,"tokens_out":8111,"duration_ms":81999,"significance":"The core mathematical identity is correct and the aggregate validation in Table S1 is strong: the RK4 sum matches the total loss change to within 0.14% in all configurations. The paper's strengths include a clear, easily implementable measurement, publicly released code, and falsifiable baseline comparisons for the synchronization claim. If the per-parameter accuracy of LCA is established, the tool would be a useful diagnostic for training dynamics, and the reported phenomena—noisy per-parameter credit, hurting layers, and synchronization—would be interesting and publishable. However, the headline insights operate at a finer granularity than the validation, and the missing per-parameter check currently limits the strength of all three conclusions.","major_comments":[{"comment":"The paper's validation of LCA only constrains the sum over parameters: Table S1 reports total error under 0.14% and average iteration error, but the claims in Sections 3-5 are about the sign and magnitude of individual A_t,i (percent of parameters helping, per-layer sums, and peak detection). An aggregate error bound does not bound per-coordinate errors: with hundreds of thousands of parameters, per-parameter quadrature errors much larger than the total can cancel in the sum. For example, the RK4 approximation of a component integral can have the wrong sign whenever the gradient component changes sign along the step and the (1,4,1) quadrature misweights the midpoint. I request a per-parameter validation on a small model: compute the true per-coordinate line integrals along the straight segment by fine subdivision (or exact integration on a quadratic surrogate) and report sign agreement rates, per-layer relative errors, and the fraction of parameters whose LCA sign flips between the RK4 estimate and the high-resolution reference. Without such a check, the three empirical insights are not established as properties of the training process; they may be properties of the quadrature approximation.","section":"Section 2, Eq. (3) and Table S1"},{"comment":"The phase-lag hypothesis is tested indirectly by varying the last layer's momentum, but changing momentum alters the effective learning rate, the variance of the updates, and the coupling with other layers, not just the information delay. The near-linear relationship between delay and last-layer LCA is suggestive, but it does not establish that phase lag is the mechanism behind the hurting last layer. To support the claim, the paper should directly measure the phase relationship between layer-wise LCA oscillations (e.g., cross-correlation or Hilbert phase) and show that the hurting layer's phase lags the others by a consistent amount. As written, the conclusion is presented as a hypothesis, which is acceptable for an exploratory paper, but given that this is one of the three central contributions, the level of support is currently insufficient.","section":"Section 4 and Figure 5"},{"comment":"The synchronization analysis is not fully specified. The main text says 'p-value < 1e-6' (written as '1−6' in the manuscript), but the supplementary baseline description ('shifting each layer in each class randomly by -2, -1, 0, 1, 2') does not state the number of random replicates, whether shifts are applied per layer-class independently, how overlapping shifts are handled, or how the CIFAR thresholds (25-50% vs 50-100% synchronization) map to the reported statistics. Without these details, the test is not reproducible from the paper text. Please provide the exact procedure, including the null-distribution generation and the number of samples, and ideally release the analysis script.","section":"Section 5 and Section S5"}],"minor_comments":[{"comment":"The text says the midpoint gradient 'doubles computation', but the RK4 rule in Eq. (2) requires three full-batch gradient evaluations per iteration (at theta_t, the midpoint, and theta_{t+1}), not two. Please correct this description.","section":"Section 2"},{"comment":"The string 'p-value < 1−6' appears to be a typo for 'p-value < 10^-6' (or '< 1e-6'). Please fix.","section":"Section 5"},{"comment":"There are typographical errors: 'seperated' should be 'separated', 'Eeah row' should be 'Each row', and 'orderred' should be 'ordered'. Please proofread the supplementary text.","section":"Figure 6 caption and Section S5"},{"comment":"The p-values for layer-level LCA (e.g., p < 10^-4 for the first and last layers) are reported without stating the statistical test used. Please specify whether these are t-tests, permutation tests, or another procedure, and describe the sample size per test.","section":"Section 4"},{"comment":"The notation A_{t,i} is defined only after Eq. (3); consider defining it in the text preceding Eq. (2) to improve readability.","section":"Eq. (1)-(3)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is simple and clearly presented, and the aggregate identity is sound. The main technical gap is the lack of per-parameter/per-layer validation of the LCA values, which is the backbone of the three central claims. I do not think this is a fatal flaw because the validation is straightforward on small models, but it is load-bearing and should be addressed before publication. The synchronization statistical test also needs to be specified more precisely. The code release and the controlled freezing experiments are strong positives and should be highlighted in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Janice, this one is worth a look. The core idea—use the path integral of the gradient along the actual training trajectory to allocate loss change to individual parameters—is simple, well-grounded, and, as far as I can tell, new. They validate the aggregate sum to under 0.14% across all configurations (Table S1), ship code, and the three empirical findings (about half of parameters hurting, some layers consistently hurting, synchronized learning spikes) are reproducible and not in the cited literature. The freezing and momentum-delay experiments are a nice touch: freezing the last layer of CIFAR-ResNet improves training, and varying last-layer momentum flips its LCA almost linearly. That is evidence the layer-level signal is real, not just quadrature noise.\n\nThe soft spot is exactly what the reader flagged: the aggregate validation does not constrain the per-parameter signs. RK4 quadrature could in principle have larger per-coordinate errors that cancel in the sum. More fundamentally, the coordinate-wise split of a path integral is path-dependent. The straight-segment path they use is the actual SGD/Adam step, which is a reasonable convention, but it is still a convention: another valid path between the same endpoints would give different per-parameter 'help/hurt' labels. They don't discuss this, and the claims about per-parameter behavior are stated as properties of training rather than properties of the chosen allocation rule. For a paper that introduces a measurement device, I'd want a paragraph acknowledging the attribution is not unique and maybe a sensitivity check over alternative integration paths (e.g., coordinate-sequential) to see if the aggregate insights (50% helping, hurting layers) survive. I suspect they would, but it's not shown.\n\nThe phase-lag explanation for hurting layers is explicitly a hypothesis, and the momentum experiment is a clever, if indirect, test. I wouldn't call it proven, but they don't overclaim.\n\nSo: the paper is not fatally flawed. It introduces a useful, well-grounded diagnostic, and the observations are honest and reproducible. The main missing piece is validation or explicit discussion of per-parameter attribution. I'd send it to serious review; the authors should be asked to address path-dependence and per-coordinate error. For a reader, the tool and the layer-level findings are the value; the per-parameter claims are the weakest part.\n\nRecommendation: engage with it, and for reviewing, accept.","headline":"A useful, well-grounded diagnostic for per-parameter credit during training; the per-parameter claims rest on a convention, but the tool and the observations are worth engaging.","tokens_in":18986,"tokens_out":3215,"would_cite":true,"duration_ms":31519,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Loss Change Allocation, a per-parameter decomposition of each training iteration's loss change, and uses it to claim that just over half of parameters help at any step, some layers consistently hurt, and the biggest…","keywords":["loss change allocation","credit assignment","training dynamics","path integral","Runge-Kutta integration","layer-wise learning behavior","stochastic gradient noise"],"falsifier":"Compute, for one curved training step of a small network, both the LCA value of each parameter and the exact loss difference caused by moving only that parameter along its update while all other parameters are fixed; if the signs disagree for a substantial fraction of parameters, the per-parameter claims fail. Alternatively, integrate the path integral along the same step with very fine quadrature and compare per-component values.","tokens_in":17911,"feed_emoji":"📉","tokens_out":7444,"duration_ms":69058,"temperature":0.7,"pith_summary":"This paper sets out to give neural-network training a per-parameter viewport: for every weight and every iteration, it allocates the change in training loss to that weight, so a user can see which parameters helped and which hurt. The allocation rests on a path-integral identity, approximated with a Runge-Kutta integrator so the per-iteration components sum to the true loss change within a small error. Using this tool, the paper claims three empirical findings: only just over half of parameters help in a typical iteration; some entire layers, notably the first and last layers of a ResNet trained with SGD, consistently hurt; and the sharpest learning moments occur at the same iterations across all layers. A fair reader should care because these claims turn the vague slogan that training is noisy into a measurable, per-weight fact, and they suggest practical interventions such as freezing or shrinking the learning rate of layers that hurt.","feed_headline":"Barely half of a network's weights help at any training step","feed_subtitle":"A path-integral decomposition also exposes layers that hurt on average and synchronized learning bursts.","key_machinery":"The central object is the path-integral identity $L(\\theta_T)-L(\\theta_0)=\\int_C \\langle \\nabla_\\theta L(\\theta), d\\theta\\rangle$, approximated one step at a time by first-order Taylor terms $A_{t,i}$. To keep the approximation accurate on curved loss surfaces, the gradient is averaged at the initial, midpoint, and endpoint of each step using the fourth-order Runge-Kutta (RK4) rule, equivalently Simpson's rule, halving the step until the per-iteration error is below 0.001. This machinery converts an otherwise opaque scalar loss curve into an additive, signed, per-parameter credit map, which is what makes the paper's per-weight, per-layer, and per-class claims measurable.","core_discovery":"The central claim is that the change in training loss from one iteration to the next can be partitioned, parameter by parameter, by the product of that parameter's component of the full-training-set gradient and its movement in that step: $A_{t,i} = (\\nabla_\\theta L(\\theta_t))_i(\\theta_{t+1}-\\theta_t)_i$, with the gradient evaluated at a Runge-Kutta-averaged point so the components sum to the true loss change to within a small error. LCA is signed and additive, so it can be summed over iterations, neurons, channels, or layers. On this basis the paper reports three empirical findings: in a typical iteration only slightly more than half of parameters have negative LCA and thus 'help'; some whole layers, in particular the first and last layers of a CIFAR-ResNet trained with SGD, accumulate positive LCA over training, which the paper attributes to their being phase-lagged relative to other layers; and the top moments of per-layer learning occur at the same iterations for all layers far more often than chance.","pith_inferences":["If per-parameter LCA signs are accurate, the near-50% helping rate implies the effective signal-to-noise ratio of gradient descent is far lower than commonly assumed; a testable extension is to mask out the hurting half of the weights each step and see how convergence changes.","The linear relationship between last-layer delay and its LCA suggests a control-theoretic optimization rule: per-layer momentum or update lag could be tuned automatically to keep every layer's LCA negative, something the paper does not attempt.","Because the method's cost scales with full-training-set gradient evaluations, a natural extension the paper mentions is approximating LCA with minibatch gradients; an editor's testable version is to measure how the per-layer conclusions degrade as the gradient batch shrinks.","The synchronized-peak result implies that the global loss curve hides sharp coincident events; a testable consequence is that learning-rate schedules triggered by these coincident peaks could be more sample-efficient than fixed schedules."],"forward_implications":["Because LCA is additive over time and over parameter groups, practitioners can produce a complete per-layer credit map of a training run and use it to decide which layers to freeze, shrink, or leave at their initialization.","The near-50% helping fraction implies that stochastic-gradient training is far noisier at the parameter level than scalar loss curves suggest; the paper's oscillation counts quantify this, with weight directions changing on average every 6.7 iterations for CIFAR-ResNet with SGD.","Freezing the last layer of CIFAR-ResNet with SGD, or cutting its learning rate tenfold, turns that layer's LCA from positive to negative and improves both train and test loss, so layer-hurting is not merely a curiosity but a lever on performance.","The synchronization of peak learning across layers implies that training advances in coherent global pulses; this is measurable from LCA data and is significant against a randomly shifted baseline, with 9.4 aligned peaks in the actual MNIST-FC network versus 0.4 in the baseline.","The heavy-tailed distribution of LCA values indicates that a Wiener-process picture of learning should be refined to a heavy-tailed process."],"supporting_citations":[{"why":"Supplies the fourth-order Runge-Kutta scheme used for accurate gradient averaging.","marker":"[23]"},{"why":"Co-author of the RK4 integration method that the paper applies to the loss-change integral.","marker":"[17]"},{"why":"Simpson's rule, equivalent to the (1,4,1) weighting used in the RK4 gradient averaging.","marker":"[31]"},{"why":"Prior result that the last layer can be frozen without performance loss, which LCA offers a mechanistic explanation for.","marker":"[12]"},{"why":"Defines a similar per-parameter importance using the mini-batch gradient; LCA contrasts by using the full training-set gradient.","marker":"[34]"},{"why":"Shows layers have heterogeneous roles and some can be reset; LCA provides a complementary per-layer credit measure.","marker":"[35]"},{"why":"Information-plane view of training as a Wiener process, which LCA's heavy-tailed measurements refine.","marker":"[25]"}],"fun_headline_variants":["Most weights don't help at any given training step","Neural nets: half of parameters are dead weight each step","LCA exposes layers that actually hurt training","Training spikes are synchronized across all layers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measurement assumes that the per-weight split of the loss change is accurate, but the paper only checks that the components sum to the true loss change; on sharply curved steps, a weight labeled 'helping' could actually be hurting.","fun_headline_variants_meta":{"raw":{"variants":["Most weights don't help at any given training step","Neural nets: half of parameters are dead weight each step","LCA exposes layers that actually hurt training","Training spikes are synchronized across all layers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2635,"prompt_tokens":982,"completion_tokens":1653,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1593}},"tokens_in":598,"tokens_out":1653,"duration_ms":12357,"temperature":1.0,"reasoning_tokens":1593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:17:40.294635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for one curved training step of a small network, both the LCA value of each parameter and the exact loss difference caused by moving only that parameter along its update while all other parameters are fixed; if the signs disagree for a substantial fraction of parameters, the per-parameter claims fail. Alternatively, integrate the path integral along the same step with very fine quadrature and compare per-component values.","supporting_citations":[{"cited_title":"Über die numerische auflösung von differentialgleichungen","cited_arxiv_id":null,"evidence_quote":"Supplies the fourth-order Runge-Kutta scheme used for accurate gradient averaging."},{"cited_title":"Beitrag zur näherungweisen integration totaler differentialgleichungen","cited_arxiv_id":null,"evidence_quote":"Co-author of the RK4 integration method that the paper applies to the loss-change integral."},{"cited_title":"Simpson’s rule","cited_arxiv_id":null,"evidence_quote":"Simpson's rule, equivalent to the (1,4,1) weighting used in the RK4 gradient averaging."},{"cited_title":"Fix your classifier: the marginal value of training the last weight layer","cited_arxiv_id":"1801.04540","evidence_quote":"Prior result that the last layer can be frozen without performance loss, which LCA offers a mechanistic explanation for."}],"review_version":1}