Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Interpretation of High-Dimensional Regression Coefficients by Comparison with Linearized Compressing Features

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that ridge and partial least squares regression paths in high dimensions contain coefficients that, under strong regularization, match the linearization of a single nonlinear compressing feature, and that relaxing…

desk verdict Useful, well-scoped short paper with a correct derivation, but the headline similarity claim is undercut by a nullspace projection issue the paper never quantifies. read the letter →

arxiv 2411.12060 v1 pith:BH2SLXN6 submitted 2024-11-18 cs.LG stat.ML

Joachim Schaeffer , Jinwook Rhyu , Robin Droop , Rolf Findeisen , Richard Braatz This is my paper · ORCID
classification cs.LGstat.ML
keywords linearregressionhigh-dimensionaldatafeaturecoefficientscompressingfeaturesregularizationpathridgepartialleastsquareslithium-ionbatteries
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Linear regression in high dimensions is hard to interpret because many coefficient vectors make the same predictions. This paper develops a reference for reading regression coefficients: take a single nonlinear compressing feature $g$ that generates the response, linearize it around the data mean, and rescale to obtain feature coefficients $\beta_{\mathrm{T1}} = m\nabla g(\bar{x})$. Using ridge regression and partial least squares on lithium-ion battery data with synthetic responses, the paper shows that strongly regularized coefficients align with these feature coefficients and are dominated by data variance, while coefficients chosen by cross-validation grow in magnitude and vary faster across the input domain, linearly approximating the nonlinear response locally. The work's aim is to explain how the shape of high-dimensional regression coefficients changes with regularization and to connect regression paths to interpretable feature engineering.

What carries the argument

The load-bearing object is the vector of feature coefficients $\beta_{\mathrm{T1}} = m\nabla g(\bar{x})$, where $g:\mathbb{R}^p\to\mathbb{R}$ is a single compressing feature, $\bar{x}$ is the column mean of the data, and $m$ is a scalar estimated by ordinary least squares after centering. This vector is the first-order Taylor approximation of $g$ rescaled to match the response. It supplies a reference against which the ridge and PLS solution paths are compared using the $\ell^2$ distance, and it is what lets the authors separate the part of the coefficients explained by the linearized feature from the part that emerges as regularization is relaxed.

What would settle it

Take the same battery data but build a response from a non-differentiable feature, for example $g(x_i)=\sum_j \mathbf{1}_{x_{i,j}>c}$, or from a sum of two compressing features. If the ridge path still passes close to the $\beta_{\mathrm{T1}}$ of some differentiable single feature at strong regularization, or if cross-validated coefficients do not vary faster than strongly regularized ones, the claimed interpretation would need to be revised.

Watch

Extended reading notes

Core claim

The central claim is that the ridge regression solution path contains coefficients that, at strong regularization, are comparable to the first-order linearization of the true scalar feature: $\beta_{\mathrm{T1}} = m\nabla g(\bar{x})$, where $m$ is an ordinary-least-squares scalar fit. In the sum-of-squares case study these coefficients are a scaled column mean; in the sinusoidal case study they take a different shape that does not match the variance of the data. The closest ridge and PLS coefficients, selected by minimizing $\|\beta(\lambda)-\beta_{\mathrm{T1}}\|_2^2$, track this shape, and under cross-validated regularization the coefficients show larger amplitude and faster variation across voltage, indicating the linear model is using local structure to approximate a nonlinear map. This gives a concrete picture: regularized linear regression first captures high-variance directions, then spends its remaining capacity on a local, piecewise-linear approximation of the nonlinearity.

Load-bearing premise

The whole comparison rests on the assumption that there exists one smooth scalar feature of the inputs that produces the response, with no noise that the linearization cannot absorb; if the true response is generated by multiple features or jumps, the reference coefficients do not represent anything real.

Editorial extensions

If this is right

  • If a response is generated by a single differentiable compressing feature, ridge coefficients at strong regularization will look like the scaled gradient of that feature at the data mean, not like an arbitrary regularization artifact.
  • At cross-validation-level regularization, coefficient magnitude and the rate of change across the input domain increase, so the shape of the coefficients can be read as a local, linear approximation of the nonlinear response.
  • The $\ell^2$ distance to $\beta_{\mathrm{T1}}$ along the regularization path gives a principled way to pick the regularization level at which a linear model best matches a candidate feature linearization.
  • PLS with one component mirrors ridge in the variance-dominated regime, but may need additional components before it picks up the characteristic shape of the feature coefficients.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: If this picture generalizes, the gap between $\beta_{\mathrm{T1}}$ and the cross-validated coefficients is a measurable 'nonlinearity load' of the data set, which could be tracked across different candidate features.
  • Editorial extension: The nullspace caveat noted in the paper means closeness of coefficients is a stricter condition than closeness of predictions; a practitioner could use prediction distance instead when the goal is forecasting, but would lose the shape interpretation.
  • Editorial extension: For real battery data where no ideal feature is known, the method suggests a practical test: propose a feature, linearize it, and check whether the ridge path comes near it; if it does, that feature explains the regression coefficients even when it was not used to build the response.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how linear regression coefficients for high-dimensional functional data relate to a known nonlinear compressing feature. The authors linearize a feature g at the training mean, define feature coefficients β_T1 = m∇g(x̄) with m estimated by OLS, and then select ridge and PLS regularization parameters by minimizing the ℓ2 distance between the regression coefficients and β_T1. They illustrate the method on two synthetic battery-data responses, one based on a sum-of-squares feature and one on a sinusoidal feature, and conclude that strongly regularized regression coefficients resemble the linearized feature coefficients, while weaker regularization produces coefficients that exploit local structure to approximate nonlinearity.

Significance. If the central claim holds, the paper offers an interpretability link between nonlinear feature maps and linear regression coefficients in the p ≫ n regime, with potential relevance to battery cycle-life prediction. The manuscript is commendably explicit about its assumptions and limitations, and the use of automatic differentiation for deriving feature coefficients is a practical strength. However, the empirical evidence is limited to two synthetic responses from a single training set with no uncertainty quantification or out-of-sample validation, and the comparison metric is partly constructed to make the closest coefficients close. The theoretical derivation in Section 2 is sound, but the load-bearing empirical conclusions in Section 4 need additional support before the claims can be accepted as stated.

major comments (4)
  1. [Section 2, Eqs. (9)-(10)] The 'closeness' of the regression path to β_T1 is enforced by construction: the paper minimizes the ℓ2 distance to define the closest regression coefficient and then presents that coefficient as evidence that the path 'contains' similar coefficients. To support the claim, the paper should report the minimized distance values, compare them with distances at other regularization parameters (e.g., cross-validation-selected values), and state how much the visual similarity in Figures 2 and 3 exceeds what would be expected under a null model or under random row-space vectors.
  2. [Section 2, A.3, Figures 2-3] In the p ≫ n setting with n = 40 and p = 1000, every ridge and PLS coefficient vector lies in the 39-dimensional row space of the mean-centered data, while β_T1 generally has a component in the 969-dimensional nullspace. The ℓ2 distance decomposes as ||β(λ) − β_T1||² = ||β(λ) − Pβ_T1||² + ||(I − P)β_T1||², where P projects onto the row space. The nullspace term is independent of λ, so minimizing Eq. (10) fits only Pβ_T1. The paper acknowledges nullspace effects in A.3 but never quantifies the decomposition for the two case studies. Without reporting ||Pβ_T1|| and ||(I − P)β_T1||, the visual shape comparisons in Figures 2 and 3 do not establish similarity in the full coefficient space; they may only show that the row-space projection resembles β_T1.
  3. [Section 3, Figures 2-3] The central empirical claim is supported by only two synthetic responses, evaluated on the same training set used both to estimate the scalar m in Eq. (7) and to select the regularization parameter in Eq. (10). There is no uncertainty quantification, no out-of-sample validation, and no quantitative measure of shape agreement other than the distance that was minimized by construction. The conclusions in Section 4 generalize from these two examples; the authors should either add bootstrap confidence bands, repeat the analysis across a broader family of features, or explicitly restrict the conclusions to illustrative demonstrations.
  4. [Section 2, Eqs. (7)-(8)] The feature coefficients β_T1 are not fully specified by the feature g alone, because the scalar m is obtained by OLS on the same response vector y that is later used in the regression and in the distance minimization of Eq. (10). Thus the target of the comparison already contains information from y, which weakens the interpretation of β_T1 as the 'linearized true mapping.' The paper should report how sensitive the conclusions are to this calibration, for instance by comparing with m fixed at 1 or by estimating m on a separate validation split.
minor comments (5)
  1. [Section 3 vs. A.1] The paper states X ∈ R^{124×1000} in Section 3 but R^{123×1000} in Appendix A.1 after outlier removal; please clarify the intended dimensions.
  2. [Section 2, Assumption 2] The text says Assumption 2 (zero irreducible error) is relaxed later, but the only mechanism shown is the OLS scalar m in Eq. (7); a single scalar cannot absorb per-sample irreducible errors ε_i^*. Please state explicitly how Assumption 2 is relaxed, or remove the claim.
  3. [Section 2, Eqs. (9)-(10)] No numerical details are given for solving the optimization in Eq. (10), such as the grid or algorithm for λ, the range of λ values, or how PLS component counts were searched. These details are needed for reproducibility.
  4. [Section 4] The sentence 'Systemically designing features' contains a typo and should read 'Systematically designing features'; similar minor typographical issues appear elsewhere, e.g., 'V oltage' in the Introduction and 'T est' in Figure 1.
  5. [Section A.2] The manuscript refers to 'the software accompanying the article' but does not provide a repository or DOI; please include the URL or a data/code availability statement.

Circularity Check

1 steps flagged · score 6.0 of 10

Closest-coefficient selection makes the reported similarity to feature coefficients partly a construction of the ℓ2 objective, not an independent finding.

  1. fitted input called prediction [Section 2, Eqs. (9)-(10); Section 3, Case Study I]
    "we suggest estimating the regularization parameters based on the ℓ2-norm, min λ ||β(λ) − βT1||2 2 ... The RR and PLS regression coefficients with minimal ℓ2-distance to the feature coefficients have a very similar shape as the feature coefficients."

    Equation (9) defines the 'closest' regression coefficient as the minimizer of ||β(λ)-βT1||², and the case studies then exhibit the coefficient at that minimizer as having 'very similar shape' to βT1. The displayed similarity is the optimum of the objective used to select λ, so the conclusion that the path 'contains regression coefficients similar to the linearized true mapping' is partly a restatement of the selection rule rather than an independent finding. Since β(λ) lies in the row space of X while βT1 generally has a nullspace component, the minimized ℓ2 distance only controls the row-space projection; the full-coefficient closeness is not quantified.

full rationale

The central comparison is in-sample: βT1=m∇g(x̄) uses an OLS scalar m fit to the same response y, and Eq. (10) chooses λ by minimizing the very ℓ2 distance to βT1 that the case studies then display. The conclusion that the path 'contains regression coefficients similar to the linearized true mapping' is therefore partly a restatement of the selection rule: the chosen coefficients are by definition the closest in that norm. The remaining independent content is the qualitative shape agreement at the minimizing λ and the fact that this λ lies in the strong-regularization regime; this content is real but not quantified. The paper explicitly labels the selected coefficients as 'min. dist.', so the circularity is not hidden. The nullspace remark in Sec. A.3 further shows that the ℓ2 distance only controls the row-space projection of βT1, so the full-coefficient similarity claim is not established by Eq. (10). Self-citations [8] and [11] are used for standard properties (nullspace orthogonality, prediction-equivalent coefficients) and are not load-bearing circularity. Overall: one step of the comparison reduces to its optimization objective, giving partial circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the existence and differentiability of a known compressing feature g, a scalar factor m fitted by OLS, and a coefficient-space distance metric. These are stated in Section 2 and Sec. A.3. The paper introduces no new physical entities; the scaled gradient beta_T1 is derived from g and m rather than postulated independently.

free parameters (2)
  • Scalar factor m = Estimated by OLS in Eq. (7)
    Scalar factor fitted to match the linearized feature response to y; without this fit the feature coefficients would not estimate the true response, but it is a fitted parameter.
  • Sinusoidal frequency constant = 2*pi/0.06 (V^-1)
    The frequency in Case Study II is chosen ad hoc and affects the shape of the feature coefficients.
assumptions (4)
  • domain assumption Assumption 1: There exists a nonlinear compressing feature g such that g(x_i) = y_i + epsilon_i.
    Stated in Section 2 before Eq. (2); without such a g the feature coefficients cannot be defined.
  • domain assumption Assumption 2: The irreducible error is zero, epsilon_i* = 0 for all i.
    Stated in Section 2; the authors relax only by fitting m to absorb some error, but the case studies construct y directly from g.
  • domain assumption Assumption 3: g is differentiable.
    Stated in Section 2; required for the first-order Taylor expansion in Eq. (2).
  • ad hoc to paper The l2 norm in coefficient space is a meaningful distance for comparing regression and feature coefficients.
    Used in Eq. (9); the authors themselves note in Sec. A.3 that coefficient-space distance can be large while predictions are similar due to the nullspace, so this metric is a modeling choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretation of High-Dimensional Regression Coefficients by Comparison with Linearized Compressing Features." pith.science (2026). https://pith.science/paper/BH2SLXN6

@misc{pith2026241112060,
  author       = {Pith},
  title        = {Pith review of: Interpretation of High-Dimensional Regression Coefficients by Comparison with Linearized Compressing Features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BH2SLXN6}},
  note         = {Machine review of arXiv:2411.12060}
}
abstract

Linear regression is often deemed inherently interpretable; however, challenges arise for high-dimensional data. We focus on further understanding how linear regression approximates nonlinear responses from high-dimensional functional data, motivated by predicting cycle life for lithium-ion batteries. We develop a linearization method to derive feature coefficients, which we compare with the closest regression coefficients of the path of regression solutions. We showcase the methods on battery data case studies where a single nonlinear compressing feature, $g\colon \mathbb{R}^p \to \mathbb{R}$, is used to construct a synthetic response, $\mathbf{y} \in \mathbb{R}$. This unifying view of linear regression and compressing features for high-dimensional functional data helps to understand (1) how regression coefficients are shaped in the highly regularized domain and how they relate to linearized feature coefficients and (2) how the shape of regression coefficients changes as a function of regularization to approximate nonlinear responses by exploiting local structures.

Figures

Figures reproduced from arXiv: 2411.12060 by the authors.

Figure 1
Figure 1. Lithium-ion data from discharge cycles [12]. a) Training, primary test, and secondary test data are plotted [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. First case study with the sum-of-squares response. a) PLS and RR with high regularization and sum-of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Second case study with the sinusoidal response. a) PLS high regularization and sinusoidal feature coeffi [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages

  1. [1]

    Special issue on feature engineer- ing editorial

    Tim Verdonck, Bart Baesens, María Óskarsdóttir, and Seppe vanden Broucke. Special issue on feature engineer- ing editorial. Machine learning, 113(7):3917–3928, 2024

    work page 2024

  2. [2]

    Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists

    Alice Zheng and Casari Amanda. Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O’Reilly, Beijing, 2018

    work page 2018

  3. [3]

    A few useful things to know about machine learning

    Pedro Domingos. A few useful things to know about machine learning. Communications of the ACM, 55(10): 78–87, 2012

    work page 2012

  4. [4]

    Friedman, and Jerome H

    Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, New York, 2009

    work page 2009

  5. [5]

    An Introduction to Statistical Learning: with Applications in R

    James Gareth, Witten Daniela, Hastie Trevor, and Tibshirani Robert. An Introduction to Statistical Learning: with Applications in R. Springer, New York, 2021

    work page 2021

  6. [6]

    Joachim Schaeffer and Richard D. Braatz. Latent variable method demonstrator – Software for understanding multivariate data analytics algorithms. Computers & Chemical Engineering, 167:108014, 2022

    work page 2022

  7. [7]

    Introduction to Linear Algebra

    Gilbert Strang. Introduction to Linear Algebra. Cambridge Press, Wellesley, Massachusetts, fifth edition, 2016

    work page 2016

  8. [8]

    Chueh, Martin Z

    Joachim Schaeffer, Eric Lenz, William C. Chueh, Martin Z. Bazant, Rolf Findeisen, and Richard D. Braatz. Interpretation of high-dimensional linear regression: Effects of nullspace and regularization demonstrated on battery data. Computers & Chemical Engineering, 180:108471, 2024

    work page 2024

Show all 16 references
  1. [9]

    J. O. Ramsay and B. W. Silverman. Functional Data Analysis. Springer, New York, second edition, 2005

    2005

  2. [10]

    James, Jing Wang, and Ji Zhu

    Gareth M. James, Jing Wang, and Ji Zhu. Functional linear regression that’s interpretable. The Annals of Statistics, 37(5A):2083–2108, 2009

    2009

  3. [11]

    Asinger, Robin Droop, Rolf Findeisen, and Richard D

    Joachim Schaeffer, Giacomo Galuppini, Jinwook Rhyu, Patrick A. Asinger, Robin Droop, Rolf Findeisen, and Richard D. Braatz. Cycle life prediction for lithium-ion batteries: Machine learning and more. In Proceedings of the American Control Conference, pages 763–768, 2024

    2024

  4. [12]

    Severson, Peter M

    Kristen A. Severson, Peter M. Attia, Norman Jin, Nicholas Perkins, Benben Jiang, Zi Yang, Michael H. Chen, Muratahan Aykol, Patrick K. Herring, Dimitrios Fraggedakis, Martin Z. Bazant, Stephen J. Harris, William C. Chueh, and Richard D. Braatz. Data-driven prediction of batter...

    2019

  5. [13]

    J. J. Duistermaat. Distributions: Theory and Applications. Birkhäuser, Boston, Massachusetts, 2010

    2010

  6. [14]

    The Analysis of Linear Partial Differential Operators I: Distribution Theory and Fourier Analysis

    Lars Hörmander. The Analysis of Linear Partial Differential Operators I: Distribution Theory and Fourier Analysis. Springer, Berlin–Heidelberg, second edition, 2003

    2003

  7. [15]

    Repeated double cross validation

    Peter Filzmoser, Bettina Liebmann, and Kurt Varmuza. Repeated double cross validation. Journal of Chemo- metrics, 23(4):160–171, 2009

    2009

  8. [16]

    Li, Xiao Cui, William C

    Jinwook Rhyu, Joachim Schaeffer, Michael L. Li, Xiao Cui, William C. Chueh, Martin Z. Bazant, and Richard D. Braatz. Systematic feature design for cycle life prediction of lithium-ion batteries during formation. arXiv preprint arXiv:2410.07458, 2024. 9

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.