Pith. sign in

REVIEW 4 major objections 4 minor 41 references

Least Squares Approximation for a Distributed System

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A one-round, inverse-covariance weighted average of local regression fits achieves the same asymptotic efficiency as the full-data estimator.

desk verdict The theory is worth engaging, but the numerical section currently fails a basic integrity check. read the letter →

arxiv 1908.04904 v4 pith:G7MTELQX submitted 2019-08-14 stat.ME cs.DCcs.LGstat.COstat.ML

classification stat.MEcs.DCcs.LGstat.COstat.ML MSC 62F1262J07
keywords distributedsystemsleastsquaresapproximationone-shotcommunicationinversecovarianceweightingadaptiveLassooraclepropertyBICheterogeneousdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a distributed estimation method for regression problems on a master-and-worker system: each worker fits the model locally and sends its estimate and an estimated covariance to the master, which combines them by inverse-covariance weighting. The central claim is that this one-round combination is asymptotically as efficient as the global estimator, meaning it has the same limiting normal distribution with the same covariance. The paper further shows that an adaptive Lasso step on the master, guided by a new distributed Bayesian information criterion, selects the true model consistently and achieves the oracle property. This matters because it would give large-scale regression analysis the statistical benefits of full-data estimation while avoiding the communication cost of iterative distributed algorithms.

What carries the argument

The load-bearing object is the local quadratic approximation of the global loss: expanding each local loss around its local minimizer and dropping the linear term gives a weighted least squares objective $\tilde{L}(\theta)=\sum_k \alpha_k (\theta-\hat{\theta}_k)^\top \hat{\Sigma}_k^{-1}(\theta-\hat{\theta}_k)$, whose minimizer is an explicit inverse-covariance weighted average. The inverse covariance $\hat{\Sigma}_k^{-1}$ is the key ingredient: it lets the master combine heterogeneous local fits by down-weighting workers with larger asymptotic variance. The same quadratic form, evaluated on the master, serves as the loss for adaptive Lasso shrinkage and as the basis for the distributed BIC selection criterion.

What would settle it

Simulate a Poisson log-linear working model on negative-binomial data distributed across workers, with each worker's sample size $n$ much larger than $\sqrt{N}$, and compare the DLSA estimator's sampling covariance with that of the global estimator; if they differ in any component at the claimed rate, the information-equality condition underlying the theorem is violated.

Watch

Extended reading notes

Core claim

The central result is Theorem 1: under regularity conditions including each local sample size $n$ growing faster than $\sqrt{N}$, the DLSA estimator $\tilde{\theta} = (\sum_k \alpha_k \hat{\Sigma}_k^{-1})^{-1} \sum_k \alpha_k \hat{\Sigma}_k^{-1} \hat{\theta}_k$ satisfies $\sqrt{N}(\tilde{\theta}-\theta_0)\to_d N(0,\Sigma)$, the same asymptotic distribution as the global estimator. Intuitively, the inverse-covariance weighting turns a collection of $\sqrt{n}$-consistent local estimators into a $\sqrt{N}$-consistent global one, and its bias term vanishes when workers are large enough. A two-step variant improves the bias rate and relaxes the local-sample condition to $n\gg N^{1/4}$, while iterating the procedure $m$ times with $m/\log N\to\infty$ allows even $n=O(1)$. The paper also proves that the adaptive Lasso on the master is selection consistent and oracle-efficient, and that the proposed distributed BIC consistently identifies the true model.

Load-bearing premise

The defining efficiency result assumes that within each worker the information equality holds: the variance of the local score equals the expected curvature of the local loss, so the inverse covariance is the right weight.

Editorial extensions

If this is right

  • Large regression problems such as linear, logistic, Poisson, and Cox-type models can be analyzed on a distributed cluster with a single round of communication while retaining full-data asymptotic efficiency.
  • Heterogeneous covariate distributions across workers do not destroy efficiency, because per-worker inverse-covariance weighting automatically accounts for differing local variability.
  • Variable selection and tuning-parameter selection can be done entirely on the master with negligible extra communication, and the selected model is consistent.
  • The two-step and multi-step variants extend the approach to settings with many weak workers or severe local memory constraints.
  • On very large datasets, the method requires modest worker memory and is substantially faster than conventional distributed algorithms, as demonstrated by the 52 GB airline application.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The inverse-covariance weighting principle is more general than the paper's likelihood setting: any collection of $\sqrt{n}$-consistent, independently computed estimates with reported covariance matrices could be combined the same way, though the efficiency guarantee would require the reported covariance to be the true one.
  • Because the efficiency theorem rests on the information equality in Condition (C4), a practical check would be to compare each worker's score outer product with its Hessian; a large discrepancy would signal that the DLSA estimator may no longer match the global estimator's covariance.
  • The paper's appendix suggests that bootstrap covariance estimates could replace the Hessian when it is hard to compute, which points toward a plug-in variant of DLSA for semi-parametric or robust losses.
  • The DBIC construction could plausibly be adapted to other penalties (for example, SCAD or MCP) because it only requires a consistent global quadratic surrogate and a consistent reference estimator.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops a one-shot distributed estimation procedure, DLSA, in which each worker fits the same parametric model locally and the master combines the local estimators by inverse-variance-type weights. Under Conditions C1–C5 it claims that the combined estimator has the same √N-limiting distribution as the global estimator, and it adds an adaptive-Lasso version with a new DBIC criterion that is claimed to have oracle and selection-consistency properties. The paper reports simulations for linear, logistic, and Poisson regressions under i.i.d. and heterogeneous covariate distributions, and it illustrates the method on a 52 GB airline dataset using a Spark implementation.

Significance. If the theoretical claims are correct, the paper provides a communication-efficient and practically useful alternative to iterative distributed estimation, and it goes beyond much of the one-shot literature by explicitly addressing heterogeneous worker-specific covariate distributions and by providing a tuning-parameter selection criterion. The proofs are self-contained from explicit conditions, and the availability of an open-source Spark implementation is a concrete strength. However, the numerical validation currently contains a duplicated table, and the scaling conventions for the covariance matrix are inconsistent between the main text, the proofs, and the reported coverage probabilities. These issues must be resolved before the claims can be accepted.

major comments (4)
  1. [Section 4.4, Table 3] Table 3, labeled Poisson regression (Example 3), is numerically identical to Table 2, labeled logistic regression, in both panels (N = 20, K = 10 and N = 100, K = 20). The paper therefore reports no actual results for the Poisson example, and the sentence in Section 4.4 discussing the Poisson model refers to a table that is a copy of the logistic results. This must be corrected by rerunning the simulation and by checking all tables for similar copy-and-paste errors.
  2. [Section 2.2, Condition (C4)] Condition (C4) is described as a local convexity condition, but it is the information equality E[score score^T] = E[Hessian] for each worker. Under a misspecified working loss, the local estimator has the sandwich covariance J_k^{-1} I_k J_k^{-1}, and the inverse-Hessian weights in (2.2) do not generally reproduce the global estimator's limiting distribution. Please state explicitly that L is a correctly specified negative log-likelihood, or provide a misspecification analysis; the unqualified claim that DLSA is statistically as efficient as the global estimator is otherwise broader than the proof supports.
  3. [Section 3.2, Eq. (3.4)] The DBIC as printed, DBIC_lambda = (theta_tilde_lambda - theta_tilde)^T Sigma_hat^{-1} (theta_tilde_lambda - theta_tilde) + log N * df_lambda, cannot be consistent with Theorem 5. With Sigma_hat denoting the asymptotic covariance matrix of sqrt(N)(theta_tilde - theta_0) from (2.2) and Proposition 1, the quadratic term is O_p(1/N), while the penalty diverges, so the criterion would select the null model. Appendix B.3 uses df_lambda (log N)/N, so the main-text formula appears to be missing a factor 1/N. Please correct (3.4) and use one unambiguous convention for Sigma_hat throughout.
  4. [Section 4.3, CI formula] The confidence intervals are defined with the standard error equal to a diagonal element of Sigma_hat = (sum_k alpha_k Sigma_hat_k^{-1})^{-1}. With Sigma_hat as defined in (2.2) and Proposition 1, this is the asymptotic covariance matrix of sqrt(N)(theta_tilde - theta_0), so the interval width is O(1) and coverage would be essentially one, not approximately 0.95 as reported in Tables 1-3. The intervals should use Sigma_hat/N, or Sigma_hat should be redefined as the covariance of theta_tilde itself. Please clarify and reconcile the stated formula with the reported coverage probabilities.
minor comments (4)
  1. [Appendix A.2] The sentence 'by the condition n >> sqrt(N), we have K >> sqrt(N)' reverses the implication: since n = N/K, n >> sqrt(N) implies K << sqrt(N), which is exactly K = o(sqrt(N)).
  2. [Appendix A.2 and A.3] The proofs of Theorems 1 and 2 cite Condition (C6), which is introduced in Section 3.1 and is not assumed by these theorems; the variance calculation should cite Condition (C4).
  3. [Section 2.2] The phrase 'asymptotic covariance estimate' for Sigma_hat_k is ambiguous: it should be stated whether this is the covariance of sqrt(n_k)(theta_hat_k - theta_0) or of theta_hat_k itself; the same distinction underlies the factor-N issue in the DBIC and confidence interval formulas.
  4. [Abstract and Section 1] The abstract lists Cox's model as a target example, but no Cox simulation or verification that the Cox partial likelihood satisfies Condition (C4) is provided; a remark or additional example would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DLSA asymptotic result is proved from explicit conditions rather than fitted to the target; the Wang-Leng self-citation is not load-bearing.

full rationale

The derivation chain is self-contained. The DLSA estimator in (2.2) is an explicit inverse-covariance weighted average of local estimators, and Theorem 1 derives its limiting distribution in Appendix A.1-A.2 by standard Taylor expansions under Conditions (C1)-(C5), rather than by plugging in the claimed global covariance. Condition (C4) is a genuine information-equality assumption about each local loss (expected outer product of score equals expected Hessian), not a parameter fitted to reproduce the global estimator; if the working likelihood is misspecified the stated covariance can fail, but that is a limitation of the assumptions, not circularity. The citations to Wang and Leng (2007) supply the LSA surrogate idea and a discussion of Condition (C6); because that paper is a published, independently falsifiable methodological basis and the present proofs do not rely on an unproved theorem from it, the overlap of one author does not make the argument circular. The paper also flags its own limitations (lack of second derivatives for quantile regression in Section 6, and post-selection bias from pre-screening in Section 4.2), consistent with an honest conditional derivation. The typographical 'K≫√N' in Appendix A.2 (the argument needs K≪√N) is a proof-reading slip and does not indicate a circular step.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The method relies on standard M-estimator regularity conditions and a common regression coefficient across heterogeneous workers. No new physical or conceptual entities are introduced. The only data-dependent quantity is the adaptive Lasso tuning parameter, selected by the proposed DBIC.

free parameters (1)
  • Adaptive Lasso tuning parameter lambda0 = Selected by DBIC
    The oracle property of the shrinkage estimator requires lambda0 to satisfy the rate conditions in Theorem 3; DBIC is proposed to select it consistently. This is a data-driven tuning parameter, not a hand-fitted constant, but it affects the variable selection claim.
assumptions (5)
  • domain assumption Condition (C1): parameter space is compact and convex, true value in interior
    Standard regularity for M-estimators, used to ensure existence and consistency of local estimates.
  • domain assumption Condition (C2): covariates within each worker are i.i.d. from a worker-specific distribution F_k
    This is the heterogeneity assumption; it allows different covariate distributions across workers but maintains a common regression parameter.
  • domain assumption Condition (C4): information equality for each local loss, E[score score'] = E[Hessian]
    This is the key assumption that makes inverse observed Hessian the correct covariance for each local estimator. It is the weakest load-bearing assumption.
  • domain assumption Condition (C5): third derivatives of the loss exist and have bounded moments
    Smoothness condition used in Taylor expansions to control bias terms.
  • domain assumption Condition (C6): covariance submatrix of the global estimator for an overfitted model matches the inverse Hessian submatrix
    This ensures the oracle estimator distribution in Theorem 4. It is stated as holding for likelihood models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Least Squares Approximation for a Distributed System." pith.science (2026). https://pith.science/paper/G7MTELQX

@misc{pith2026190804904,
  author       = {Pith},
  title        = {Pith review of: Least Squares Approximation for a Distributed System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7MTELQX}},
  note         = {Machine review of arXiv:1908.04904}
}
read the original abstract

In this work, we develop a distributed least squares approximation (DLSA) method that is able to solve a large family of regression problems (e.g., linear regression, logistic regression, and Cox's model) on a distributed system. By approximating the local objective function using a local quadratic form, we are able to obtain a combined estimator by taking a weighted average of local estimators. The resulting estimator is proved to be statistically as efficient as the global estimator. Moreover, it requires only one round of communication. We further conduct a shrinkage estimation based on the DLSA estimation using an adaptive Lasso approach. The solution can be easily obtained by using the LARS algorithm on the master node. It is theoretically shown that the resulting estimator possesses the oracle property and is selection consistent by using a newly designed distributed Bayesian information criterion (DBIC). The finite sample performance and computational efficiency are further illustrated by an extensive numerical study and an airline dataset. The airline dataset is 52 GB in size. The entire methodology has been implemented in Python for a {\it de-facto} standard Spark system. The proposed DLSA algorithm on the Spark system takes 26 minutes to obtain a logistic regression estimator, which is more efficient and memory friendly than conventional methods.

Figures

Figures reproduced from arXiv: 1908.04904 by the authors.

Figure 1
Figure 1. Illustration of the DLSA method. function. The objective functions can be easily solved by the LARS algorithm (Efron et al., 2004) with minimal computation cost on the master. Thus, no communication is required. Accordingly, a solution path can be obtained on the master node. Thereafter, the best estimator can be selected from the solution path in conjunction with the proposed distributed Bayesian information criter… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 36 canonical work pages

  1. [1]

    Apache Software Foundation (2019 a ), Apache Hadoop,

  2. [2]

    --- (2019 b ), Apache Spark,

  3. [3]

    (2015), Distributed estimation and inference with statistical guarantees, arXiv preprint arXiv:1509.05457

    Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2015), Distributed estimation and inference with statistical guarantees, arXiv preprint arXiv:1509.05457

  4. [4]

    Cameron, A. C. and Trivedi, P. K. (2013), Regression analysis of count data, vol. 53, Cambridge university press

  5. [5]

    (2017 a ), Divide and conquer local average regression, Electronic Journal of Statistics, 11, 1326--1350

    Chang, X., Lin, S.-B., and Wang, Y. (2017 a ), Divide and conquer local average regression, Electronic Journal of Statistics, 11, 1326--1350

  6. [6]

    (2017 b ), Distributed semi-supervised learning with kernel ridge regression, The Journal of Machine Learning Research, 18, 1493--1514

    Chang, X., Lin, S.-B., and Zhou, D.-X. (2017 b ), Distributed semi-supervised learning with kernel ridge regression, The Journal of Machine Learning Research, 18, 1493--1514

  7. [7]

    and Chen, Z

    Chen, J. and Chen, Z. (2008), Extended Bayesian information criteria for model selection with large model spaces, Biometrika, 95, 759--771

  8. [8]

    and Guestrin, C

    Chen, T. and Guestrin, C. (2016), Xgboost: A scalable tree boosting system, in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, ACM, pp. 785--794

Show all 41 references
  1. [9]

    (2018), Quantile regression under memory constraint, arXiv preprint arXiv:1810.08264

    Chen, X., Liu, W., and Zhang, Y. (2018), Quantile regression under memory constraint, arXiv preprint arXiv:1810.08264

  2. [10]

    and Xie, M.-g

    Chen, X. and Xie, M.-g. (2014), A split-and-conquer approach for analysis of extraordinarily large data, Statistica Sinica, 1655--1684

  3. [11]

    (2004), Least angle regression, Annals of Statistics, 32, 407--499

    Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004), Least angle regression, Annals of Statistics, 32, 407--499

  4. [12]

    and Li, R

    Fan, J. and Li, R. (2001), Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348--1360

  5. [13]

    --- (2002), Variable selection for Cox's proportional hazards model and frailty model, The Annals of Statistics, 30, 74--99

  6. [14]

    (2017), Distributed estimation of principal eigenspaces, arXiv preprint arXiv:1702.06488

    Fan, J., Wang, D., Wang, K., and Zhu, Z. (2017), Distributed estimation of principal eigenspaces, arXiv preprint arXiv:1702.06488

  7. [15]

    Harrell Jr, F. E. (2015), Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis, Springer

  8. [16]

    (2016), Dual-loco: Distributing statistical estimation using random projections, in Artificial Intelligence and Statistics, pp

    Heinze, C., McWilliams, B., and Meinshausen, N. (2016), Dual-loco: Distributing statistical estimation using random projections, in Artificial Intelligence and Statistics, pp. 875--883

  9. [17]

    W., Lemeshow, S., and Sturdivant, R

    Hosmer Jr, D. W., Lemeshow, S., and Sturdivant, R. X. (2013), Applied logistic regression, vol. 398, John Wiley & Sons

  10. [18]

    I., Lee, J

    Jordan, M. I., Lee, J. D., and Yang, Y. (2018), Communication-efficient distributed statistical inference, Journal of the American Statistical Association, 1--14

  11. [19]

    D., Sun, Y., Liu, Q., and Taylor, J

    Lee, J. D., Sun, Y., Liu, Q., and Taylor, J. E. (2015), Communication-efficient sparse regression: a one-shot approach, arXiv preprint arXiv:1503.04337

  12. [20]

    Lehmann, E. L. and Casella, G. (2006), Theory of point estimation, Springer Science & Business Media

  13. [21]

    (2019), Distributed feature screening via componentwise debiasing, arXiv preprint arXiv:1903.03810

    Li, X., Li, R., Xia, Z., and Xu, C. (2019), Distributed feature screening via componentwise debiasing, arXiv preprint arXiv:1903.03810

  14. [22]

    and Ihler, A

    Liu, Q. and Ihler, A. T. (2014), Distributed estimation, information loss and exponential families, in Advances in neural information processing systems, pp. 1098--1106

  15. [23]

    (2014), Communication-efficient distributed optimization using an approximate newton-type method, in International conference on machine learning, pp

    Shamir, O., Srebro, N., and Zhang, T. (2014), Communication-efficient distributed optimization using an approximate newton-type method, in International conference on machine learning, pp. 1000--1008

  16. [24]

    (1997), An asymptotic theory for linear model selection, Statistica Sinica, 221--242

    Shao, J. (1997), An asymptotic theory for linear model selection, Statistica Sinica, 221--242

  17. [25]

    I., and Jaggi, M

    Smith, V., Forte, S., Ma, C., Tak \'a c , M., Jordan, M. I., and Jaggi, M. (2018), CoCoA: A General Framework for Communication-Efficient Distributed Optimization, Journal of Machine Learning Research, 18, 1--49

  18. [26]

    (1996), Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society

    Tibshirani, R. (1996), Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society. Series B, 267--288

  19. [27]

    and Leng, C

    Wang, H. and Leng, C. (2007), Unified LASSO estimation by least squares approximation, Journal of the American Statistical Association, 102, 1039--1048

  20. [28]

    (2007), Tuning parameter selectors for the smoothly clipped absolute deviation method, Biometrika, 94, 553--568

    Wang, H., Li, R., and Tsai, C.-L. (2007), Tuning parameter selectors for the smoothly clipped absolute deviation method, Biometrika, 94, 553--568

  21. [29]

    (2018), Optimal subsampling for large sample logistic regression, Journal of the American Statistical Association, 113, 829--844

    Wang, H., Zhu, R., and Ma, P. (2018), Optimal subsampling for large sample logistic regression, Journal of the American Statistical Association, 113, 829--844

  22. [30]

    (2017 a ), Efficient distributed learning with sparsity, in Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR

    Wang, J., Kolar, M., Srebro, N., and Zhang, T. (2017 a ), Efficient distributed learning with sparsity, in Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp. 3636--3645

  23. [31]

    (2017 b ), Memory and communication efficient distributed stochastic optimization with minibatch prox, in Proceedings of the 2017 Conference on Learning Theory, eds

    Wang, J., Wang, W., and Srebro, N. (2017 b ), Memory and communication efficient distributed stochastic optimization with minibatch prox, in Proceedings of the 2017 Conference on Learning Theory, eds. Kale, S. and Shamir, O., Amsterdam, Netherlands: PMLR, vol. 65 of Proceeding...

  24. [32]

    (2013), Calibrating non-convex penalized regression in ultra-high dimension, Annals of statistics, 41, 2505

    Wang, L., Kim, Y., and Li, R. (2013), Calibrating non-convex penalized regression in ultra-high dimension, Annals of statistics, 41, 2505

  25. [33]

    W., Saunders, M., and Sun, Y

    Yang, J., Mahoney, M. W., Saunders, M., and Sun, Y. (2016), Feature-distributed sparse regression: a screen-and-clean approach, in Advances in Neural Information Processing Systems, pp. 2712--2720

  26. [34]

    J., Shenker, S., and Stoica, I

    Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012), Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing, in Proceedings of the 9th USENIX conference on Networked Syst...

  27. [35]

    (2010), Nearly unbiased variable selection under minimax concave penalty, Annals of Statistics, 38, 894--942

    Zhang, C.-H. (2010), Nearly unbiased variable selection under minimax concave penalty, Annals of Statistics, 38, 894--942

  28. [36]

    Zhang, H. H. and Lu, W. (2007), Adaptive Lasso for Cox's proportional hazards model, Biometrika, 94, 691--703

  29. [37]

    Zhang, T. (2004), Solving large scale linear prediction problems using stochastic gradient descent algorithms, in Proceedings of the twenty-first international conference on Machine learning, ACM, p. 116

  30. [38]

    C., and Wainwright, M

    Zhang, Y., Duchi, J. C., and Wainwright, M. J. (2013), Communication-efficient algorithms for statistical optimization, The Journal of Machine Learning Research, 14, 3321--3363

  31. [39]

    (2006), The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418--1429

    Zou, H. (2006), The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418--1429

  32. [40]

    and Li, R

    Zou, H. and Li, R. (2008), One-step sparse estimates in nonconcave penalized likelihood models, Annals of statistics, 36, 1509

  33. [41]

    and Zhang, H

    Zou, H. and Zhang, H. H. (2009), On the adaptive elastic-net with a diverging number of parameters, Annals of statistics, 37, 1733

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.