REVIEW 4 major objections 4 minor 41 references
Least Squares Approximation for a Distributed System
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A one-round, inverse-covariance weighted average of local regression fits achieves the same asymptotic efficiency as the full-data estimator.
desk verdict The theory is worth engaging, but the numerical section currently fails a basic integrity check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the local quadratic approximation of the global loss: expanding each local loss around its local minimizer and dropping the linear term gives a weighted least squares objective $\tilde{L}(\theta)=\sum_k \alpha_k (\theta-\hat{\theta}_k)^\top \hat{\Sigma}_k^{-1}(\theta-\hat{\theta}_k)$, whose minimizer is an explicit inverse-covariance weighted average. The inverse covariance $\hat{\Sigma}_k^{-1}$ is the key ingredient: it lets the master combine heterogeneous local fits by down-weighting workers with larger asymptotic variance. The same quadratic form, evaluated on the master, serves as the loss for adaptive Lasso shrinkage and as the basis for the distributed BIC selection criterion.
What would settle it
Simulate a Poisson log-linear working model on negative-binomial data distributed across workers, with each worker's sample size $n$ much larger than $\sqrt{N}$, and compare the DLSA estimator's sampling covariance with that of the global estimator; if they differ in any component at the claimed rate, the information-equality condition underlying the theorem is violated.
Extended reading notes
Core claim
The central result is Theorem 1: under regularity conditions including each local sample size $n$ growing faster than $\sqrt{N}$, the DLSA estimator $\tilde{\theta} = (\sum_k \alpha_k \hat{\Sigma}_k^{-1})^{-1} \sum_k \alpha_k \hat{\Sigma}_k^{-1} \hat{\theta}_k$ satisfies $\sqrt{N}(\tilde{\theta}-\theta_0)\to_d N(0,\Sigma)$, the same asymptotic distribution as the global estimator. Intuitively, the inverse-covariance weighting turns a collection of $\sqrt{n}$-consistent local estimators into a $\sqrt{N}$-consistent global one, and its bias term vanishes when workers are large enough. A two-step variant improves the bias rate and relaxes the local-sample condition to $n\gg N^{1/4}$, while iterating the procedure $m$ times with $m/\log N\to\infty$ allows even $n=O(1)$. The paper also proves that the adaptive Lasso on the master is selection consistent and oracle-efficient, and that the proposed distributed BIC consistently identifies the true model.
Load-bearing premise
The defining efficiency result assumes that within each worker the information equality holds: the variance of the local score equals the expected curvature of the local loss, so the inverse covariance is the right weight.
Editorial extensions
If this is right
- Large regression problems such as linear, logistic, Poisson, and Cox-type models can be analyzed on a distributed cluster with a single round of communication while retaining full-data asymptotic efficiency.
- Heterogeneous covariate distributions across workers do not destroy efficiency, because per-worker inverse-covariance weighting automatically accounts for differing local variability.
- Variable selection and tuning-parameter selection can be done entirely on the master with negligible extra communication, and the selected model is consistent.
- The two-step and multi-step variants extend the approach to settings with many weak workers or severe local memory constraints.
- On very large datasets, the method requires modest worker memory and is substantially faster than conventional distributed algorithms, as demonstrated by the 52 GB airline application.
Reading between the lines
- The inverse-covariance weighting principle is more general than the paper's likelihood setting: any collection of $\sqrt{n}$-consistent, independently computed estimates with reported covariance matrices could be combined the same way, though the efficiency guarantee would require the reported covariance to be the true one.
- Because the efficiency theorem rests on the information equality in Condition (C4), a practical check would be to compare each worker's score outer product with its Hessian; a large discrepancy would signal that the DLSA estimator may no longer match the global estimator's covariance.
- The paper's appendix suggests that bootstrap covariance estimates could replace the Hessian when it is hard to compute, which points toward a plug-in variant of DLSA for semi-parametric or robust losses.
- The DBIC construction could plausibly be adapted to other penalties (for example, SCAD or MCP) because it only requires a consistent global quadratic surrogate and a consistent reference estimator.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops a one-shot distributed estimation procedure, DLSA, in which each worker fits the same parametric model locally and the master combines the local estimators by inverse-variance-type weights. Under Conditions C1–C5 it claims that the combined estimator has the same √N-limiting distribution as the global estimator, and it adds an adaptive-Lasso version with a new DBIC criterion that is claimed to have oracle and selection-consistency properties. The paper reports simulations for linear, logistic, and Poisson regressions under i.i.d. and heterogeneous covariate distributions, and it illustrates the method on a 52 GB airline dataset using a Spark implementation.
Significance. If the theoretical claims are correct, the paper provides a communication-efficient and practically useful alternative to iterative distributed estimation, and it goes beyond much of the one-shot literature by explicitly addressing heterogeneous worker-specific covariate distributions and by providing a tuning-parameter selection criterion. The proofs are self-contained from explicit conditions, and the availability of an open-source Spark implementation is a concrete strength. However, the numerical validation currently contains a duplicated table, and the scaling conventions for the covariance matrix are inconsistent between the main text, the proofs, and the reported coverage probabilities. These issues must be resolved before the claims can be accepted.
major comments (4)
- [Section 4.4, Table 3] Table 3, labeled Poisson regression (Example 3), is numerically identical to Table 2, labeled logistic regression, in both panels (N = 20, K = 10 and N = 100, K = 20). The paper therefore reports no actual results for the Poisson example, and the sentence in Section 4.4 discussing the Poisson model refers to a table that is a copy of the logistic results. This must be corrected by rerunning the simulation and by checking all tables for similar copy-and-paste errors.
- [Section 2.2, Condition (C4)] Condition (C4) is described as a local convexity condition, but it is the information equality E[score score^T] = E[Hessian] for each worker. Under a misspecified working loss, the local estimator has the sandwich covariance J_k^{-1} I_k J_k^{-1}, and the inverse-Hessian weights in (2.2) do not generally reproduce the global estimator's limiting distribution. Please state explicitly that L is a correctly specified negative log-likelihood, or provide a misspecification analysis; the unqualified claim that DLSA is statistically as efficient as the global estimator is otherwise broader than the proof supports.
- [Section 3.2, Eq. (3.4)] The DBIC as printed, DBIC_lambda = (theta_tilde_lambda - theta_tilde)^T Sigma_hat^{-1} (theta_tilde_lambda - theta_tilde) + log N * df_lambda, cannot be consistent with Theorem 5. With Sigma_hat denoting the asymptotic covariance matrix of sqrt(N)(theta_tilde - theta_0) from (2.2) and Proposition 1, the quadratic term is O_p(1/N), while the penalty diverges, so the criterion would select the null model. Appendix B.3 uses df_lambda (log N)/N, so the main-text formula appears to be missing a factor 1/N. Please correct (3.4) and use one unambiguous convention for Sigma_hat throughout.
- [Section 4.3, CI formula] The confidence intervals are defined with the standard error equal to a diagonal element of Sigma_hat = (sum_k alpha_k Sigma_hat_k^{-1})^{-1}. With Sigma_hat as defined in (2.2) and Proposition 1, this is the asymptotic covariance matrix of sqrt(N)(theta_tilde - theta_0), so the interval width is O(1) and coverage would be essentially one, not approximately 0.95 as reported in Tables 1-3. The intervals should use Sigma_hat/N, or Sigma_hat should be redefined as the covariance of theta_tilde itself. Please clarify and reconcile the stated formula with the reported coverage probabilities.
minor comments (4)
- [Appendix A.2] The sentence 'by the condition n >> sqrt(N), we have K >> sqrt(N)' reverses the implication: since n = N/K, n >> sqrt(N) implies K << sqrt(N), which is exactly K = o(sqrt(N)).
- [Appendix A.2 and A.3] The proofs of Theorems 1 and 2 cite Condition (C6), which is introduced in Section 3.1 and is not assumed by these theorems; the variance calculation should cite Condition (C4).
- [Section 2.2] The phrase 'asymptotic covariance estimate' for Sigma_hat_k is ambiguous: it should be stated whether this is the covariance of sqrt(n_k)(theta_hat_k - theta_0) or of theta_hat_k itself; the same distinction underlies the factor-N issue in the DBIC and confidence interval formulas.
- [Abstract and Section 1] The abstract lists Cox's model as a target example, but no Cox simulation or verification that the Cox partial likelihood satisfies Condition (C4) is provided; a remark or additional example would be helpful.
Circularity Check
No significant circularity: the DLSA asymptotic result is proved from explicit conditions rather than fitted to the target; the Wang-Leng self-citation is not load-bearing.
full rationale
The derivation chain is self-contained. The DLSA estimator in (2.2) is an explicit inverse-covariance weighted average of local estimators, and Theorem 1 derives its limiting distribution in Appendix A.1-A.2 by standard Taylor expansions under Conditions (C1)-(C5), rather than by plugging in the claimed global covariance. Condition (C4) is a genuine information-equality assumption about each local loss (expected outer product of score equals expected Hessian), not a parameter fitted to reproduce the global estimator; if the working likelihood is misspecified the stated covariance can fail, but that is a limitation of the assumptions, not circularity. The citations to Wang and Leng (2007) supply the LSA surrogate idea and a discussion of Condition (C6); because that paper is a published, independently falsifiable methodological basis and the present proofs do not rely on an unproved theorem from it, the overlap of one author does not make the argument circular. The paper also flags its own limitations (lack of second derivatives for quantile regression in Section 6, and post-selection bias from pre-screening in Section 4.2), consistent with an honest conditional derivation. The typographical 'K≫√N' in Appendix A.2 (the argument needs K≪√N) is a proof-reading slip and does not indicate a circular step.
Assumptions & free parameters
free parameters (1)
- Adaptive Lasso tuning parameter lambda0 =
Selected by DBIC
assumptions (5)
- domain assumption Condition (C1): parameter space is compact and convex, true value in interior
- domain assumption Condition (C2): covariates within each worker are i.i.d. from a worker-specific distribution F_k
- domain assumption Condition (C4): information equality for each local loss, E[score score'] = E[Hessian]
- domain assumption Condition (C5): third derivatives of the loss exist and have bounded moments
- domain assumption Condition (C6): covariance submatrix of the global estimator for an overfitted model matches the inverse Hessian submatrix
Cite this review
Pith. "Pith review of Least Squares Approximation for a Distributed System." pith.science (2026). https://pith.science/paper/G7MTELQX
@misc{pith2026190804904,
author = {Pith},
title = {Pith review of: Least Squares Approximation for a Distributed System},
year = {2026},
howpublished = {\url{https://pith.science/paper/G7MTELQX}},
note = {Machine review of arXiv:1908.04904}
}
read the original abstract
In this work, we develop a distributed least squares approximation (DLSA) method that is able to solve a large family of regression problems (e.g., linear regression, logistic regression, and Cox's model) on a distributed system. By approximating the local objective function using a local quadratic form, we are able to obtain a combined estimator by taking a weighted average of local estimators. The resulting estimator is proved to be statistically as efficient as the global estimator. Moreover, it requires only one round of communication. We further conduct a shrinkage estimation based on the DLSA estimation using an adaptive Lasso approach. The solution can be easily obtained by using the LARS algorithm on the master node. It is theoretically shown that the resulting estimator possesses the oracle property and is selection consistent by using a newly designed distributed Bayesian information criterion (DBIC). The finite sample performance and computational efficiency are further illustrated by an extensive numerical study and an airline dataset. The airline dataset is 52 GB in size. The entire methodology has been implemented in Python for a {\it de-facto} standard Spark system. The proposed DLSA algorithm on the Spark system takes 26 minutes to obtain a logistic regression estimator, which is more efficient and memory friendly than conventional methods.
Figures
Reference graph
Works this paper leans on
-
[1]
Apache Software Foundation (2019 a ), Apache Hadoop,
work page 2019
-
[2]
--- (2019 b ), Apache Spark,
work page 2019
-
[3]
Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2015), Distributed estimation and inference with statistical guarantees, arXiv preprint arXiv:1509.05457
arXiv 2015
-
[4]
Cameron, A. C. and Trivedi, P. K. (2013), Regression analysis of count data, vol. 53, Cambridge university press
work page 2013
-
[5]
Chang, X., Lin, S.-B., and Wang, Y. (2017 a ), Divide and conquer local average regression, Electronic Journal of Statistics, 11, 1326--1350
work page 2017
-
[6]
Chang, X., Lin, S.-B., and Zhou, D.-X. (2017 b ), Distributed semi-supervised learning with kernel ridge regression, The Journal of Machine Learning Research, 18, 1493--1514
work page 2017
-
[7]
and Chen, Z
Chen, J. and Chen, Z. (2008), Extended Bayesian information criteria for model selection with large model spaces, Biometrika, 95, 759--771
2008
-
[8]
Chen, T. and Guestrin, C. (2016), Xgboost: A scalable tree boosting system, in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, ACM, pp. 785--794
work page 2016
Show all 41 references
-
[9]
(2018), Quantile regression under memory constraint, arXiv preprint arXiv:1810.08264
Chen, X., Liu, W., and Zhang, Y. (2018), Quantile regression under memory constraint, arXiv preprint arXiv:1810.08264
2018 arXiv
-
[10]
and Xie, M.-g
Chen, X. and Xie, M.-g. (2014), A split-and-conquer approach for analysis of extraordinarily large data, Statistica Sinica, 1655--1684
2014
-
[11]
(2004), Least angle regression, Annals of Statistics, 32, 407--499
Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004), Least angle regression, Annals of Statistics, 32, 407--499
2004
-
[12]
and Li, R
Fan, J. and Li, R. (2001), Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348--1360
2001
-
[13]
--- (2002), Variable selection for Cox's proportional hazards model and frailty model, The Annals of Statistics, 30, 74--99
2002
-
[14]
(2017), Distributed estimation of principal eigenspaces, arXiv preprint arXiv:1702.06488
Fan, J., Wang, D., Wang, K., and Zhu, Z. (2017), Distributed estimation of principal eigenspaces, arXiv preprint arXiv:1702.06488
2017 arXiv
-
[15]
Harrell Jr, F. E. (2015), Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis, Springer
2015
-
[16]
(2016), Dual-loco: Distributing statistical estimation using random projections, in Artificial Intelligence and Statistics, pp
Heinze, C., McWilliams, B., and Meinshausen, N. (2016), Dual-loco: Distributing statistical estimation using random projections, in Artificial Intelligence and Statistics, pp. 875--883
2016
-
[17]
W., Lemeshow, S., and Sturdivant, R
Hosmer Jr, D. W., Lemeshow, S., and Sturdivant, R. X. (2013), Applied logistic regression, vol. 398, John Wiley & Sons
2013
-
[18]
I., Lee, J
Jordan, M. I., Lee, J. D., and Yang, Y. (2018), Communication-efficient distributed statistical inference, Journal of the American Statistical Association, 1--14
2018
-
[19]
D., Sun, Y., Liu, Q., and Taylor, J
Lee, J. D., Sun, Y., Liu, Q., and Taylor, J. E. (2015), Communication-efficient sparse regression: a one-shot approach, arXiv preprint arXiv:1503.04337
2015 arXiv
-
[20]
Lehmann, E. L. and Casella, G. (2006), Theory of point estimation, Springer Science & Business Media
2006
-
[21]
(2019), Distributed feature screening via componentwise debiasing, arXiv preprint arXiv:1903.03810
Li, X., Li, R., Xia, Z., and Xu, C. (2019), Distributed feature screening via componentwise debiasing, arXiv preprint arXiv:1903.03810
2019 arXiv
-
[22]
and Ihler, A
Liu, Q. and Ihler, A. T. (2014), Distributed estimation, information loss and exponential families, in Advances in neural information processing systems, pp. 1098--1106
2014
-
[23]
(2014), Communication-efficient distributed optimization using an approximate newton-type method, in International conference on machine learning, pp
Shamir, O., Srebro, N., and Zhang, T. (2014), Communication-efficient distributed optimization using an approximate newton-type method, in International conference on machine learning, pp. 1000--1008
2014
-
[24]
(1997), An asymptotic theory for linear model selection, Statistica Sinica, 221--242
Shao, J. (1997), An asymptotic theory for linear model selection, Statistica Sinica, 221--242
1997
-
[25]
I., and Jaggi, M
Smith, V., Forte, S., Ma, C., Tak \'a c , M., Jordan, M. I., and Jaggi, M. (2018), CoCoA: A General Framework for Communication-Efficient Distributed Optimization, Journal of Machine Learning Research, 18, 1--49
2018
-
[26]
(1996), Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society
Tibshirani, R. (1996), Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society. Series B, 267--288
1996
-
[27]
and Leng, C
Wang, H. and Leng, C. (2007), Unified LASSO estimation by least squares approximation, Journal of the American Statistical Association, 102, 1039--1048
2007
-
[28]
(2007), Tuning parameter selectors for the smoothly clipped absolute deviation method, Biometrika, 94, 553--568
Wang, H., Li, R., and Tsai, C.-L. (2007), Tuning parameter selectors for the smoothly clipped absolute deviation method, Biometrika, 94, 553--568
2007
-
[29]
(2018), Optimal subsampling for large sample logistic regression, Journal of the American Statistical Association, 113, 829--844
Wang, H., Zhu, R., and Ma, P. (2018), Optimal subsampling for large sample logistic regression, Journal of the American Statistical Association, 113, 829--844
2018
-
[30]
(2017 a ), Efficient distributed learning with sparsity, in Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR
Wang, J., Kolar, M., Srebro, N., and Zhang, T. (2017 a ), Efficient distributed learning with sparsity, in Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp. 3636--3645
2017
-
[31]
(2017 b ), Memory and communication efficient distributed stochastic optimization with minibatch prox, in Proceedings of the 2017 Conference on Learning Theory, eds
Wang, J., Wang, W., and Srebro, N. (2017 b ), Memory and communication efficient distributed stochastic optimization with minibatch prox, in Proceedings of the 2017 Conference on Learning Theory, eds. Kale, S. and Shamir, O., Amsterdam, Netherlands: PMLR, vol. 65 of Proceeding...
2017
-
[32]
(2013), Calibrating non-convex penalized regression in ultra-high dimension, Annals of statistics, 41, 2505
Wang, L., Kim, Y., and Li, R. (2013), Calibrating non-convex penalized regression in ultra-high dimension, Annals of statistics, 41, 2505
2013
-
[33]
W., Saunders, M., and Sun, Y
Yang, J., Mahoney, M. W., Saunders, M., and Sun, Y. (2016), Feature-distributed sparse regression: a screen-and-clean approach, in Advances in Neural Information Processing Systems, pp. 2712--2720
2016
-
[34]
J., Shenker, S., and Stoica, I
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. (2012), Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing, in Proceedings of the 9th USENIX conference on Networked Syst...
2012
-
[35]
(2010), Nearly unbiased variable selection under minimax concave penalty, Annals of Statistics, 38, 894--942
Zhang, C.-H. (2010), Nearly unbiased variable selection under minimax concave penalty, Annals of Statistics, 38, 894--942
2010
-
[36]
Zhang, H. H. and Lu, W. (2007), Adaptive Lasso for Cox's proportional hazards model, Biometrika, 94, 691--703
2007
-
[37]
Zhang, T. (2004), Solving large scale linear prediction problems using stochastic gradient descent algorithms, in Proceedings of the twenty-first international conference on Machine learning, ACM, p. 116
2004
-
[38]
C., and Wainwright, M
Zhang, Y., Duchi, J. C., and Wainwright, M. J. (2013), Communication-efficient algorithms for statistical optimization, The Journal of Machine Learning Research, 14, 3321--3363
2013
-
[39]
(2006), The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418--1429
Zou, H. (2006), The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418--1429
2006
-
[40]
and Li, R
Zou, H. and Li, R. (2008), One-step sparse estimates in nonconcave penalized likelihood models, Annals of statistics, 36, 1509
2008
-
[41]
and Zhang, H
Zou, H. and Zhang, H. H. (2009), On the adaptive elastic-net with a diverging number of parameters, Annals of statistics, 37, 1733
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.