Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Law of the Iterated Logarithm and Model Selection Consistency for GLMs with Independent and Dependent Responses

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For generalized linear models, the MLE obeys the law of the iterated logarithm, which makes BIC and SCC select the simplest correct model almost surely while AIC does not.

desk verdict Useful upper-bound and model-selection results wrapped in an unproved LIL lower bound; referee it, but expect the sharp claims to be repaired or dropped. read the letter →

arxiv 1908.03676 v3 pith:B74SETMO submitted 2019-08-10 math.ST math.PRstat.TH

Xiaowei Yang , Shuang Song , Huiming Zhang This is my paper · ORCID
classification math.STmath.PRstat.TH MSC 62F1260F1562J1262M10
keywords lawoftheiteratedlogarithmgeneralizedlinearmodelsmodelselectionconsistencyBICAICSCCρ-mixingm-dependent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to prove that maximum likelihood estimation in fixed-dimensional generalized linear models reaches the sharpest almost-sure convergence rate known for such problems: under mild conditions, the Euclidean error $\|\hat\beta-\beta_0\|$ is $O(\sqrt{n^{-1}\log\log n})$ almost surely, and the limsup of the normalized error is a strictly positive constant. That rate is exactly what is needed to settle model selection: with it, any penalized-likelihood criterion whose penalty grows faster than $\log\log n$ but slower than $n$ selects the simplest correct submodel with probability one. The paper shows BIC and the stochastic complexity criterion have precisely this property, while AIC does not. The same conclusions are claimed when the responses are weakly dependent — $\rho$-mixing with geometric decay or $m$-dependent — extending the theory to time-series GLMs. Simulations for negative-binomial, probit, and dependent linear models confirm the BIC/AIC contrast that the theory predicts.

What carries the argument

The argument's engine is the weighted score function evaluated at the truth, $S_n(\beta_0) = \sum_{k=1}^n w_k x_k \dot{u}(x_k^{\scriptscriptstyle T}\beta_0)[y_k - \dot{b}(u(x_k^{\scriptscriptstyle T}\beta_0))]$, a sum of centered random variables. A classical LIL for independent sums — Lemma A.2 in the paper — fixes the almost-sure limsup of each coordinate of $S_n(\beta_0)$ at the scale $\sqrt{n\log\log n}$. The strongly convex log-likelihood (Lemma A.1) then turns this score bound into a quadratic lower bound on the log-likelihood ratio $K_n(\beta,\beta_0)$ on spheres $\partial B_n$ of radius $\tau_n\sqrt{n^{-1}\log\log n}$; because the quadratic term dominates the linear and cross terms on that boundary, the minimizer must lie inside $B_n$, giving the $O(\sqrt{n^{-1}\log\log n})$ error rate. The positive limsup $b>0$ is forced by a subsequence argument: one coordinate of the score reaches its LIL limit along $n_i$, and if the estimator's error were $o(\sqrt{n^{-1}\log\log n})$, the likelihood ratio at a cleverly chosen competitor would have to be both $o(\log\log n)$ and $\le -O(\log\log n)$. For dependent responses the same machinery is run with the score LIL replaced by mixing-sequence LILs (Lemmas H.1 and H.2).

What would settle it

For a probit GLM with $n=10{,}000$, fixed bounded design, and geometric $\rho$-mixing Bernoulli responses, compute the weighted score coordinate $S_n(\beta_0)_j$ and its empirical variance $\sigma_n^2$, and trace the ratio $|S_n|/\sqrt{2\sigma_n^2\log\log\sigma_n^2}$ along the LIL subsequence $n_k = \lfloor e^{k}\rfloor$; if the ratio does not approach 1 almost surely (e.g., it converges to 0 or oscillates with the weight pattern), Theorem 3.5's positive lower bound fails.

Watch

Extended reading notes

Core claim

The central discovery is that the MLE in a GLM satisfies a law of the iterated logarithm. Theorem 3.1 states that under (H.1)–(H.4), for every correct submodel $\alpha$, $\|\hat\beta(\alpha)-\beta_0(\alpha)\| = O(\sqrt{n^{-1}\log\log n})$ almost surely and $\limsup_n \|\hat\beta(\alpha)-\beta_0(\alpha)\|/\sqrt{n^{-1}\log\log n} = b > 0$ almost surely. The proof goes through the weighted score at $\beta_0$, whose coordinates obey a classical LIL for independent sums, and a strong-convexity/local-quadratic argument that converts score bounds into estimator bounds. From this, Theorem 3.2 bounds the maximized log-likelihood gap of any correct model by $O(\log\log n)$ almost surely, and Theorem 3.3 shows any wrong model loses to the true likelihood by a positive multiple of $n$; Theorem 3.4 then concludes that penalties of order strictly between $\log\log n$ and $n$ — in particular BIC and SCC but not AIC — select the smallest correct model almost surely. Theorem 3.5 extends the LIL to $\rho$-mixing (geometric decay) and $m$-dependent responses, and Theorem 3.6 carries the same model-selection conclusion to those settings.

Load-bearing premise

Everything rests on the weighted score sums obeying a law of the iterated logarithm with a strictly positive limsup constant; for dependent responses the proof invokes an LIL that assumes stationarity, while the score sums themselves are not stationary because of the fixed weights and covariates.

Editorial extensions

If this is right

  • Any penalty term whose order lies strictly between $O(\log\log n)$ and $O(n)$ yields strong model-selection consistency; the $\log n$ penalties of BIC and SCC are the standard instances, and AIC's $O(1)$ penalty is not.
  • Under (H.1)–(H.4), every fixed-dimensional correct GLM submodel has MLE error $O(\sqrt{n^{-1}\log\log n})$ almost surely, so the estimators are strongly consistent at the iterated-logarithm rate.
  • With geometric $\rho$-mixing or $m$-dependence, the same LIL and hence the same BIC/SCC consistency holds for time-series GLM responses.
  • Since the conditions cover non-natural links, probit regression and negative-binomial regression fall within the theory, not just canonical-link GLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof structure — strong convexity plus a LIL for the score — should carry over to penalized convex M-estimators outside exponential families, since the GLM-specific inputs are only the moment bound and the bounded-Hessian conditions.
  • The strictly positive constant $b$ is an existence result: the paper neither identifies it nor suggests how to estimate it, so the LIL is best used as a rate guarantee rather than as a basis for confidence sets.
  • A natural stress test is to run the dependent-case simulation with deterministic oscillating weights $w_k = 1 + 0.5\sin(k)$ (allowed by (H.4)) and check whether the normalized score still has limsup 1; this checks whether the stationarity assumption in the cited mixing LIL matters for the conclusion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the law of the iterated logarithm (LIL) for maximum likelihood estimation in fixed-dimensional generalized linear models with a weighted log-likelihood, for independent responses and for weakly dependent (rho-mixing or m-dependent) responses. The main theoretical claims are Theorem 3.1 and Theorem 3.5: for any correct sub-model, the Euclidean distance between the weighted MLE and the true parameter is O(sqrt(n^{-1} log log n)) almost surely, with a positive limsup constant b when normalized by sqrt(n^{-1} log log n). Theorems 3.2 and 3.3 give corresponding bounds on log-likelihood discrepancies for correct and incorrect sub-models, and Theorems 3.4 and 3.6 conclude that BIC and SCC are strongly consistent for model selection while AIC is not. A simulation study in Section 4 compares BIC and AIC for negative-binomial, probit, and dependent linear models.

Significance. If the LIL results were valid, they would provide the sharpest strong-convergence rate for GLM estimators in fixed dimension, and they would give a clean proof of strong consistency for BIC and SCC, including non-canonical links and weakly dependent responses. That would be a genuinely useful contribution. The paper has a sensible high-level strategy: use strong convexity of the negative log-likelihood (Lemma A.1) to localize the estimator, bound the linear and quadratic error terms by score LILs, and then separate correct from incorrect sub-models. The simulation results for BIC versus AIC are also in the expected direction, and the authors honestly state that no real-data analysis is included. However, the proof of the LIL statements has load-bearing gaps in both the independent and dependent cases: the variance normalization in Corollary A.1 is not the correct normalization for general weights, and the dependent-case proof applies a stationary LIL to nonstationary weighted sums. These issues are central, not cosmetic, and they affect the headline theorems.

major comments (3)
  1. [Appendix B, Corollary A.1, Eqs. (20)-(24)] Petrov's LIL, Lemma A.2, requires condition (i) as a limit: lim_{n->infty} n^{-1} Var(S_n) = sigma^2 < infinity. The proof of Corollary A.1 verifies only n^{-1} Var(S_{nj}) <= W n^{-1} I_n(beta_0)(j,j) = O(1), which is not the same as convergence. In addition, Var(S_{nj}) = sum_k w_k^2 dot{u}(x_k^t beta_0)^2 ddot{b}(u(x_k^t beta_0)) x_{kj}^2, whereas I_n(beta_0)(j,j) uses w_k rather than w_k^2. For a constant weight c, the limsup in Eq. (20) is c^{1/2} times the claimed value, so Eq. (20) as stated is false for general weights. Consequently Corollary A.1 is not established, and the positive lower-bound construction in the proof of Theorem 3.1 after Eq. (41) is unsupported.
  2. [Section H, Lemma H.1 and Eq. (12)] Lemma H.1 is a LIL for strictly stationary rho-mixing sequences, but the score components S_n(beta_0)_j = sum_k w_k x_{kj} dot{u}(x_k^t beta_0)[y_k - dot{b}(u(x_k^t beta_0))] have coefficients and means that depend on k through the fixed design x_k. Under (H.1), the linear predictors x_k^t beta_0 cannot be constant, so the summands are non-identically distributed and the process is not strictly stationary. The restriction to strictly stationary sequences in Section 3.4 is therefore incompatible with the model (12) and with (H.1). The same issue applies to the m-dependent case via Lemma H.2. Even if a nonstationary LIL were supplied, the calculation around (69)-(70) establishes only I_n(beta_0)(k,k) = O(n) and gives limsup |S_n|/sqrt(2n log log n) = O(1), which cannot yield the positive lower bound b > 0 claimed in Theorem 3.5.
  3. [Propagation to Theorems 3.2-3.4 and 3.6] The model-selection claims inherit the same gaps. For example, Eq. (57) in the proof of Theorem 3.2 uses the score bound from Corollary A.1, and Eq. (58) uses Corollary A.2, both of which rest on the unproved normalization in Corollary A.1. In the dependent case, Section H is the only score-LIL argument, and it is not valid for the nonstationary weighted sums. Thus Theorems 3.4 and 3.6 are not established as written. This is not to say the upper-bound rate or BIC/SCC consistency are false; they may be salvageable with additional assumptions such as two-sided bounded weights and an appropriate nonstationary LIL, but the present manuscript does not provide such a proof.
minor comments (5)
  1. [Section 3.2, condition (H.1)] The statement of (H.1) says 'Fisher information of beta given by (6)', but Eq. (6) defines S_n(alpha); the Fisher information I_n(beta) is defined just before Eq. (6). This cross-reference should be corrected.
  2. [Eq. (30) and surrounding text] The notation ||x_k^t||_2^1 is garbled; it should likely be ||x_k||_1 or a clearly defined norm. The same notation recurs in later bounds and should be fixed for readability.
  3. [Lemma H.2] The displayed statement says limsup |S_n|/sqrt(2n log log n) = sigma^2, but under the displayed definition sigma^2 is a variance; the right-hand side should be sigma, not sigma^2, unless the definition is changed.
  4. [Table 1] There is a typo 'Simualtion' in the table caption, and the AIC rows for the dependent linear model at n=300 are identical across MR(2) and MR(3), which suggests a copy-paste error that should be corrected.
  5. [References] The citation 'Nelder (1972)' should be 'Nelder and Wedderburn (1972)', since the cited work is the joint paper introducing generalized linear models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proof chain relies on external LIL and convexity results, not on the target conclusions; the dependent-case stationarity gap is a correctness concern, not a circular derivation.

full rationale

The paper's central claims are the LIL rates for GLM MLEs and the resulting strong consistency of BIC/SCC. These are derived from external strong-limit theorems: Petrov's LIL (Lemma A.2, attributed to Stout) for independent sums, the Lin-Lu LIL for strictly stationary rho-mixing sequences (Lemma H.1), and Chen's m-dependent LIL (Lemma H.2), together with a convexity inequality (Lemma A.1 from Wright). No parameter is fitted to data, and no target result is assumed in the derivation. The only self-citations are Zhang and Jia (2017), cited for negative-binomial regression context, and Ai et al. (2020), mentioned in the Discussion as a possible application; neither is load-bearing for Theorems 3.1, 3.4, 3.5, or 3.6. The lower-bound proof for the independent case uses Corollary A.1 to find a subsequence where a score component is large; that Corollary is itself proved from Petrov's LIL, not from the theorem being established. The model-selection consistency argument (Theorems 3.4, 3.6) follows the standard penalty-order comparison and does not reuse the LIL conclusion as an assumption. The identified weaknesses are of a different kind: for the dependent case, Lemma H.1 requires strict stationarity, while the weighted score summands have deterministic, time-varying coefficients w_k x_kj u'(x_k' beta0) and response means depending on x_k, so the lemma may not apply; and in the independent case, condition (i) of Lemma A.2 is checked only as an O(1) upper bound in (24), not as a stated limit. These are potential gaps in verifying the hypotheses of cited external theorems, not circular dependencies in which an input is defined as the output or a fitted quantity is renamed as a prediction. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central derivation rests on six background axioms: exponential family regularity, the standing conditions H.1-H.6, and three external LILs plus a convexity lemma. There are no fitted parameters and no invented entities.

assumptions (6)
  • domain assumption The exponential family density (1) has b twice differentiable on the interior of the natural parameter space, with first and second derivatives bounded on compact sets (Remark 1, Eq. (8)).
    Used to control moments of e_k and to apply Taylor expansions in the proof of Theorem 3.1 (Appendix D).
  • domain assumption Regularity conditions H.1-H.4: Fisher information eigenvalues grow linearly; design bounded; derivatives of u and b uniformly bounded; weights uniformly bounded and a restricted eigenvalue condition holds.
    These are the standing assumptions stated in Section 3.2 and used throughout the proofs.
  • domain assumption For the dependent case, the response sequence y_i is strictly stationary and either ρ-mixing with geometric decay (H.5) or m-dependent (H.6).
    Used to apply mixing LILs in Section H; strict stationarity is required by Lemma H.1.
  • standard math Petrov's LIL (Lemma A.2) gives limsup S_n / sqrt(2 n log log n) = sigma for independent centered variables with bounded (2+η) moments and variance/n -> sigma^2.
    External theorem (Stout 1974) used to control the score components; not proved in the paper.
  • standard math Lemma H.1 (Lin and Lu 1997) gives a LIL for strictly stationary ρ-mixing sequences, and Lemma H.2 (Chen 1997) gives a LIL for m-dependent stationary sequences.
    External theorems used in the dependent-case proof; their applicability to the weighted score is not established in the paper.
  • standard math Lemma A.1 (Wright 2017) provides quadratic upper and lower bounds for a strongly convex, Lipschitz-gradient function.
    Used to lower-bound the likelihood ratio K_n(β,β0) on a shrinking ball and to upper-bound it near the MLE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Law of the Iterated Logarithm and Model Selection Consistency for GLMs with Independent and Dependent Responses." pith.science (2026). https://pith.science/paper/B74SETMO

@misc{pith2026190803676,
  author       = {Pith},
  title        = {Pith review of: Law of the Iterated Logarithm and Model Selection Consistency for GLMs with Independent and Dependent Responses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B74SETMO}},
  note         = {Machine review of arXiv:1908.03676}
}
abstract

We study the law of the iterated logarithm (LIL) for the maximum likelihood estimation of the parameters (as a convex optimization problem) in the generalized linear models with independent or weakly dependent ($\rho$-mixing, $m$-dependent) responses under mild conditions. The LIL is useful to derive the asymptotic bounds for the discrepancy between the empirical process of the log-likelihood function and the true log-likelihood. As the application of the LIL, the strong consistency of some penalized likelihood based model selection criteria can be shown. Under some regularity conditions, the model selection criterion will be helpful to select the simplest correct model almost surely when the penalty term increases with model dimension and the penalty term has an order higher than $O({\rm{loglog}}n)$ but lower than $O(n)$. Simulation studies are implemented to verify the selection consistency of BIC.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages

  1. [1]

    Ai, M., Yu, J., Zhang, H., & Wang, H. (2020). Optimal Subsampling Algorithms for Big Data Regressions. Statistica Sinica. DOI: 10.5705/ss.202018.0439

    work page arXiv 2020

  2. [2]

    Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In 2nd ed International Symposium on Information Theory, 1973. Akademiai Kaido

    work page 1973

  3. [3]

    Bosq, D. (1998). Nonparametric statistics for stochastic processes: estimation and prediction. Springer

    work page 1998

  4. [4]

    Brown, L. D. (1986). Fundamentals of statistical exponential families: with applications in statistical decision theory. IMS

    work page 1986

  5. [5]

    Chen, X. (1997). The law of the iterated logarithm for m-dependent Banach space valued random variables. Journal of Theoretical Probability, 10(3), 695-732

    work page 1997

  6. [6]

    Chen, X. (2011). Quasi Likelihood Method for Generalized Linear Model (in Chinese). Press of University of Science and Technology of China

    work page 2011

  7. [7]

    Czado, C., & Munk, A. (2000). Noncanonical links in generalized linear models when is the effort justified?. Journal of Statistical Planning and Inference, 87(2), 317-345

    work page 2000

  8. [8]

    Efron, B., & Hastie, T. (2016). Computer age statistical inference: algorithms, evidence, and data science. Cambridge University Press

    work page 2016

Show all 38 references
  1. [9]

    Fahrmeir, L., & Kaufmann, H. (1985). Consistency and asymptotic normality of the maximum likelihood estimator in generalized linear models. The Annals of Statistics, 342-368

    1985

  2. [10]

    Fahrmeir, L., & Tutz, G. (2001). Multivariate statistical modelling based on generalized linear models, 2ed. Springer

    2001

  3. [11]

    Fan, J., Qi, L., & Tong, X. (2016). Penalized least squares estimation with weakly dependent data. Science China Mathematics, 59(12), 2335-2354

    2016

  4. [12]

    Fang, X. (1998). Laws of the iterated logarithm for maximum likelihood estimates of parameter vectors in nonhomogeneous Poisson processes (Chinese). Acta Scientiarum Naturalium Universitatis Pekinensis, 1998, 34(5), 563-573. MR1681923

    1998

  5. [13]

    Hansen, B. (2018). Econometrics. Version: Jan 2018\\ https://www.ssc.wisc.edu/ bhansen/econometrics

    2018

  6. [14]

    He, X., & Wang, G. (1995). Law of the iterated logarithm and invariance principle for M-estimators. Proceedings of the American Mathematical Society, 123(2), 563-573

    1995

  7. [15]

    Kim, Y., & Jeon, J. J. (2016). Consistent model selection criteria for quadratically supported risks. The Annals of Statistics, 44(6), 2467-2496

    2016

  8. [16]

    Kroll, M. (2019). Non-parametric Poisson regression from independent and weakly dependent observations by model selection. Journal of Statistical Planning and Inference, 199, 249-270

    2019

  9. [17]

    L., & Wei, C

    Lai, T. L., & Wei, C. Z. (1982). A law of the iterated logarithm for double arrays of independent random variables with applications to regression and time series models. The Annals of Probability, 320-335

    1982

  10. [18]

    Lin, Z., & Lu, C. (1997). Limit theory for mixing dependent random variables. Springer

    1997

  11. [19]

    Markatou, M., Basu, A., & Lindsay, B. G. (1998). Weighted likelihood equations with bootstrap root search. Journal of the American Statistical Association, 93(442), 740-750

    1998

  12. [20]

    & Nelder, J

    McCullagh, P. & Nelder, J. A. (1989). Generalized linear models. Second Edition. Chapman and Hall, London

    1989

  13. [21]

    Miao, Y., & Yang, G. (2011). The loglog law for LS estimator in simple linear EV regression models. Statistics, 45(2), 155-162

    2011

  14. [22]

    Nelder, J., & Wedderburn, R. (1972). Generalized Linear Models. Journal of the Royal Statistical Society. Series A (General), 135(3), 370-384

    1972

  15. [23]

    Petrov, V. V. (1995). Limit theorems of probability theory: sequences of independent random variables. Oxford, New York

    1995

  16. [24]

    Qian, G., & Wu, Y. (2006). Strong limit theorems on model selection in generalized linear regression with binomial responses. Statistica Sinica, 1335-1365

    2006

  17. [25]

    Rao, R., & Wu, Y. (1989). A strongly consistent procedure for model selection in a regression problem. Biometrika, 76(2), 369-374

    1989

  18. [26]

    R., & Zhao, L

    Rao, C. R., & Zhao, L. C. (1992). Linear representation of M-estimates in linear models. Canadian Journal of Statistics, 20(4), 359-368

    1992

  19. [27]

    Rigollet, P. (2012). Kullback-Leibler aggregation and misspecified generalized linear models. The Annals of Statistics, 40(2), 639-665

    2012

  20. [28]

    Rissanen, J. (1989). Stochastic complexity in statistical inquiry. World Scientific

    1989

  21. [29]

    Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464

    1978

  22. [30]

    Stout, W. F. (1974). Almost sure convergence, Academic Press, NewYork

    1974

  23. [31]

    Shao, J. (2003). Mathematical Statistics 2ed. Springer New York

    2003

  24. [32]

    Tutz, G. (2011). Regression for categorical data (Vol. 34). Cambridge University Press

    2011

  25. [33]

    van der Vaart, A. W. (1998). Asymptotic statistics (Vol. 3). Cambridge university press

    1998

  26. [34]

    Wright, S. J. (2017). Optimization algorithms for data analysis. IAS/Park City Mathematics Series, to appear

    2017

  27. [35]

    Wu, Y., & Zen, M. M. (1999). A strongly consistent information criterion for linear model selection based on M-estimation. Probability Theory and Related Fields, 113(4), 599-625

    1999

  28. [36]

    Yin, C., Zhao, L., & Wei, C. (2006). Asymptotic normality and strong consistency of maximum quasi-likelihood estimates in generalized linear models. Science in China Series A, 49(2), 145-157

    2006

  29. [37]

    Zhang, H., & Jia, J. (2017). Elastic-net regularized high-dimensional negative binomial regression: consistency and weak signals detection. arXiv preprint arXiv:1712.03412

    2017 arXiv

  30. [38]

    Zheng, Q., & Peng, L. (2017). Consistent model identification of varying coefficient quantile regression with BIC tuning parameter selection. Communications in Statistics-Theory and Methods, 46(3), 1031-1049

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.