{"id":"50c73be6-a336-4846-bbf7-6ef317b7c179","arxiv_id":"1908.06600","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This chapter surveys asymptotic, projection-based, and discrete-model approaches for high-dimensional mean testing, covariance inference, and count data analysis.","lead":"This paper is a handbook chapter that reviews methods for high-dimensional statistical inference, including tests for mean vectors, covariance estimation, and multivariate count data. A generalist reader might consult it for a broad map of tools used in genomics and other big-data fields where the number of variables far exceeds the number of samples.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The chapter's own Section 2.5 flags published proof errors in the T_APR test but never states the corrected assumptions or estimator, so the 'comprehensive overview' claim is unsupported at exactly the dependent-observation point.","rationale":"I read the chapter as a survey intended to let a statistician learn the main tests and estimators for high-dimensional means, covariances, and discrete models. The three-part structure is coherent, the literature coverage is broad, and the remarks on projection, scale invariance, and sparse multinomials are useful. I do not see a mathematical error in the core derivations I checked, and this is not a case of disagreeing with consensus: the issue is the chapter's own admission. Section 2.5 is the only place dependent observations are treated, and the central test presented there is the T_APR statistic from Ayyala et al. [6]. The chapter explicitly states that Cho et al. [23] found errors in the proofs and corrected 'some results and assumptions,' but it does not integrate those corrections. If the corrections changed any of (APR I)-(APR IV) or the leave-out variance estimator, then a reader who follows the chapter will be taught an invalid test. If the corrections only repaired proof details, the chapter should still say so; the current one-sentence note leaves the status ambiguous. This is load-bearing because the central claim is comprehensiveness, not novelty; a survey that cannot be relied upon at exactly the point where it flags its own basis is not yet a reliable reference. The placeholders and typos are symptoms of the same incompleteness. My recommended verdict is unchanged: conditional acceptance pending integration of the correction and completion of the missing text.","tokens_in":35844,"tokens_out":3545,"duration_ms":32871,"concrete_test":"Obtain Cho et al. (arXiv:1904.09344) and compare its corrected theorem statements with Section 2.5. In particular, check whether (APR II) M=O(n^{1/8}) and (APR III) tr{Σk1(a)Σk2(b)Σk3(c)Σk3(d)} = o{(M+1)^{-4} tr^2(Ω1+Ω2)^2} are altered, and check whether the variance estimator in equation (33) must be replaced. If any displayed assumption or estimator changes, Section 2.5 is incorrect as written and the overview needs revision; if only proof details change, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The chapter's central claim is that it provides a comprehensive overview of high-dimensional inference. For a review, accuracy and completeness are the product. The most load-bearing gap is in Section 2.5: after presenting the T_APR statistic (equation 33) and assumptions (APR I)-(APR IV), the text says 'While the test statistic and the empirical studies of Ayyala et al. are valid, Cho et al. [23] identified some theoretical errors in the proofs and provided some corrections to some results and assumptions in Ayyala et al.' The chapter never says which displayed assumptions were corrected, whether (APR II)'s M=O(n^{1/8}) or (APR III)'s trace condition survive, or whether the leave-out variance estimator referenced only to Ayyala et al. [6] is the corrected one. A reader cannot learn the actual T_APR test from this 'comprehensive' chapter. The nearby placeholder '(write motivation)' and the missing 'John [ ? ]' citation reinforce that the manuscript is unfinished, but the unintegrated correction is the substantive reason the central claim is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a survey chapter, intended for the Handbook of Statistics, that promises a comprehensive overview of high-dimensional statistical inference with an emphasis on data-analytic applicability. It covers tests for the mean vector under independence (Bai-Saranadasa, Chen-Qin, Srivastava-Du, Park-Ayyala), projection-based and random-projection approaches, and a section on dependent observations; estimation and tests for covariance matrices; and discrete multivariate models (multinomial, Dirichlet-multinomial, Bernoulli, binomial, Poisson). The chapter also discusses open problems and computational considerations.","tokens_in":35988,"tokens_out":14235,"duration_ms":137633,"significance":"If completed and corrected, the chapter would fill a useful niche: it collects the main asymptotic mean tests, compares their invariance and assumptions, and connects high-dimensional theory to genomics and text-mining applications. The comparison of the invariance properties of TBS, TCQ, TSD, and TPA, as well as the discussion of the Dirichlet-multinomial model, are informative and would be valuable pedagogical material. However, the current text contains a literal placeholder, an unresolved citation, and, more importantly, an unintegrated correction to a presented test, so the promised 'comprehensive overview' is not yet delivered. The significance of the review will depend on these issues being fixed.","major_comments":[{"comment":"The claim that for k < p and a full-row-rank matrix R, R(µ1−µ2)=0 iff µ1−µ2=0 is false, because R has a (p−k)-dimensional null space. Consequently, the projected hypothesis (14) is not equivalent to the original hypothesis (1): alternatives with µ1−µ2 in the null space of R are invisible to any test based on the projected data. The power simulation in Figure 1 illustrates the loss for small k but does not repair the equivalence statement, which is load-bearing for the motivation of all projection-based tests in Sections 2.2 and 2.3.","section":"Section 2.2, Eq. (14)"},{"comment":"The description of RAPTT contains an inconsistent rejection rule. The text defines ψα by P(p>ψα|H0)=1−α and then says to reject if p>ψα, while the algorithm estimates ψα as p[M(1−α)], the (1−α)-quantile of the simulated null distribution of p. Under the stated alternatives, the individual projected Hotelling p-values are stochastically smaller than uniform, so the average p is small; a test that rejects for large average p-values has power tending to zero under those alternatives. The rejection region should be the lower tail (reject if p < ψα, with ψα the α-quantile), and the two definitions of ψα should be made consistent.","section":"Section 2.3, Eqs. (19)-(21)"},{"comment":"The chapter presents TAPR and assumptions (APR I)-(APR IV), then states that Cho et al. [23] identified theoretical errors in Ayyala et al. and provided corrections, but it does not say which assumptions were corrected, whether (APR II)'s M=O(n^{1/8}) and (APR III) survive, or whether the leave-out variance estimator referenced only to [6] is the corrected one. Since the chapter also refers the reader elsewhere for the estimator's exact form, a reader cannot actually learn the TAPR test from this 'comprehensive' chapter. The section opens with the placeholder '(write motivation)', which further confirms that this part of the manuscript is unfinished.","section":"Section 2.5, Eq. (33) and following paragraph"}],"minor_comments":[{"comment":"In the leave-out definitions, Y(i) and Y(i,j) are defined with denominators n−1 and n−2; these should be m−1 and m−2 to match the sample size of the Y group.","section":"Section 2.1, below Eq. (8)"},{"comment":"The expression 'Bn = 2n^{1−α}λmax' appears to have the exponent reversed; for p=Cn^α, the bias is of order n^{α−1}, which is what makes the subsequent statement 'diverges when α>1' correct.","section":"Section 2.1, paragraph after Eq. (5)"},{"comment":"The second indicator inside the double sum contains an undefined index 'i'; it should presumably be 'a' or another explicitly defined index, and the expression should be checked.","section":"Section 2.5, Eq. (31)"},{"comment":"The sentence saying the banded estimator is 'clearly indicating Σ̂(k) as the diagonal estimator' is imprecise; the diagonal estimator is the special case k=1, while for k>1 the estimator is banded.","section":"Section 3.1, below Eq. (35)"},{"comment":"The citation 'John [ ? ]' is unresolved; the reference must be supplied before publication.","section":"Section 3.2, Eq. (42)"},{"comment":"The sentence 'While the density function is known to be globally convex, maximization can still lead us to a local maxima' is contradictory as written; if the relevant function is convex, interior local maxima are not a concern, and if the intended statement concerns the log-likelihood, concavity is the usual property.","section":"Section 4.2"},{"comment":"The displayed mass function is not readable as printed; the notation with products and subscripts should be rewritten with explicit joint probabilities.","section":"Section 4.3.1, Eq. (58)"},{"comment":"There are numerous typos (e.g., 'projeting', 'repsectively', 'distributiosn', 'likeliho0d', 'conjuction', 'moreknown than unkown') and at least one grammatical break ('R1 and R2 are the for notational convenience'); a careful copyedit is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is a draft of a book chapter and contains several placeholders; this is not unusual for a preprint, but the published version must resolve them. I also note that the author's own methods (Park-Ayyala and Ayyala et al.) receive more detailed presentation than several external methods of comparable importance; the author should be asked to balance the coverage in the revision. These concerns do not drive my recommendation, which is based on the technical errors in Sections 2.2, 2.3, and 2.5."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"It's a review chapter, not a research contribution, and the honest way to read it is as a handbook entry: organize the literature, state the assumptions, point readers to the sources. On that front it mostly works. The mean-vector section is readable and does a fair job laying out the Bai-Saranadasa, Chen-Qin, Srivastava-Du, and Park-Ayyala tests, including the differences between orthogonal and scale invariance and the practical meaning of the asymptotic rate conditions. The random-projection material, including RAPTT, is a helpful summary. The discrete multivariate section is actually the most useful part: a compact tour of multinomial testing, Dirichlet-multinomial models, LDA, and the sparse literature on multivariate Bernoulli/binomial/Poisson, with open problems stated clearly. A reader with no prior exposure would leave with a decent map of the area.\n\nNow the soft spots. The manuscript is visibly unfinished: Section 2.5 literally contains '(write motivation)', Section 3.2 has the missing 'John [ ? ]' citation, and there are typos throughout. Those are minor in themselves, but they undercut the 'comprehensive overview' claim.\n\nThe substantive problem is in Section 2.5. The chapter presents the T_APR statistic and assumptions (APR I)-(APR IV), then says Cho et al. identified theoretical errors in the proofs and provided corrections to some results and assumptions. It never says which assumptions are wrong, what the corrected versions are, or whether the variance estimator described only by reference to Ayyala et al. is the corrected one. For a chapter whose value is supposed to be accuracy and completeness, that is a real hole: a reader cannot learn the actual T_APR test from this chapter. This is not a manufactured flaw; it is exactly where the dependent-observations story should be most careful.\n\nOne more thing, mild: the author's own tests (Park-Ayyala and Ayyala et al.) get noticeably more space and detail than some external methods. That is a bias common in surveys, and not disqualifying, but it is worth noting.\n\nWho is this for? Practitioners and graduate students who want a first pass over high-dimensional inference methods, plus a bibliography to follow up. For that audience it will be useful once the gaps are fixed. As it stands, the chapter deserves peer review because a good revision would be a genuinely useful reference, but I would not accept it in its current form. The revision should fill the placeholder, fix the citations and typos, and, most importantly, integrate the Cho et al. correction into Section 2.5 explicitly.","headline":"A useful but unfinished survey chapter: solid organization and coverage of high-dimensional mean/covariance testing, with a real hole where the author's own dependent-data test is presented but its published correction is never integrated.","tokens_in":36564,"tokens_out":1422,"would_cite":false,"duration_ms":18363,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62-02","62H15","62H12","62H10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey of high-dimensional inference claims the field is best understood through bias-corrected quadratic forms, random projections, and sparse regularization, and it organizes the main tests and estimators around those mechanisms.","keywords":["high-dimensional inference","mean vector testing","random projections","covariance matrix estimation","graphical lasso","Dirichlet-multinomial","multivariate count models","asymptotic normality"],"falsifier":"A concrete check: simulate two samples from an exchangeable covariance model where all pairwise correlations equal a fixed $\\rho>0$, run $T_{BS}$ and $T_{CQ}$, and compare empirical rejection rates to the nominal level; the chapter predicts inflated type I error because the trace-ratio condition fails, so a simulation that still controls size would refute that specific claim about the strength of covariance assumptions.","tokens_in":35566,"feed_emoji":"📊","tokens_out":9614,"duration_ms":90268,"temperature":0.7,"pith_summary":"This chapter is a survey of high-dimensional statistical inference, and its central claim is that the field can be organized around a few recurring mechanisms: bias-corrected quadratic forms of the mean difference, random projections, and sparse regularization of covariance structure. This matters for genomics, metagenomics, and text mining, where the number of variables routinely exceeds the sample size and classical tools such as Hotelling's $T^2$ are undefined. The chapter compares the four standard asymptotic mean tests, the random-projection alternative, and the dependent-observation extension, and it identifies open gaps such as covariance testing by projections and feasible inference for multivariate Poisson models. The manuscript itself contains unfinished spots--a placeholder at the start of Section 2.5 and a missing citation in Section 3.2--that qualify the literal promise of completeness even though the coverage is systematic.","feed_headline":"Bias-corrected norms and random projections organize high-dim tests","feed_subtitle":"A handbook chapter groups mean-vector tests, covariance estimators, and count models into a few reusable mechanisms.","key_machinery":"The load-bearing objects are: (1) unbiased quadratic-form functionals such as $M_n = (\\bar{X}-\\bar{Y})^\\top(\\bar{X}-\\bar{Y}) - \\frac{n+m}{nm}\\mathrm{tr}(S)$, whose expectation is the squared mean difference; (2) leave-out ratio-consistent variance estimators built from expressions like $\\hat{\\mathrm{tr}}(\\Sigma_1^2)$; (3) random projection matrices derived from the Johnson-Lindenstrauss lemma; (4) $\\ell_1$ penalties on the precision matrix, notably the graphical lasso; and (5) the Dirichlet-multinomial hierarchy for overdispersed count data. These carry the argument because each method is defined by which functional it uses and by which variance or penalty estimator it pairs with that functional.","core_discovery":"On its own terms, the chapter establishes that high-dimensional inference now has a coherent toolkit: for mean vectors, replace the singular sample covariance inverse with bias-corrected norms of the mean difference and ratio-consistent variance estimators; for covariance matrices, impose structure through banding or $\\ell_1$-regularized precision estimation; for counts, work mainly with multinomial and Dirichlet-multinomial models, where high-dimensional testing succeeds only under sparsity conditions. The four asymptotic mean tests $T_{BS}$, $T_{CQ}$, $T_{SD}$, and $T_{PA}$ are presented as successive relaxations of assumptions on distributions, covariance strength, and the relationship between dimension and sample size, and random-projection tests are presented as an exact alternative for small samples. The chapter also notes that the proofs for the dependent-observation test $T_{APR}$ required later corrections, a caveat it states in Section 2.5.","pith_inferences":["The invariance distinction implies a testable practical rule: when variables are measured on very different scales, scale-invariant tests should detect uniform standardized shifts more often than orthogonal-invariant tests, which a simulation study could verify directly.","Because asymptotic growth-rate conditions like $p/n\\to\\delta$ cannot be confirmed from any single finite dataset, the chapter's own comparison suggests that simulation-based calibration of test choice is more defensible in practice than relying on the stated rates.","Extending random projections to covariance-matrix hypotheses, which the chapter identifies as open, should work for sphericity and identity tests because those hypotheses are invariant under orthogonal projection of the data.","For latent Dirichlet allocation, the chapter notes that statistical properties of estimators are not established, so developing hypothesis tests for comparing topic parameters across corpora is an open problem it implicitly highlights."],"forward_implications":["A practitioner comparing two high-dimensional samples can choose among $T_{BS}$, $T_{CQ}$, $T_{SD}$, and $T_{PA}$ based on whether the signal is in raw or standardized units, because the first two are orthogonal-invariant and the last two are scale-invariant.","Random-projection tests such as RAPTT give exact rather than asymptotic p-values when $n+m$ is small, but they require multiple projections and bootstrap calibration, making them computationally heavier than the asymptotic tests.","Sparse precision-matrix estimation through the graphical lasso turns covariance estimation into network discovery, but the theory is developed mainly under Gaussian assumptions.","High-dimensional testing of multinomial parameters is feasible only when the probability vectors are not too concentrated and the dimension grows at most linearly with sample size.","Multivariate Bernoulli, binomial, and Poisson models remain largely without feasible parameter inference, so count data from text mining and genomics cannot yet be handled with the same maturity as continuous data."],"supporting_citations":[{"why":"Supplies the $T_{BS}$ test, the first high-dimensional mean test built on a bias-corrected Euclidean norm with normality relaxed.","marker":"[7]"},{"why":"Supplies the $T_{CQ}$ test and the leave-out ratio-consistent variance estimators that later tests adapt.","marker":"[22]"},{"why":"Supplies the $T_{SD}$ test using a diagonal-weighted norm under normality and restrictive correlation conditions.","marker":"[69]"},{"why":"Supplies the $T_{PA}$ test, the scale-invariant leave-out version used as the current benchmark.","marker":"[64]"},{"why":"Supplies the $T_{APR}$ test for M-dependent observations, whose proof the chapter notes was later corrected.","marker":"[6]"},{"why":"Supplies RAPTT, the exact p-value combining method for random-projection tests.","marker":"[78]"},{"why":"Supplies the graphical lasso algorithm for sparse precision-matrix estimation.","marker":"[33]"},{"why":"Supplies latent Dirichlet allocation, the main application of the Dirichlet-multinomial model.","marker":"[15]"},{"why":"Introduces binary-coin random projections used for sparse embeddings in projection-based tests.","marker":"[1]"},{"why":"Provides the Johnson-Lindenstrauss lemma, the distance-preserving embedding guarantee behind random projection tests.","marker":"[46]"}],"fun_headline_variants":["High-dim inference: bias-corrected norms and random projections","A coherent toolkit for high-dimensional statistical inference","Random projections and bias-corrected norms refine high-dim tests","High-dimensional inference in data analytics: a practical guide","Unifying high-dimensional inference for big data challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The chapter's value as a comprehensive reference rests on the correctness of its condensed descriptions of each test, and the manuscript itself flags incomplete parts--a placeholder in Section 2.5, a missing citation in Section 3.2, and known proof corrections to the dependent-observation test--so that promise is not yet fully delivered.","fun_headline_variants_meta":{"raw":{"variants":["High-dim inference: bias-corrected norms and random projections","A coherent toolkit for high-dimensional statistical inference","Random projections and bias-corrected norms refine high-dim tests","High-dimensional inference in data analytics: a practical guide","Unifying high-dimensional inference for big data challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2586,"prompt_tokens":898,"completion_tokens":1688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":1611}},"tokens_in":514,"tokens_out":1688,"duration_ms":13041,"temperature":1.0,"reasoning_tokens":1611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:38:58.996999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: simulate two samples from an exchangeable covariance model where all pairwise correlations equal a fixed $\\rho>0$, run $T_{BS}$ and $T_{CQ}$, and compare empirical rejection rates to the nominal level; the chapter predicts inflated type I error because the trace-ratio condition fails, so a simulation that still controls size would refute that specific claim about the strength of covariance assumptions.","supporting_citations":[{"cited_title":"S., Srivastava and M","cited_arxiv_id":null,"evidence_quote":"Supplies the $T_{SD}$ test using a diagonal-weighted norm under normality and restrictive correlation conditions."},{"cited_title":"Park and D","cited_arxiv_id":null,"evidence_quote":"Supplies the $T_{PA}$ test, the scale-invariant leave-out version used as the current benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the $T_{APR}$ test for M-dependent observations, whose proof the chapter notes was later corrected."}],"review_version":1}