Optimal estimation of the null distribution in large-scale inference

Chao Gao; Subhodh Kotekal

RECORD

Optimal estimation of the null distribution in large-scale inference

Subhodh Kotekal, Chao Gao

Statistics Theory · Submitted 12 Jan 2024

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim It is shown consistent estimation of the location parameter is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, and consistent estimation of the scale parameter is possible in the entire regime $k < \frac{n}{2}$.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,928 input tokens.

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.23 · human participants 0.02 · controlled comparison 0.05 · randomized assignment 0.01 · states quantitative result 0.96 · compares with prior results 0.95 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

The advent of large-scale inference has spurred reexamination of conventional statistical thinking. In a Gaussian model for $n$ many $z$-scores with at most $k < \frac{n}{2}$ nonnulls, Efron suggests estimating the location and scale parameters of the null distribution. Placing no assumptions on the nonnull effects, the statistical task can be viewed as a robust estimation problem. However, the best known robust estimators fail to be consistent in the regime $k \asymp n$ which is especially relevant in large-scale inference. The failure of estimators which are minimax rate-optimal with respect to other formulations of robustness (e.g. Huber's contamination model) might suggest the impossibility of consistent estimation in this regime and, consequently, a major weakness of Efron's suggestion. A sound evaluation of Efron's model thus requires a complete understanding of consistency. We sharply characterize the regime of $k$ for which consistent estimation is possible and further establish the minimax estimation rates. It is shown consistent estimation of the location parameter is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, and consistent estimation of the scale parameter is possible in the entire regime $k < \frac{n}{2}$. Faster rates than those in Huber's contamination model are achievable by exploiting the Gaussian character of the data. The minimax upper bound is obtained by considering estimators based on the empirical characteristic function. The minimax lower bound involves constructing two marginal distributions whose characteristic functions match on a wide interval containing zero. The construction notably differs from those in the literature by sharply capturing a scaling of $n-2k$ in the minimax estimation rate of the location.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	The advent of large-scale inference has spurred reexamination of conventional statistical thinking.	background 1.00	0.04	0.01	asserted 0.99
s1	In a Gaussian model for $n$ many $z$-scores with at most $k < \frac{n}{2}$ nonnulls, Efron suggests estimating the location and scale parameters of the null distribution.	background 0.98	0.04	0.02	hedged possibility 0.60
s2	Placing no assumptions on the nonnull effects, the statistical task can be viewed as a robust estimation problem.	background 0.81	0.16	0.02	asserted 0.81
s3	However, the best known robust estimators fail to be consistent in the regime $k \asymp n$ which is especially relevant in large-scale inference.	background 0.59	0.10	0.04	asserted 1.00
s4	The failure of estimators which are minimax rate-optimal with respect to other formulations of robustness (e.g. Huber's contamination model) might suggest the impossibility of consistent estimation in this regime and, consequently, a major weakness of Efron's suggestion.	background 0.48	0.06	0.03	speculative 0.71
s5	A sound evaluation of Efron's model thus requires a complete understanding of consistency.	gap 0.67	0.07	0.04	asserted 0.95
s6	We sharply characterize the regime of $k$ for which consistent estimation is possible and further establish the minimax estimation rates.	contribution 0.98	0.95	0.42	asserted 1.00
s7	It is shown consistent estimation of the location parameter is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, and consistent estimation of the scale parameter is possible in the entire regime $k < \frac{n}{2}$.	result 1.00	0.93	0.78	asserted 1.00
s8	Faster rates than those in Huber's contamination model are achievable by exploiting the Gaussian character of the data.	result 0.89	0.82	0.07	hedged possibility 0.63
s9	The minimax upper bound is obtained by considering estimators based on the empirical characteristic function.	method 0.96	0.88	0.05	asserted 1.00
s10	The minimax lower bound involves constructing two marginal distributions whose characteristic functions match on a wide interval containing zero.	method 0.93	0.87	0.03	asserted 1.00
s11	The construction notably differs from those in the literature by sharply capturing a scaling of $n-2k$ in the minimax estimation rate of the location.	contribution 0.56	0.90	0.07	asserted 1.00

What kind of paper the classifier reports this is: theoretical result 1.00 · evidence from proof or derivation 1.00.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

The advent of large-scale inference has spurred reexamination of conventional statistical thinking. In a Gaussian model for $n$ many $z$-scores with at most $k < \frac{n}{2}$ nonnulls, Efron suggests estimating the location and scale parameters of the null distribution. Placing no assumptions on the nonnull effects, the statistical task can be viewed as a robust estimation problem. However, the best known robust estimators fail to be consistent in the regime $k \asymp n$ which is especially relevant in large-scale inference. The failure of estimators which are minimax rate-optimal with respect to other formulations of robustness (e.g. Huber's contamination model) might suggest the impossibility of consistent estimation in this regime and, consequently, a major weakness of Efron's suggestion. A sound evaluation of Efron's model thus requires a complete understanding of consistency. We sharply characterize the regime of $k$ for which consistent estimation is possible and further establish the minimax estimation rates. It is shown consistent estimation of the location parameter is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, and consistent estimation of the scale parameter is possible in the entire regime $k < \frac{n}{2}$. Faster rates than those in Huber's contamination model are achievable by exploiting the Gaussian character of the data. The minimax upper bound is obtained by considering estimators based on the empirical characteristic function. The minimax lower bound involves constructing two marginal distributions whose characteristic functions match on a wide interval containing zero. The construction notably differs from those in the literature by sharply capturing a scaling of $n-2k$ in the minimax estimation rate of the location.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 8 sentences without a confident label

Uses the method or tool of 1 paper

For a bandwidth $h > 0$, define $Ĝ_h(t) := 1/2nh ∑_j=1^n 1_{|t - X_j| ≤h}$ and set [equation] To simplify presentation, the variance will be assumed known and equal to one in this section; adaptation to $σ^2$, as well as $k$, has been discussed in [ref] .

Sparsity meets correlation in Gaussian sequence model · Upper bound · 0.80

Compares with 1 paper

The construction in the proof of Theorem [ref] is, at its core, similar to that in [ref] .

Sparsity meets correlation in Gaussian sequence model · Lower bound · 0.80

Cites as background 2 papers

Correlation's effects and methodology to handle it were further studied in [ref] , but a complete theoretical picture has not yet been attained.

Solving the Empirical Bayes Normal Means Problem with Correlated Noise · Large-scale inference · 0.87
Correlation's effects and methodology to handle it were further studied in [ref] , but a complete theoretical picture has not yet been attained.

Large-scale Multiple Testing: Fundamental Limits of False Discovery Rate Control and Compound Oracle · Large-scale inference · 0.87

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.