Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

This paper claims that a bootstrap-based selective inference procedure can control the probability of k or more false discoveries when testing thousands of concept-level hypotheses on only a few hundred observations, even if the test statis

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:15 UTC pith:MUDJY7K5

load-bearing objection Real new theory and a useful framework, but the applications overclaim: Assumption R is violated in both empirical sections, so the stated k-FWER guarantee doesn't cover the reported discoveries. the 4 major comments →

arxiv 2511.01680 v4 pith:MUDJY7K5 submitted 2025-11-03 econ.EM cs.LG

Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach

Jacob Carlson This is my paper
classification econ.EM cs.LG MSC 62H1562F4062G20
keywords high-dimensional multiple hypothesis testingk-FWERsparse autoencodersinterpretable discoveryunstructured text dataGaussian multiplier bootstrapautomated interpretation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that researchers can make statistically principled 'discoveries' from unstructured text by turning every document into a high-dimensional binary vector of human-interpretable concepts, testing each concept separately, and then using a bootstrap-based selective inference procedure that keeps the probability of k or more false rejections under control even when the number of concepts p is far larger than the sample size n. To make that work, it proves high-dimensional central limit theorems for the k-th largest coordinate of studentized sums, extending existing bootstrap theory to arbitrary dependence among test statistics. If the theorems hold, a researcher can automatically test thousands or tens of thousands of concept-level null hypotheses, extract the winners, describe them in natural language with an LLM, and score the quality of those descriptions, all without hand-labeling data or pre-specifying concepts. The payoff would be a pipeline that replaces cherry-picked qualitative coding with an error-controlled, reproducible screening procedure.

Core claim

The paper's central claim is that a Gaussian multiplier bootstrap version of step-down hypothesis testing controls the k-familywise error rate asymptotically in high dimensions: under sub-exponential tail assumptions and the rate condition B_n^2 log^5(p_n) = o(n), the probability of making k or more false rejections is at most alpha in the large-sample limit, and the procedure is consistent against alternatives of size larger than B_n sqrt(log p / n). This is stated for one-sided and two-sided tests, for studentized statistics, and for approximate means with small estimation error. The paper also formalizes automatic interpretation evaluation: the accuracy, precision, and recall of an LLM-wr

What carries the argument

The engine is a dictionary function that maps each unstructured data point to a sparse binary vector of monosemantic features via a pretrained sparse autoencoder, so each coordinate is one interpretable concept hypothesis. Inference then runs on the k-max of the bootstrap distribution: the critical value is the 1-alpha quantile of the k-th largest absolute coordinate of a Gaussian multiplier bootstrap draw, which lets the procedure inherit the logic of step-down multiple testing algorithms. Two assumptions carry the asymptotics: sub-exponential tails and bounded moments (Assumption M), and the rate condition B_n^2 log^5(p) = o(n) (Assumption R), which make high-dimensional central limit theo

Load-bearing premise

The load-bearing premise is the rate condition B_n^2 log^5(p_n) = o(n) together with k held fixed, meaning the sub-exponential tail parameter must grow slowly enough and the number of tolerated false discoveries must stay small; no finite-sample evidence is given that this asymptotic approximation holds at the paper's application sizes.

What would settle it

Monte Carlo null simulations: generate i.i.d. sparse binary dictionaries with realistic cross-feature correlation at n=930 and p=12,000, compute the multiplier bootstrap 5-FWER critical value at alpha=0.05, and count replications with five or more false rejections. If the empirical k-FWER exceeds 0.05 by more than simulation error, the asymptotic guarantee is not biting at this sample size.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A researcher can test every concept in a pretrained dictionary, tens of thousands of hypotheses, on a few hundred observations and report a selected set with a k-FWER guarantee rather than a Bonferroni-corrected handful.
  • The same bootstrap quantile yields generalized simultaneous confidence intervals for all tested concept parameters, not just a rejection list.
  • The framework removes most researcher degrees of freedom: no hand-coded topics, no manual labels, and no discretionary choice of which concepts to test, making the analysis easier to preregister and replicate.
  • Concept descriptions can be generated and evaluated statistically, so automatic interpretation becomes part of the inferential pipeline instead of an unquantified post hoc step.
  • Because the theory covers one-sided, two-sided, studentized, and approximate-mean statistics, the testing core transfers to other high-dimensional selective inference problems beyond text.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the theorems hold, the framework effectively redefines unsupervised discovery as screening over a fixed concept basis; the 'space of concepts' is whatever the pretrained dictionary learned, so discoveries are relative to that model's ontology rather than an absolute scientific ground truth.
  • The asymptotic guarantee is paired with finite-sample applications at n=930 and n=453, but the paper presents no simulations; a realistic correlated-binary Monte Carlo would show whether the k-FWER bound is already reliable at those sample sizes.
  • The A-score validates a description against dictionary activations, not against an external scientific construct; the framework still needs domain-level validation before a discovered concept is treated as a real social or economic mechanism.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an end-to-end framework for unsupervised discovery from unstructured data: pretrained sparse autoencoder features are mapped to high-dimensional binary concept dictionaries, feature-level hypotheses are tested with procedures claimed to control the k-FWER in high dimensions, and the resulting discoveries are labeled and evaluated by an LLM-based 'autointerp' pipeline with a new A-score estimator. The main theoretical contribution is an extension of Romano-Wolf (2007) k-FWER-controlling step-down/step-up algorithms to the high-dimensional regime, using the k-max multiplier bootstrap and high-dimensional CLT results of Ding et al. (2025). The paper reanalyzes Bursztyn et al. (2023) and Stantcheva (2024) as applications.

Significance. If the main theorem is correct, the paper provides a useful extension of resampling-based k-FWER control to settings where p grows much faster than n and test statistics are arbitrarily dependent. The proof is serious and builds on state-of-the-art high-dimensional CLT results. The framework is also unusually transparent and reproducible: a Jupyter notebook is provided, the SAE features are pretrained externally, and the pipeline has few researcher degrees of freedom. The A-score formalization is a reasonable step toward quantitative evaluation of autointerp descriptions. However, the empirical sections currently overstate the support provided by the theory: the implemented algorithm as written is internally inconsistent, the asymptotic regime is not connected to the reported sample sizes by any finite-sample evidence, and the A-score's target is defined relative to the SAE feature itself rather than an independent semantic ground truth. These issues are fixable but require substantive revision.

major comments (4)
  1. [Section 3.5, Algorithm 1 (lines 14-16)] The algorithm computes studentized t-statistics T_{n,j} = sqrt(n) * theta_hat_j / sqrt(Sigma_hat_jj), but line 15 defines the critical value as the quantile of |S^B_n|_{[k]} from Section 3.2, which is the unstudentized multiplier bootstrap. Propositions 1-2 show that studentization requires the bootstrap statistic to be premultiplied by Lambda_hat^{-1/2}. As written, the threshold in line 16 is on the raw covariance scale, not the standardized scale of the test statistics, so Algorithm 1 does not implement Theorem 2 or Propositions 1-2. This is not a cosmetic issue: the comparison in line 16 is invalid for features with heterogeneous variances unless the bootstrap is studentized.
  2. [Section 4.1 (claim after Figure 1); Section 4.2] The text states: 'By (large sample) guarantee of Theorem 2, the probability that 5 or more of these 9 discoveries is false is less than 5%.' Theorem 2 is only an asymptotic claim, and no asymptotic embedding is specified. For bounded features B_n=O(1), and at the reported sample sizes log^5(p)/n is about 78 (n=930, p=11901) and 84 (n=453, p=3826). The Ding et al. approximation error bound is therefore not small at these (n,p); with k=5 the k^8 factor makes the theoretical bound even larger. If one instead treats p as fixed, the fixed-p Romano-Wolf asymptotics apply, but that is not the high-dimensional regime claimed. No simulations or finite-sample diagnostics are provided. Either supply simulations demonstrating k-FWER control under realistic dependence and feature frequencies, or explicitly state that the applications are exploratory and not covered by Theorem 2 at the reported sample
  3. [Section 3.2, Assumption M(ii); Section 4.1/4.2 feature filtering] Assumption M(ii) requires the variances of all p coordinate statistics to be uniformly bounded below by b_1^2. The empirical sections drop only deterministic zeros; they do not address near-degenerate features whose population activation probabilities are very small. In a large SAE dictionary many features will activate in very few responses, so their variances can be arbitrarily close to zero. The Stantcheva application filters features by pretraining token frequency, not by sample frequency, so it does not ensure M(ii) holds for the tested set. The paper should state whether M(ii) is plausibly satisfied and, if not, how the set of tested hypotheses is to be restricted without data snooping.
  4. [Section 3.4, A-score estimand] The A-score is defined as the probability that the LLM classifier CLS(Z_i, eta_hat_j) agrees with the SAE feature indicator Y_ij. This is a coherence measure with respect to the SAE feature, not an independent measure of whether the natural-language description captures a scientifically meaningful concept. A description can receive a high A-score by tracking an artifact of the SAE/LLM pipeline rather than a true latent concept. Since the paper's discoveries are ultimately reported in natural language, this limitation is load-bearing. The manuscript should explicitly state that A-score validates the description as a predictor of the SAE feature, and should discuss what external validation (e.g., human labels or an independent concept benchmark) would be needed before substantive conclusions are drawn.
minor comments (5)
  1. [Section 4.1 vs Table 1] The text says 'A-scores for all discoveries range from 0.98 to 0.66,' but Table 1 lists A-scores including 0.515 (feature 3518) and 0.650. The range statement should be corrected or the table/figure reconciled.
  2. [Section 3.2, Assumption R] The notation p_n is used, but the paper does not specify how p is defined in the empirical applications: is p the number of non-degenerate features in the estimation sample, and is it treated as fixed or growing with n in the asymptotic thought experiment? This should be clarified, since Theorem 2 is stated as n,p going to infinity jointly.
  3. [Algorithm 1, line 17] The notation CIC_j is introduced without definition. It appears to denote a generalized simultaneous confidence interval for theta_j; please define it explicitly.
  4. [References] The text repeatedly cites 'Chernozhuokov et al.' but the reference list and standard spelling use 'Chernozhukov.' Please standardize.
  5. [Section 4.1] The 'less than 5%' formulation should be 'asymptotically at most alpha' (or 'approximately alpha') to avoid implying an exact finite-sample guarantee.

Circularity Check

0 steps flagged

No significant circularity: Theorem 2 is an external-result application; the A-score is an explicit operationalization, and the Assumption R violation is a validity concern, not circularity.

full rationale

The central statistical derivation is not circular. Theorem 2's proof directly invokes external results: the step-down construction of Romano and Wolf (2007) and the high-dimensional CLT/bootstrap for the k-th largest coordinate of Ding et al. (2025). The bootstrap critical values are computed from the data, and the claimed k-FWER guarantee is a theorem conditional on Assumptions M and R; it is not obtained by fitting a parameter to the target quantity and then renaming that fit as a prediction. Lemma 1 and Lemma 2 extend Ding et al. to approximate means and studentization, but the proofs quote the external Lemmas A.6/A.8 rather than assuming the conclusion. The paper's own prior work (Carlson and Dell 2025) is cited only as related literature, not as load-bearing support, and no uniqueness theorem from the authors is invoked to force a choice. The A-score evaluation defines quality as agreement between an LLM classifier and the SAE feature activation indicator; this is an explicitly operationalized internal consistency measure, not a hidden circular derivation, and it does not feed back into the k-FWER theorem. The applications do violate or at least do not verify Assumption R at n=930, p=11901 and n=453, p=3826, so the theorem's finite-sample guarantee may not be operative for the reported discoveries; however, that is an applicability/robustness concern, not an instance of the derivation being equivalent to its inputs. No fitted quantity is relabeled as a prediction, and no self-citation chain supplies the central result. Thus there is no significant circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central statistical theory rests on standard high-dimensional CLT assumptions and external results from Ding et al. (2025). The empirical framework additionally assumes that SAE features are human-interpretable and exhaustive, which is a domain hypothesis rather than a proven fact. The main free parameters are applied-researcher choices (k, feature filters, exemplar counts) that affect the reported discoveries but not the theorems.

free parameters (5)
  • k (error-control parameter) = 5
    The number of tolerated false rejections. Set to 5 in both applications, but only after observing that k=1 yielded too few discoveries (Section 4.1: "we make discoveries now setting k=5"). This is a post hoc choice that affects the reported discovery set.
  • Feature frequency filter = keep features with below-median token activation frequency in the SAE training corpus
    In Section 4.2, the researcher "reduce[s] the space of features by half, filtering out all features that, in the corpus on which the Gemma Scope SAEs were trained, had greater than median empirical token activation frequency." This is a hand-chosen dimensionality reduction not preregistered.
  • Number of autointerp exemplars L = 20
    Appendix 6.3: "We use L=20 in the empirical examples." The number of activating texts given to the explainer LLM is a modeling choice.
  • SAE layer selection = layer 12 of Gemma 2 2B
    Section 4.1: "we implement a Dict function based on the Gemma Scope autoencoders ... specifically the SAE trained on the residual stream of layer 12." Layer choice affects which concepts are captured.
  • Evaluation sample split proportion = 10% held out
    Section 4.1 and 4.2 use a 10% held-out evaluation dataset. The paper recommends this uneven split but does not justify it beyond stability concerns.
axioms (6)
  • domain assumption Assumptions M and R (sub-exponential tails, moment bounds, and rate condition B_n^2 log^5(pn)=o(n))
    Section 3.2 states these as the conditions for all theorems. They are standard in the high-dimensional CLT literature, but are unverified for the empirical applications (especially at n≈500).
  • standard math High-dimensional CLT and bootstrap results of Ding et al. (2025) for the k-th largest coordinate
    Theorems 1 and 2 and Lemmas 1 and 2 rely directly on Theorem 2.2 and Lemmas A.6/A.8 of Ding et al. (2025). The paper does not prove these external results, only extends them.
  • domain assumption SAE features are monosemantic and collectively exhaustive of the concepts of interest
    Section 3.1 claims each dictionary entry is a single intelligible concept and that the full set is exhaustive. This is a contested hypothesis from the interpretability literature (e.g., Leask et al. 2025), and the paper's interpretation of discoveries depends on it.
  • domain assumption i.i.d. sampling from a super-population P and SUTVA for causal claims
    Section 3.1 assumes observations are i.i.d.; Example 1 uses the Horvitz-Thompson transformation requiring SUTVA and fixed treatment probability π. These are standard but unstated in the empirical applications.
  • domain assumption The LLM classifier CLS is non-stochastic and the A-score estimand is well-defined
    Section 3.4 defines CLS as non-stochastic, but in practice GPT-OSS 20B with low reasoning effort is stochastic. Proposition 3 conditions on η_j and treats S_ij as i.i.d., which may not hold exactly for stochastic LLM outputs.
  • domain assumption P(S_ij=1|η_j) is bounded away from 0 and 1
    Proposition 3 assumes this for asymptotic normality of the A-score. For very good or very bad descriptions this may fail, but the paper uses large enough sets.

pith-pipeline@v1.3.0-alltime-deepseek · 33111 in / 11246 out tokens · 113509 ms · 2026-08-04T00:15:48.660565+00:00 · methodology

0 comments
read the original abstract

Social scientists are increasingly turning to unstructured datasets to unlock new empirical insights, e.g., estimating descriptive statistics of or causal effects on quantitative measures derived from text, audio, or video data. In many settings, unsupervised analysis is of primary interest, in that the researcher does not want to (or cannot) manually pre-specify all important aspects of the unstructured data to measure; they are interested in "discovery." This paper proposes a general and flexible framework for pursuing such discovery from unstructured data in a statistically principled way. The framework leverages recent methods from the literature on AI interpretability to map unstructured data points to high-dimensional, sparse, and interpretable "concept embeddings"; computes statistics from these concept embeddings for testing interpretable, concept-by-concept hypotheses; performs selective inference on these hypotheses using algorithms validated by new results in high-dimensional central limit theory, producing a selected set ("discoveries"); and both generates and evaluates human-interpretable natural language descriptions of these discoveries. The proposed framework has few researcher degrees of freedom, is robust to data snooping and other post-selection inference concerns, and facilitates fast and inexpensive sensitivity analysis and replication. Applications to recent descriptive and causal analyses of unstructured data in empirical economics are explored.

Figures

Figures reproduced from arXiv: 2511.01680 by Jacob Carlson.

Figure 1
Figure 1. Figure 1: Discoveries for Experiment 2 of Bursztyn et al. (2023) [PITH_FULL_IMAGE:figures/full_fig_p023_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ranked Discoveries for “High inflation is caused by...” in Stantcheva (2024) [PITH_FULL_IMAGE:figures/full_fig_p025_2.png] view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p033_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Econometrics with Pre-Trained Embeddings for Unstructured Data

    econ.EM 2026-07 accept novelty 7.0

    Pre-trained embeddings are valid in double machine learning when the target nuisance function lies in the span of the source-task representation; under that condition the downstream estimator can converge faster than ...

  2. In your own words: computationally identifying interpretable themes in free-text survey data

    cs.CY 2026-03 unverdicted novelty 6.0

    A computational framework identifies more coherent themes in free-text survey data on race, gender, and sexual orientation than previous methods, with applications for survey design, explaining variation, and detectin...

Reference graph

Works this paper leans on

57 extracted references · 21 linked inside Pith · cited by 2 Pith papers

  1. [1]

    The Voice of Monetary Policy

    Yuriy Gorodnichenko, Tho Pham, and Oleksandr Talavera. The Voice of Monetary Policy . American Economic Review, 113 0 (2): 0 548--584, February 2023. ISSN 0002-8282. doi:10.1257/aer.20220129. URL https://pubs.aeaweb.org/doi/10.1257/aer.20220129

    doi:10.1257/aer.20220129 2023

  2. [2]

    Persuading Investors : A Video ‐ Based Study

    Allen Hu and Song Ma. Persuading Investors : A Video ‐ Based Study . The Journal of Finance, 80 0 (5): 0 2639--2688, October 2025. ISSN 0022-1082, 1540-6261. doi:10.1111/jofi.13471. URL https://onlinelibrary.wiley.com/doi/10.1111/jofi.13471

    doi:10.1111/jofi.13471 2025

  3. [3]

    Understanding Economic Behavior Using Open -ended Survey Data

    Ingar Haaland, Christopher Roth, Stefanie Stantcheva, and Johannes Wohlfart. Understanding Economic Behavior Using Open -ended Survey Data . Technical Report w32421, National Bureau of Economic Research, Cambridge, MA, May 2024. URL http://www.nber.org/papers/w32421.pdf

    2024

  4. [4]

    Machine Learning as a Tool for Hypothesis Generation

    Jens Ludwig and Sendhil Mullainathan. Machine Learning as a Tool for Hypothesis Generation . The Quarterly Journal of Economics, 139 0 (2): 0 751--827, March 2024. ISSN 0033-5533, 1531-4650. doi:10.1093/qje/qjad055. URL https://academic.oup.com/qje/article/139/2/751/7515309

    doi:10.1093/qje/qjad055 2024

  5. [5]

    State of the Art : Economic Development Through the Lens of Paintings

    Clément Gorin, Stephan Heblich, and Yanos Zylberberg. State of the Art : Economic Development Through the Lens of Paintings . Technical Report w33976, National Bureau of Economic Research, Cambridge, MA, June 2025. URL http://www.nber.org/papers/w33976.pdf

    2025

  6. [6]

    American Life Histories

    David Lagakos, Stelios Michalopoulos, and Hans-Joachim Voth. American Life Histories . Technical Report w33373, National Bureau of Economic Research, Cambridge, MA, January 2025. URL http://www.nber.org/papers/w33373.pdf

    2025

  7. [7]

    Katz, and Christopher Palmer

    Peter Bergman, Raj Chetty, Stefanie DeLuca, Nathaniel Hendren, Lawrence F. Katz, and Christopher Palmer. Creating Moves to Opportunity : Experimental Evidence on Barriers to Neighborhood Choice . American Economic Review, 114 0 (5): 0 1281--1337, May 2024. ISSN 0002-8282. doi:10.1257/aer.20200407. URL https://pubs.aeaweb.org/doi/10.1257/aer.20200407

    doi:10.1257/aer.20200407 2024

  8. [8]

    The Impact of Unconditional Cash Transfers on Parenting and Children

    Patrick Krause, Elizabeth Rhodes, Sarah Miller, Alexander Bartik, David Broockman, and Eva Vivalt. The Impact of Unconditional Cash Transfers on Parenting and Children . Technical Report w34040, National Bureau of Economic Research, Cambridge, MA, July 2025. URL http://www.nber.org/papers/w34040.pdf

    2025

  9. [9]

    Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I

    Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382 0 (6671): 0 669--674, November 2023. doi:10.1126/science.adi6000. URL https://www.science.org/doi/10.1126/science.adi6000. Publisher: American Association for the Advancement of Science

    doi:10.1126/science.adi6000 2023

  10. [10]

    Large Language Models : An Applied Econometric Framework , December 2024

    Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. Large Language Models : An Applied Econometric Framework , December 2024. URL http://arxiv.org/abs/2412.07031. arXiv:2412.07031 [econ]

    arXiv 2024

  11. [11]

    A Unifying Framework for Robust and Efficient Inference with Unstructured Data , 2025

    Jacob Carlson and Melissa Dell. A Unifying Framework for Robust and Efficient Inference with Unstructured Data , 2025. URL https://arxiv.org/abs/2505.00282. Version Number: 2

    arXiv 2025

  12. [12]

    Justifying Dissent

    Leonardo Bursztyn, Georgy Egorov, Ingar Haaland, Aakaash Rao, and Christopher Roth. Justifying Dissent . The Quarterly Journal of Economics, 138 0 (3): 0 1403--1451, June 2023. ISSN 0033-5533, 1531-4650. doi:10.1093/qje/qjad007. URL https://academic.oup.com/qje/article/138/3/1403/7000850

    doi:10.1093/qje/qjad007 2023

  13. [13]

    Why Do We Dislike Inflation ? Technical Report w32300, National Bureau of Economic Research, Cambridge, MA, April 2024

    Stefanie Stantcheva. Why Do We Dislike Inflation ? Technical Report w32300, National Bureau of Economic Research, Cambridge, MA, April 2024. URL http://www.nber.org/papers/w32300.pdf

    2024

  14. [14]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Ch...

    2023

  15. [15]

    Romano and Michael Wolf

    Joseph P. Romano and Michael Wolf. Control of generalized error rates in multiple testing. The Annals of Statistics, 35 0 (4), August 2007. ISSN 0090-5364. doi:10.1214/009053606000001622. URL https://projecteuclid.org/journals/annals-of-statistics/volume-35/issue-4/Control-of-generalized-error-rates-in-multiple-testing/10.1214/009053606000001622.full

    doi:10.1214/009053606000001622 2007

  16. [16]

    Towards universality: Studying mechanistic similarity across language model architectures

    Junxuan Wang, Xuyang Ge, Wentao Shu, Qiong Tang, Yunhua Zhou, Zhengfu He, and Xipeng Qiu. Towards universality: Studying mechanistic similarity across language model architectures. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=2J18i8T0oI

    2025

  17. [17]

    Machine Learning : An Applied Econometric Approach

    Sendhil Mullainathan and Jann Spiess. Machine Learning : An Applied Econometric Approach . Journal of Economic Perspectives, 31 0 (2): 0 87--106, May 2017. ISSN 0895-3309. doi:10.1257/jep.31.2.87. URL https://pubs.aeaweb.org/doi/10.1257/jep.31.2.87

    doi:10.1257/jep.31.2.87 2017

  18. [18]

    Text as Data

    Matthew Gentzkow, Bryan Kelly, and Matt Taddy. Text as Data . Journal of Economic Literature, 57 0 (3): 0 535--574, September 2019 a . ISSN 0022-0515. doi:10.1257/jel.20181020. URL https://pubs.aeaweb.org/doi/10.1257/jel.20181020

    doi:10.1257/jel.20181020 2019

  19. [19]

    Text Algorithms in Economics

    Elliott Ash and Stephen Hansen. Text Algorithms in Economics . Annual Review of Economics, 15 0 (1): 0 659--688, September 2023. ISSN 1941-1383, 1941-1391. doi:10.1146/annurev-economics-082222-074352. URL https://www.annualreviews.org/content/journals/10.1146/annurev-economics-082222-074352

    doi:10.1146/annurev-economics-082222-074352 2023

  20. [20]

    Deep Learning for Economists

    Melissa Dell. Deep Learning for Economists . Journal of Economic Literature, 63 0 (1): 0 5--58, March 2025. ISSN 0022-0515, 2328-8175. doi:10.1257/jel.20241733. URL https://pubs.aeaweb.org/doi/10.1257/jel.20241733

    doi:10.1257/jel.20241733 2025

  21. [21]

    Program Evaluation with Remotely Sensed Outcomes , 2024

    Ashesh Rambachan, Rahul Singh, and Davide Viviano. Program Evaluation with Remotely Sensed Outcomes , 2024. URL https://arxiv.org/abs/2411.10959. Version Number: 2

    Pith/arXiv arXiv 2024

  22. [22]

    Shapiro, and Matt Taddy

    Matthew Gentzkow, Jesse M. Shapiro, and Matt Taddy. Measuring Group Differences in High ‐ Dimensional Choices : Method and Application to Congressional Speech . Econometrica, 87 0 (4): 0 1307--1340, 2019 b . ISSN 0012-9682. doi:10.3982/ECTA16566. URL https://www.econometricsociety.org/doi/10.3982/ECTA16566

    doi:10.3982/ecta16566 2019

  23. [23]

    Inference for Regression with Variables Generated by AI or Machine Learning , 2024

    Laura Battaglia, Timothy Christensen, Stephen Hansen, and Szymon Sacher. Inference for Regression with Variables Generated by AI or Machine Learning , 2024. URL https://arxiv.org/abs/2402.15585. Version Number: 5

    Pith/arXiv arXiv 2024

  24. [24]

    Causal Inference on Outcomes Learned from Text , 2025

    Iman Modarressi, Jann Spiess, and Amar Venugopal. Causal Inference on Outcomes Learned from Text , 2025. URL https://arxiv.org/abs/2503.00725. Version Number: 1

    Pith/arXiv arXiv 2025

  25. [25]

    Machine- Learning Tests for Effects on Multiple Outcomes , 2017

    Jens Ludwig, Sendhil Mullainathan, and Jann Spiess. Machine- Learning Tests for Effects on Multiple Outcomes , 2017. URL https://arxiv.org/abs/1707.01473. Version Number: 2

    Pith/arXiv arXiv 2017

  26. [26]

    Romano, Azeem M

    Joseph P. Romano, Azeem M. Shaikh, and Michael Wolf. Hypothesis Testing in Econometrics . Annual Review of Economics, 2 0 (1): 0 75--104, September 2010. ISSN 1941-1383, 1941-1391. doi:10.1146/annurev.economics.102308.124342. URL https://www.annualreviews.org/doi/10.1146/annurev.economics.102308.124342

    Pith/arXiv arXiv 2010

  27. [27]

    Controlling the False Discovery Rate : A Practical and Powerful Approach to Multiple Testing

    Yoav Benjamini and Yosef Hochberg. Controlling the False Discovery Rate : A Practical and Powerful Approach to Multiple Testing . Journal of the Royal Statistical Society. Series B (Methodological), 57 0 (1): 0 289--300, 1995. ISSN 00359246. URL http://www.jstor.org/stable/2346101. Publisher: [Royal Statistical Society, Oxford University Press]

    arXiv 1995

  28. [28]

    The Control of the False Discovery Rate in Multiple Testing under Dependency

    Yoav Benjamini and Daniel Yekutieli. The Control of the False Discovery Rate in Multiple Testing under Dependency . The Annals of Statistics, 29 0 (4): 0 1165--1188, 2001. ISSN 00905364, 21688966. URL http://www.jstor.org/stable/2674075. Publisher: Institute of Mathematical Statistics

    arXiv 2001

  29. [29]

    Improved central limit theorem and bootstrap approximations in high dimensions

    Victor Chernozhuokov, Denis Chetverikov, Kengo Kato, and Yuta Koike. Improved central limit theorem and bootstrap approximations in high dimensions. The Annals of Statistics, 50 0 (5), October 2022. ISSN 0090-5364. doi:10.1214/22-AOS2193. URL https://projecteuclid.org/journals/annals-of-statistics/volume-50/issue-5/Improved-central-limit-theorem-and-boots...

    doi:10.1214/22-aos2193 2022

  30. [30]

    Gaussian Multiplier Bootstrap Procedure for the k-th Largest Coordinate of High - Dimensional Statistics , 2025

    Yixi Ding, Qizhai Li, Yuke Shi, and Liuquan Sun. Gaussian Multiplier Bootstrap Procedure for the k-th Largest Coordinate of High - Dimensional Statistics , 2025. URL https://arxiv.org/abs/2508.14400. Version Number: 1

    Pith/arXiv arXiv 2025

  31. [31]

    High- Dimensional Econometrics and Regularized GMM , 2018

    Alexandre Belloni, Victor Chernozhukov, Denis Chetverikov, Christian Hansen, and Kengo Kato. High- Dimensional Econometrics and Regularized GMM , 2018. URL https://arxiv.org/abs/1806.01888. Version Number: 2

    Pith/arXiv arXiv 2018

  32. [32]

    Phase transition and regularized bootstrap in large-scale \ t\ -tests with false discovery rate control

    Weidong Liu and Qi-Man Shao. Phase transition and regularized bootstrap in large-scale \ t\ -tests with false discovery rate control. The Annals of Statistics, 42 0 (5), October 2014. ISSN 0090-5364. doi:10.1214/14-AOS1249. URL https://projecteuclid.org/journals/annals-of-statistics/volume-42/issue-5/Phase-transition-and-regularized-bootstrap-in-large-sca...

    doi:10.1214/14-aos1249 2014

  33. [33]

    Sparse Autoencoders for Hypothesis Generation , 2025

    Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse Autoencoders for Hypothesis Generation , 2025. URL https://arxiv.org/abs/2502.04382. Version Number: 3

    Pith/arXiv arXiv 2025

  34. [34]

    Towards A Rigorous Science of Interpretable Machine Learning , 2017

    Finale Doshi-Velez and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning , 2017. URL https://arxiv.org/abs/1702.08608. Version Number: 2

    Pith/arXiv arXiv 2017

  35. [35]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosema...

    2024

  36. [36]

    Sparse Autoencoders Do Not Find Canonical Units of Analysis , February 2025

    Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse Autoencoders Do Not Find Canonical Units of Analysis , February 2025. URL http://arxiv.org/abs/2502.04878. arXiv:2502.04878 [cs]

    Pith/arXiv arXiv 2025

  37. [37]

    Use Sparse Autoencoders to Discover Unknown Concepts , Not to Act on Known Concepts , 2025

    Kenny Peng, Rajiv Movva, Jon Kleinberg, Emma Pierson, and Nikhil Garg. Use Sparse Autoencoders to Discover Unknown Concepts , Not to Act on Known Concepts , 2025. URL https://arxiv.org/abs/2506.23845. Version Number: 1

    Pith/arXiv arXiv 2025

  38. [38]

    Transcoders Beat Sparse Autoencoders for Interpretability , 2025

    Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders Beat Sparse Autoencoders for Interpretability , 2025. URL https://arxiv.org/abs/2501.18823. Version Number: 2

    Pith/arXiv arXiv 2025

  39. [39]

    Language models can explain neurons in language models

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html, 2023

    2023

  40. [40]

    A Multimodal Automated Interpretability Agent , 2024

    Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A Multimodal Automated Interpretability Agent , 2024. URL https://arxiv.org/abs/2404.14394. Version Number: 2

    Pith/arXiv arXiv 2024

  41. [41]

    Automatically Interpreting Millions of Features in Large Language Models , 2024

    Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically Interpreting Millions of Features in Large Language Models , 2024. URL https://arxiv.org/abs/2410.13928. Version Number: 3

    Pith/arXiv arXiv 2024

  42. [42]

    Jumping Ahead : Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders , 2024

    Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping Ahead : Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders , 2024. URL https://arxiv.org/abs/2407.14435. Version Number: 3

    Pith/arXiv arXiv 2024

  43. [43]

    Alexander, and Daniel C

    Ahmed Abdulaal, Hugo Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, and Daniel C. Castro. An X - Ray Is Worth 15 Features : Sparse Autoencoders for Interpretable Radiology Report Generation , 2024. URL https://arxiv.org/abs/2410.03334. Version Number: 1

    Pith/arXiv arXiv 2024

  44. [44]

    Calmon, and Himabindu Lakkaraju

    Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon, and Himabindu Lakkaraju. Interpreting CLIP with Sparse Linear Concept Embeddings ( SpLiCE ), 2024. URL https://arxiv.org/abs/2402.10376. Version Number: 2

    Pith/arXiv arXiv 2024

  45. [45]

    Towards multimodal interpretability: Learning sparse interpretable features in vision transformers, Apr 2024

    Hugo Fry. Towards multimodal interpretability: Learning sparse interpretable features in vision transformers, Apr 2024. URL https://www.lesswrong.com/posts/iYFuZo9BMvr6GgMs5/case-study-interpreting-manipulating-and-controlling-clip. Accessed: 2024-05-16

    2024

  46. [46]

    Case study: Interpreting, manipulating, and controlling clip with sparse autoencoders, Aug 2024

    Gytis Daujotas. Case study: Interpreting, manipulating, and controlling clip with sparse autoencoders, Aug 2024. URL https://www.lesswrong.com/posts/iYFuZo9BMvr6GgMs5/case-study-interpreting-manipulating-and-controlling-clip. Accessed: 2025-10-03

    2024

  47. [47]

    Daniel Pluth, Yu Zhou, and Vijay K. Gurbani. Sparse Autoencoder Insights on Voice Embeddings , 2025. URL https://arxiv.org/abs/2502.00127. Version Number: 1

    Pith/arXiv arXiv 2025

  48. [48]

    Central limit theorems and bootstrap in high dimensions

    Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Central limit theorems and bootstrap in high dimensions. The Annals of Probability, 45 0 (4), July 2017. ISSN 0091-1798. doi:10.1214/16-AOP1113. URL https://projecteuclid.org/journals/annals-of-probability/volume-45/issue-4/Central-limit-theorems-and-bootstrap-in-high-dimensions/10.1214/16-AOP1113.full

    doi:10.1214/16-aop1113 2017

  49. [49]

    High- Dimensional Data Bootstrap

    Victor Chernozhukov, Denis Chetverikov, Kengo Kato, and Yuta Koike. High- Dimensional Data Bootstrap . Annual Review of Statistics and Its Application, 10 0 (1): 0 427--449, March 2023. ISSN 2326-8298, 2326-831X. doi:10.1146/annurev-statistics-040120-022239. URL https://www.annualreviews.org/doi/10.1146/annurev-statistics-040120-022239

    doi:10.1146/annurev-statistics-040120-022239 2023

  50. [50]

    Inference on Winners

    Isaiah Andrews, Toru Kitagawa, and Adam McCloskey. Inference on Winners . The Quarterly Journal of Economics, 139 0 (1): 0 305--358, January 2024. ISSN 0033-5533, 1531-4650. doi:10.1093/qje/qjad043. URL https://academic.oup.com/qje/article/139/1/305/7276491

    doi:10.1093/qje/qjad043 2024

  51. [51]

    Adding Error Bars to Evals : A Statistical Approach to Language Model Evaluations , 2024

    Evan Miller. Adding Error Bars to Evals : A Statistical Approach to Language Model Evaluations , 2024. URL https://arxiv.org/abs/2411.00640. Version Number: 1

    Pith/arXiv arXiv 2024

  52. [52]

    Matthew Gentzkow and Jesse M. Shapiro. What Drives Media Slant ? Evidence From U . S . Daily Newspapers . Econometrica, 78 0 (1): 0 35--71, January 2010. ISSN 0012-9682. doi:10.3982/ECTA7195. URL https://doi.org/10.3982/ECTA7195. Publisher: John Wiley & Sons, Ltd

    doi:10.3982/ecta7195 2010

  53. [53]

    Gemma Scope : Open Sparse Autoencoders Everywhere All At Once on Gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope : Open Sparse Autoencoders Everywhere All At Once on Gemma 2, 2024. URL https://arxiv.org/abs/2408.05147. Version Number: 2

    Pith/arXiv arXiv 2024

  54. [54]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  55. [55]

    Eliciting People ’s First - Order Concerns : Text Analysis of Open - Ended Survey Questions

    Beatrice Ferrario and Stefanie Stantcheva. Eliciting People ’s First - Order Concerns : Text Analysis of Open - Ended Survey Questions . AEA Papers and Proceedings, 112: 0 163--169, May 2022. ISSN 2574-0768, 2574-0776. doi:10.1257/pandp.20221071. URL https://pubs.aeaweb.org/doi/10.1257/pandp.20221071

    doi:10.1257/pandp.20221071 2022

  56. [56]

    High- Dimensional Probability : An Introduction with Applications in Data Science

    Roman Vershynin. High- Dimensional Probability : An Introduction with Applications in Data Science . Cambridge University Press, 1 edition, September 2018. ISBN 978-1-108-23159-6 978-1-108-41519-4. doi:10.1017/9781108231596. URL https://www.cambridge.org/core/product/identifier/9781108231596/type/book

    arXiv 2018

  57. [57]

    Moving beyond sub- Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression

    Arun Kumar Kuchibhotla and Abhishek Chakrabortty. Moving beyond sub- Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression. Information and Inference: A Journal of the IMA, 11 0 (4): 0 1389--1456, December 2022. ISSN 2049-8772. doi:10.1093/imaiai/iaac012. URL https://academic.oup.com/imaiai/article/11/4/13...

    doi:10.1093/imaiai/iaac012 2022