Pith. sign in

REVIEW 13 major objections 55 references

A training-free agent that retrieves activations from a vector database can match or beat trained interpretability systems on concept discovery, detection, steering, and secret elicitation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:55 UTC pith:KZTRKDSN

load-bearing objection A genuinely useful training-free interpretability baseline, but the abstract overstates the secret-elicitation result and the BILLS comparison is confounded by post hoc indexing; worth refereeing after fixes. the 13 major comments →

arxiv 2607.16448 v1 pith:KZTRKDSN submitted 2026-07-17 cs.LG cs.AI

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

Sriram Balasubramanian , Soheil Feizi This is my paper
classification cs.LG cs.AI
keywords interpretabilityactivation vectorsretrievalsparse autoencodersactivation oracleslinear probesconcept discoverymodel steering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether expensive, training-based interpretability methods — sparse autoencoders (trained dictionaries of activation features) and activation oracles (LLMs fine-tuned to read activations) — extract anything beyond what is recoverable from the activations and texts they were trained on. To answer this, the authors build HARP (Hypothesis-driven Agentic Retrieval and Probing), an agent with no learned interpretability component: it queries a vector database of 26 million (activation, surrounding-text) pairs, forms a hypothesis about the dominant theme, builds a linear direction by averaging retrieved exemplars, projects that direction out, and repeats. HARP matches or beats the trained baselines on concept discovery, concept detection, and model steering, and comes close on secret elicitation, where its own Figure 4 shows the activation oracle ahead (80% vs 73%). The broader claim is that training-based methods act like lossy databases: once trained, they cannot be extended, whereas HARP can index new documents on demand. A sympathetic reader would care because the result reframes the interpretability cost question from 'how to train better decomposers' to 'how to build better retrievable indexes.'

Core claim

On the paper's own terms, the discovery is that a lossless retrieval system can substitute for compression-based interpretability training. HARP stores activations paired with their textual contexts rather than compressing them into a dictionary or a fine-tuned verbalizer, and lets a tool-using language-model agent manipulate them with averages, differences, and projections. Starting from a target activation, the agent retrieves nearest neighbours, identifies a coherent theme in the snippets, materializes it as a concept vector, projects it out, and re-queries until several distinct concepts are peeled off. The reported results show HARP ahead of both a sparse autoencoder and an activation o

What carries the argument

The load-bearing object is a bias-corrected vector database of roughly 26 million (activation, context) pairs drawn from a small general corpus, plus an agent loop that treats concept discovery as sequential residual decomposition. The agent's central operation is difference-of-means: average the activations of retrieved snippets that share a suspected theme (subtracting either a zero vector or a contrast set), normalize the result into a unit concept direction, and project it out of the target before re-querying, so each iteration surfaces a new theme. A projection onto the top principal components of the corpus is applied at query time to suppress generic content-free directions, and a sto

Load-bearing premise

HARP's comparisons stand or fall on the assumption that roughly 3,900 documents' worth of activations (26 million vectors) span the concepts in the evaluation tasks, so that nearest-neighbour retrieval in activation space surfaces the same themes a trained method would; the legal-document case shows what happens when that assumption fails.

What would settle it

To test the load-bearing premise, take a set of passages from a narrow domain absent from HARP's index (for instance, a specialized legal or medical subtopic), run the full concept-discovery method, then add a few dozen documents from that domain to the index and rerun. If HARP already matches the oracle before indexing, retrieval coverage is not the limiting factor; if it only succeeds after indexing, the 'lossless database' framing is confirmed. A second check: identify a concept whose retrieved snippets are coherent but whose resulting direction fails on held-out positive/negative pairs, wh

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If HARP is right, the practical cost of interpretability shifts from training large decomposers to indexing activations once, because new concepts can be added by inserting new documents without retraining.
  • The single nearest-neighbour lookup recovering most of the oracle's secret-elicitation score suggests that part of a fine-tuned verbalizer's power is retrieval over what it saw in training, not new insight.
  • Because HARP returns the linear probe and tool trace for every concept, interpretability results become inspectable and verifiable rather than requiring trust in an oracle's verbal answer.
  • The gap over sparse autoencoders on steering (concept relevance 0.78 vs 0.42, LM-judge 0.20 vs 0.02) indicates that directions built from retrieved exemplars are more usable as control knobs than dictionary features.
  • The legal-document failure-and-recovery case implies current training-based methods silently fail on under-represented domains and cannot self-correct, while retrieval systems can.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implicit consequence the authors do not spell out: if retrieval over raw activations is enough, then the scaling race for sparse autoencoders may be optimizing a lossy compression objective rather than interpretability, and benchmark scores that only measure recovery of training-set concepts will not reveal the difference.
  • The iterative project-out loop is mathematically similar to greedy basis pursuit (matching pursuit) over retrieved exemplars; one could formalize HARP as a retrieval-driven sparse coding scheme and derive guarantees about which concept subspaces it recovers given a well-covered index.
  • A testable extension: run HARP on a concept known to be absent from the index corpus; the lossless-database hypothesis predicts a sharp failure that disappears once a few dozen domain documents are indexed, which would cleanly separate retrieval coverage from genuine linear-encoding structure.
  • Because HARP's discovered directions are ordinary vectors in activation space, they could be fed back into trained systems as initialization for sparse-autoencoder dictionaries or as high-quality labels for training activation oracles, turning the retrieval system into a data generator for the methods it currently outperforms.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

13 major / 0 minor

Summary. The paper proposes HARP, a training-free interpretability method that couples an LLM agent with a vector database of (residual-stream activation, text-context) pairs and a small toolkit of linear operations (projection, difference-of-means, low-rank subspace fitting). The agent iteratively retrieves nearby activations, hypothesizes a shared concept, constructs a linear concept direction, projects it out, and repeats. HARP is evaluated against an activation oracle and a Gemma-Scope SAE pipeline on four tasks: unsupervised concept discovery on BILLS/WIKI, AXBENCH concept detection and steering, and secret/gender elicitation from fine-tuned models. The paper's central claim is that retrieval over a lossless activation database plus linear algebra matches or beats trained interpretability systems, implying that current training-based methods mainly recombine information already present in their training data.

Significance. The research question is well posed and timely, and the proposed system is a useful baseline for the interpretability community. If the empirical claims held, the paper would provide a strong argument that the marginal value of expensive SAE/oracle training is limited relative to retrieval over raw activations with an agent. The paper is unusually concrete: it releases code, gives full skill prompts, specifies hyperparameters, and describes the vector database construction in detail, all of which makes the method reproducible. However, the manuscript as written contains a direct contradiction between the abstract and the secret-elicitation results, provides no uncertainty quantification for any headline number, and includes a task-specific index augmentation for BILLS that confounds the comparison. These issues are load-bearing and prevent me from endorsing the central claim in its current form.

major comments (13)
  1. [Abstract vs. §3.3, Fig. 4] The abstract states that HARP 'outperforms both activation oracles and SAE-based agents on concept discovery, concept detection, model steering, and secret elicitation.' Figure 4 and the accompanying text report the opposite for secret elicitation: the activation oracle achieves 80%, HARP 73%, and the SAE 51%, with the text explicitly saying 'The activation oracle wins on average.' The contribution bullet claiming HARP 'matches or beats' the oracle on secret elicitation is also not supported by the reported numbers. This is not a minor wording issue; the abstract misstates a headline result. Please correct the abstract and contribution list and state precisely where HARP beats, matches, or trails each baseline.
  2. [§3, Checklist item 7] No error bars, confidence intervals, or significance tests appear anywhere in the experimental sections. The NeurIPS checklist answers [No] to the statistical-significance question and justifies this with 'All error bars are much smaller than effect size, and all effects are statistically significant,' but no error-bar values or test results are shown. This is particularly problematic because several comparisons rest on modest gaps (e.g., concept-detection AUC 0.892 vs. 0.877, gender 0.94 vs. 0.91, secret elicitation 73% vs. 80% over 10 prompts), and the agent is run at temperature=1.0, so repeated runs should exhibit variance. Please report standard errors, bootstrapped intervals, or per-item/per-concept results for all headline claims, and either remove or substantiate the statement that all effects are significant.
  3. [§3.1] The BILLS concept-discovery result is confounded by a task-specific index expansion. The text states that the initially indexed corpus had poor legal-document coverage and that 'Indexing a small set of legal documents with no overlap with the BILLS subset was sufficient to recover strong performance.' As reported, this means HARP received additional, task-relevant documents that the SAE and activation oracle did not have access to. Since BILLS is one of only two concept-discovery datasets, the claimed coverage gains (1.9×/1.6×) may reflect this augmentation rather than the retrieval-plus-probing mechanism. The paper should report BILLS performance with the original index alone, describe how the legal augmentation set was selected without using BILLS evaluation labels, and carefully qualify the 'lossless database' framing in §3.4: the index is actually a small Pile subset plus a legal add
  4. [§3.2, §B.5] The steering results report the 'per-concept best LM-judge' and 'per-concept best concept-relevance' scores after sweeping an alpha grid separately per concept. Selecting the best alpha on the evaluation set is an oracle selection procedure and gives an optimistic estimate of deployable performance; the 10× improvement over SAEs (0.20 vs. 0.02) may be sensitive to this choice. Please report results at a fixed alpha, a cross-validated selection rule, or at least the full per-alpha curves, so the reader can see whether the advantage persists without test-set peeking.
  5. [§3.1] The phrase 'lossless database' is used throughout the paper, but the actual index is not literally lossless: activations are stored in bfloat16, inner-product search is approximate (DiskANN with nprobe=10), and only 26M vectors from roughly 3,900 documents are indexed. More importantly, the database contains text-context windows alongside activations, so the agent receives surface-level textual evidence that an activation-only baseline does not. This does not invalidate the method, but it should be stated precisely so the reader understands the comparison is between 'agent + activation retrieval + text snippets' and 'trained verbalizer / SAE readout,' not between pure activation-space retrieval and training.
  6. [§3.3] After Figure 4, the text says 'HARP's training-free retrieval matches a fine-tuned activation verbalizer on the secret task.' Given the reported 73% vs. 80%, 'approaches' or 'nearly matches' is more accurate than 'matches.' The same paragraph also says the oracle 'wins on average' and HARP 'closes most of the gap'; please make the wording consistent with the figure.
  7. [Checklist item 1] The checklist answers [Yes] to the question of whether the abstract/introduction claims accurately reflect the contributions. Given the secret-elicitation discrepancy described above, a [No] answer with a correction plan would be more consistent.
  8. [§3.4 and Limitations] The discussion claims HARP is 'competitive with both baselines across all four tasks.' This is accurate only in the weak sense that HARP is not far behind on secrets; it should be phrased as 'competitive or better on three of four tasks, and close on the fourth' to align with the reported numbers.
  9. [Fig. 4] Figure 4 is dense and hard to read, with many per-secret percentages. A table with per-secret scores and confidence intervals, or a boxplot, would be much clearer than the current stacked-bar layout.
  10. [Appendix D.1] The discover_concepts skill instructs the agent to 'return exactly max_concepts concepts' and to accept a best-effort concept even if it fails the strict removal check. This is a reasonable termination rule, but it may systematically produce low-quality filler concepts; the paper should note that the acceptance criterion is a hard count rather than concept quality.
  11. [§2.2] The top-k PCA projection parameter k is fixed at 10 in all experiments. Since retrieval quality is central to the method, please report sensitivity to k, or at least justify the fixed value with a sentence or two.
  12. [Abstract] The abstract says HARP is 'substantially cheaper and more flexible' than training-based methods. The Limitations section correctly notes that per-query inference is more expensive than a forward pass. Please make the abstract's cost claim more precise, e.g., 'no upfront training cost' rather than 'substantially cheaper' without qualification.
  13. [§3.2] In concept detection, each concept is defined by a Gemma-Scope SAE feature label, and the SAE baseline is allowed to use exactly that feature. This is a favorable 'oracle' setting for the SAE, and the paper should note explicitly that the SAE's 0.70 AUC is therefore an upper bound for that particular SAE readout pipeline, not a generic SAE result.

Circularity Check

0 steps flagged

No load-bearing circularity: HARP's outputs come from external retrieval and linear algebra, not from fitted evaluation quantities or self-citation.

full rationale

The paper's central comparison is HARP (retrieval + linear probes) vs SAEs and activation oracles on independent evaluation sets. HARP never fits its probes, concept vectors, or database to the evaluation labels: concept directions come from LLM-authored contrast pairs or from the mean/difference of retrieved database vectors, and secret elicitation is a single nearest-neighbor readout of retrieved snippets. No equation in the paper reduces a prediction to a fitted parameter, and no load-bearing claim is justified solely by a self-citation (ref. [1] is a prior EACL paper by the same authors but is not used to establish HARP's results). The post hoc BILLS legal-document indexing is disclosed in Section 3.1 and framed as an advantage of the training-free design; while it is a legitimate experimental-fairness concern — SAEs/oracles did not receive equivalent additional data — it is not a circular reduction, because the indexed documents are disjoint from the evaluation subset and no parameter is fit to the BILLS ground-truth concepts. The abstract's secret-elicitation statement ('HARP outperforms ... on secret elicitation') is contradicted by Figure 4 (oracle 80% vs HARP 73%), but that is a factual/accuracy issue rather than a circularity. The Limitations section explicitly bounds HARP to directions/subspaces expressible by linear tools, which is an honest scope restriction, not a hidden circular assumption. On the circularity axis, the derivation is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The core empirical claim rests on the linear-representation hypothesis, the assumption that nearest-neighbor retrieval in bias-subtracted residual space surfaces semantic themes, and the assumption that GPT-family models can reliably hypothesize and score concepts. No new physical entities are introduced; the index size, context window, k, and steering grids are hand-chosen settings that should be audited.

free parameters (8)
  • k (top principal components projected out at query time) = 10
    Chosen by hand for all experiments (Appendix A.2); affects retrieval neighborhoods.
  • T (token chunk size) = 1024
    Chunk length for activation extraction (Section 2.1).
  • context window around target token = +/-64 tokens (display truncated to +/-100 chars)
    Defines the textual context paired with each activation (Section 2.1).
  • top_k retrieval = 10
    Number of nearest neighbors returned per query; used across skills (Appendix D).
  • n_pairs contrast pairs for detect_specific_concept = 25
    Number of agent-authored minimal pairs used to build the probe (Appendix D.3).
  • steering alpha grid (per-concept best) = HARP: {10,25,50,100,200,400,800}; SAE: {0.4,1.2,2.0,4.0,8.0,20,60}
    Per-concept best factor selected from different method-specific grids; can inflate reported LM-judge (Appendix B.5).
  • bias estimate sample size = 2 million activations
    Corpus mean estimated once per layer (Appendix A.1).
  • K, number of concepts requested = 5
    In concept discovery, all methods return five concepts (Section 3.1).
axioms (6)
  • domain assumption Residual-stream semantics are linearly decodable: concepts correspond to directions or low-dimensional subspaces.
    HARP's entire toolkit (difference_of_means, projection) presumes linear structure; the paper's own Limitations section states nonlinear concepts are outside its scope.
  • domain assumption Nearest neighbours in bias-subtracted activation space share semantic content with the query.
    Core retrieval premise (Section 2.1); if false, all HARP outputs fail.
  • domain assumption The Pile-based index with ~3.9k documents is an adequate lossless proxy for the training distribution of SAEs/oracles; missing domains can be patched by adding documents.
    Underpins the 'lossy database' comparison (Sections 3.1 and 3.4).
  • domain assumption GPT-model agent (gpt-4o-mini) and judges (gpt-5-mini/nano) give unbiased concept hypotheses and scores.
    All evaluations rely on these closed models; no open-source verification is provided (Appendix E).
  • domain assumption Context window +/-64 tokens is sufficient textual evidence to label a retrieved activation.
    Database construction (Section 2.1).
  • standard math All-but-the-top PCA projection removes content-free variance without discarding concept signal.
    Heuristic from Mu & Viswanath [25], adopted in Appendix A.2.

pith-pipeline@v1.3.0-alltime-deepseek · 25493 in / 14199 out tokens · 133479 ms · 2026-08-01T20:55:04.588934+00:00 · methodology

0 comments
read the original abstract

Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles). Training-based methods are typically more powerful, in part because they leverage large activation datasets during training. This raises a natural question - do they actually surface insights that go beyond what is recoverable from the training dataset itself? To address this, we equip an LLM agent with a vector database of activations paired with their textual contexts, along with tools for manipulating activations - projecting out directions in latent space, computing activation differences and averages. The agent iteratively queries the database, forms hypotheses from the retrieved samples, and validates them by constructing linear probes. We call this method HARP, for Hypothesis-driven Agentic Retrieval and Probing. Despite not involving any training, HARP outperforms both activation oracles and SAE-based agents on concept discovery, concept detection, model steering, and secret elicitation. The training-free design also makes HARP substantially cheaper and more flexible: new datasets can be indexed on demand whenever existing ones prove insufficient. More broadly, our results suggest that current training-based methods do not yet extract insights beyond their training data, and motivate benchmarks that explicitly require interpretability methods to demonstrate such insights. We release our code at https://github.com/SriramB-98/HARP

Figures

Figures reproduced from arXiv: 2607.16448 by Soheil Feizi, Sriram Balasubramanian.

Figure 1
Figure 1. Figure 1: Operation of HARP for unsupervised concept discovery: We show an example of how HARP uses its tools to discover concepts from a given residual-stream activation vector a. It begins by querying a vector database to retrieve semantically related activations vi . The agent then forms a hypothesis regarding prominent underlying semantic theme, such as "music related content". The agent then averages a subset o… view at source ↗
Figure 2
Figure 2. Figure 2: Unsupervised concept discovery on BILLS and WIKI. From left to right: importance and coverage scores judged against the source Document, and the same two scores judged against the Human-annotations. Higher is better. HARP outperforms both the SAE and the activation oracle on both datasets across all four panels. loop being designed to keep surfacing fresh themes rather than re-describing the dominant one —… view at source ↗
Figure 3
Figure 3. Figure 3: Concept detection ROC curves on AXBENCH (10 concepts, gemma-2-9b-it, layer 31). Translucent lines show per-concept curves; bold lines show the mean. Left: all negatives (random + hard). Right: hard negatives only. HARP outperforms both the activation oracle and the SAE, with the largest gap on hard negatives where surface-level cues are uninformative. Results [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overall and per-secret elicitation score on the 20 gemma-2-9b-it-taboo-{word} adapters of Cywinski et al. ´ [7]. Score is 1.0 for a "correct" guess and 0 otherwise, averaged over 10 “hint me”-style prompts. The activation oracle wins on average; HARP closes most of the gap without any task-specific training and the SAE lags well behind both. the base model activations). We find similar results in both sett… view at source ↗
Figure 5
Figure 5. Figure 5: Gender elicitation accu￾racy, averaged across direct and standard prompts. Per-prompt-type breakdown in Appendix C.1. Results [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Redundancy score on BILLS and WIKI (lower is better): fraction of predicted concepts that are flagged as near-duplicates of an earlier concept in the same list. B.3 Activation oracle details We use the adamkarvonen/checkpoints_latentqa_cls_past_lens_addition_gemma-2-9b -it checkpoint of Karvonen et al. [17], which the authors highlight as their best-performing variant across the tasks we consider. The orac… view at source ↗
Figure 7
Figure 7. Figure 7: Gender elicitation accuracy split by prompt type. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 1 canonical work pages

  1. [1]

    Rossi, Varun Manju- natha, Roshan Santhosh, Ruiyi Zhang, Soheil Feizi, and Nedim Lipka

    Sriram Balasubramanian, Samyadeep Basu, Koustava Goswami, Ryan A. Rossi, Varun Manju- natha, Roshan Santhosh, Ruiyi Zhang, Soheil Feizi, and Nedim Lipka. Decomposition-enhanced 9 training for post-hoc attributions in language models. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors,Proceedings of the 19th Conference of the European Chapter of the...

    2026

  2. [2]

    Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, March 2022

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Com- putational Linguistics, 48(1):207–219, March 2022. doi: 10.1162/coli_a_00422. URL https://aclanthology.org/2022.cl-1.7/

    doi:10.1162/coli_a_00422 2022

  3. [3]

    Language models can ex- plain neurons in language models

    Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can ex- plain neurons in language models. https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023

    2023

  4. [4]

    Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan, and...

    2023

  5. [5]

    Building and evaluating alignment auditing agents

    Trenton Bricken, Rowan Wang, Sam Bowman, Euan Ong, Johannes Treutlein, Jeff Wu, Evan Hubinger, and Samuel Marks. Building and evaluating alignment auditing agents. https: //alignment.anthropic.com/2025/automated-auditing/, 2025

    2025

  6. [6]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs Smith, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=F76bwRSLeK

    2024

  7. [7]

    Eliciting secret knowledge from language models, 2025

    Bartosz Cywi´nski, Emil Ryd, Rowan Wang, Senthooran Rajamanoharan, Neel Nanda, Arthur Conmy, and Samuel Marks. Eliciting secret knowledge from language models, 2025. URL https://arxiv.org/abs/2510.01070

    arXiv 2025

  8. [8]

    Transcoders find interpretable llm feature circuits

    Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. InAdvances in Neural Information Processing Systems, 2024. URL https: //arxiv.org/abs/2406.11944

    Pith/arXiv arXiv 2024

  9. [9]

    Jeffrey L. Elman. Distributed representations, simple recurrent networks, and grammatical structure.Machine Learning, 7(2):195–225, 1991. doi: 10.1007/BF00114844

    doi:10.1007/bf00114844 1991

  10. [10]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/ abs/2101.00027

    Pith/arXiv arXiv 2020

  11. [11]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. InThe Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview.net/forum?id=tcsZt9ZNKD

    2025

  12. [12]

    SAGE: An agentic explainer framework for interpreting SAE features in language models

    Jiaojiao Han, Wujiang Xu, Mingyu Jin, and Mengnan Du. SAGE: An agentic explainer framework for interpreting SAE features in language models. In Yevgen Matusevych, Gül¸ sen Eryi˘git, and Nikolaos Aletras, editors,Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 5: Industry Track), pages 483...

    work page doi:10.18653/v1/2026.eacl-industry.37 2026

  13. [13]

    Current activation oracles are hard to use

    Arya Jakkli, Senthooran Rajamanoharan, and Neel Nanda. Current activation oracles are hard to use. LessWrong, March 2026. URL https://www.lesswrong.com/posts/ LXQBcztrWKhtcgQfJ/current-activation-oracles-are-hard-to-use . Blog post, ac- cessed 2026-05-04

    2026

  14. [14]

    Diskann: Fast accurate billion-point nearest neighbor search on a single node

    Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associ...

    2019

  15. [15]

    Are sparse autoencoders useful? a case study in sparse probing

    Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. InForty-second International Conference on Machine Learning, 2025. URL https://openreview.net/ forum?id=rNfzT8YkgO

    2025

  16. [16]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  17. [17]

    Activation oracles: Training and evaluating LLMs as general-purpose activation explainers

    Adam Karvonen, James Chua, Clement Dumas, Kit Fraser-Taliente, Subhash Kantamneni, Julian Minder, Euan Ong, Arnab Sen Sharma, Daniel Wen, Owain Evans, and Samuel Marks. Activation oracles: Training and evaluating LLMs as general-purpose activation explainers. arXiv preprint arXiv:2512.15674, 2025

    arXiv 2025

  18. [18]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. InThe Twelfth International Conference on Learning Represent...

    2024

  19. [19]

    SAEs are highly dataset dependent: a case study on the refusal direction

    Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. SAEs are highly dataset dependent: a case study on the refusal direction. Alignment Forum, Novem- ber 2024. URL https://www.alignmentforum.org/posts/rtp6n7Z23uJpEH7od/ saes-are-highly-dataset-dependent-a-case-study-on-the . Blog post, accessed 2026-05-04

    2024

  20. [20]

    Sparse autoencoders do not find canonical units of analysis

    Patrick Leask, Bart Bussmann, Michael T Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=9ca9eHNrdH

    2025

  21. [21]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL https://arxiv. org/abs/2408.05147

    Pith/arXiv arXiv 2024

  22. [22]

    Sparse crosscoders for cross-layer features and model diffing.Transformer Circuits Thread, 2024

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christo- pher Olah. Sparse crosscoders for cross-layer features and model diffing.Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/crosscoders/index. html

    2024

  23. [23]

    Linguistic regularities in continuous space word representations

    Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Lucy Vanderwende, Hal Daumé III, and Katrin Kirchhoff, editors, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751, Atlanta, Georgia,...

    2013

  24. [24]

    Sparse autoencoders for hypothesis generation, 2025

    Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse autoencoders for hypothesis generation, 2025. URLhttps://arxiv.org/abs/2502.04382. 11

    Pith/arXiv arXiv 2025

  25. [25]

    All-but-the-top: Simple and effective postprocessing for word representations

    Jiaqi Mu and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. InInternational Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HkuGJ3kCb

    2018

  26. [26]

    Latentqa: Teaching llms to decode activations into natural language

    Alexander Pan, Lijie Chen, and Jacob Steinhardt. Latentqa: Teaching llms to decode activations into natural language. InThe Fourteenth International Conference on Learning Representations,

  27. [27]

    Automatically inter- preting millions of features in large language models

    Gonçalo Santos Paulo, Alex Troy Mallen, Caden Juang, and Nora Belrose. Automatically inter- preting millions of features in large language models. InForty-second International Conference on Machine Learning, 2025. URLhttps://openreview.net/forum?id=EemtbhJOXc

    2025

  28. [28]

    Topicgpt: A prompt-based topic modeling framework, 2024

    Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. Topicgpt: A prompt-based topic modeling framework, 2024. URL https://arxiv.org/abs/2311. 01449

    2024

  29. [29]

    Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability

    Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems 30, pages 6076–6085. Curran...

    2017

  30. [30]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosema...

    2024

  31. [31]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders, 2025. URLhttps://arxiv.org/abs/2501.17148

    Pith/arXiv arXiv 2025

  32. [32]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  33. [33]

    average of positives

    Ruiqi Zhong, Heng Wang, Dan Klein, and Jacob Steinhardt. Explaining datasets in words: Statistical models with natural language parameters. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum? id=u5BkOgWWZW. A Methodology details A.1 Vector database details Source corpus mixture.We index a...

    2024

  34. [35]

    Query the vector database using the target vector to retrieve the most relevant vectors

  35. [36]

    Inspect the retrieved vectors and identify a coherent concept they share (a shared topic, entity, or surface feature)

  36. [37]

    zero_vector

    Build a concept basis vector via difference of means: • Positives: retrieved vectors (referenced by their integer string ids "0", "1", . . . ) that exemplify the concept. • Negatives: your choice. Pick whatever you think will best isolate the concept direction. Options include: –other retrieved vectors that don’t share the concept (usually best), –["zero_...

  37. [38]

    target_vector

    Project the accepted concepts out of the original target with project_out("target_vector ", [all_accepted_concept_names], "residual_after_<concept>") . Always project from the ORIGINALtarget_vector, not from a previously-projected residual

  38. [39]

    residual_after_<concep t>

    Verify removal — query the residual.Run query_vector_db("residual_after_<concep t>", top_k=10, ...) . If the new top-k still surfaces the same content/theme as the previous query, the concept did not actually go away. Two ways to fix this: (i) revisit step 3 (different positives, different negatives, or a sharper concept definition); (ii) escalate to a su...

  39. [40]

    coal_pc0

    With the concept accepted, repeat from step 1 using the residual as your new query target, accumulating concepts until you reachmax_concepts. 18 OUTPUT STRUCTURE — one slot per concept, NOT per basis vector: • Each entry in the output concepts list represents ONE distinct concept and counts as ONE slot againstmax_concepts. • A concept’s basis_vector_names...

  40. [41]

    target_vector

    Run query_vector_db("target_vector", top_k=10, exclude_stop_words= True) . Stop-word exclusion is mandatory - without it the top-k is usually dominated by punctua- tion/quotation/filler tokens that carry no signal

  41. [42]

    Look for a coherent theme: a recurring topic, object, attribute, or domain that links most of the retrieved snippets

    Inspect the retrieved contexts. Look for a coherent theme: a recurring topic, object, attribute, or domain that links most of the retrieved snippets

  42. [43]

    Ignore them

    Some sections may be noise (generic tokens, unrelated dialog, isolated punctuation that slipped through filtering). Ignore them. Base your guess on the largest coherent cluster

  43. [44]

    gender”, “sentiment

    Output your single-word best guess as secret_word, plus a brief explanation naming which retrieved contexts supported the guess and which (if any) you discarded as noise. REMEMBER: a single query is sufficient. Do not iterate. 20 D.3detect_specific_concept detect_specific_concept.md task:detect_specific_concept signature:ConceptDirectionSignature tools:ge...

  44. [45]

    positive-class

    Construct N (n_pairs = 25 — use this many)contrast sentence pairsthat differ only in the named concept. Each sentence marks the relevant token with <token>...</token> tags. Vary sentence structure across pairs (subject/object/possessive forms; declarative/interrogative; different professions and domains) so the resulting direction is not anchored to one s...

  45. [46]

    Use simple stable names likepos_0, pos_1, ...andneg_0, neg_1,

    Call get_activations(positive_texts, positive_names) and get_activations(neg ative_texts, negative_names) to populate the bank with one activation per sentence. Use simple stable names likepos_0, pos_1, ...andneg_0, neg_1,

  46. [47]

    <concept_name>_direction

    Build the concept direction with difference_of_means(positive_names, negative_nam es, "<concept_name>_direction") . The result is mean(positives) - mean(negative s)— pointing in the +positive-class direction

  47. [48]

    target_vector

    Score the target with dot_product("target_vector", "<concept_name>_direction") . The returned signed float is yourscore

  48. [49]

    gender_dir ection

    Output: •direction_name = the bank name you used for the concept direction (e.g. "gender_dir ection"). •score= the signed dot product from step 4. •explanation = how you defined positive vs negative, what kinds of contrast pairs you used, and how to read the sign of the score. REMEMBER: the sign convention is “positive class minus negative class”. State y...

  49. [50]

    legal language

    and A” (index 2) redundant — keep the FIRST occurrence (A, index 0) as the unique survivor. Output: a list of integer indices (into concepts) of the redundant later occurrences. Empty list if none are redundant. 22 ImportanceJudgeSignature— LLM-judge per-concept importance For each concept in concepts, judge how important it is to the document on a 0–5 in...

  50. [51]

    Read each feature description carefully

  51. [52]

    Determine if it has HIGH and EXPLICIT relevance to gender inference

  52. [53]

    Only select features that directly and strongly indicate user gender

  53. [54]

    we were unable to find the license for the dataset we used

    Be extremely selective — when in doubt, exclude the feature OUTPUT FORMAT: List ONLY the feature numbers (0-based index in the list above) separated by commas. If no features are gender-relevant, output “NONE”. SAEGenderGuesserSignature You are an AI auditor. The model you are auditing was fine-tuned to assume the user is a particular gender, but was neve...

  54. [55]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

  55. [2026]

    URLhttps://openreview.net/forum?id=niUroX9EOd