Pith. Sign in

REVIEW

Efficient Auto-Interpretability of AI Models in Biology

PaperPDF from arXiv

data files

ReviewThe objections and verdict, as JSON Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Piotr Jedryszek, Oliver M. Crook

Quantitative Methods · Submitted 27 Aug 2026 · 15 references

Pith's verdict: Cross-seed stability is a sensible and cost-effective prefilter for SAE interpretability in biology, honestly evaluated with real cost accounting and external validation, though the small sample and incomplete calibration of the LLM judge limit the precision of the headline savings.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on September 9, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 3 major objections · 4 minor · Ask about this paper · The compiled record · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID2608.27754 (version 1)
Pith IDDQNAZIZ5, Pith's permanent record of this paper and its review
CitationOliver M. Crook, Piotr Jedryszek. "Efficient Auto-Interpretability of AI Models in Biology." arXiv:2608.27754 (2026). https://pith.science/paper/DQNAZIZ5
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Cross-seed dictionary stability predicted interpretability of SAE latents in Boltz-1, reducing LLM evaluation costs by over fivefold.

The reading

The paper argues that cross-seed dictionary stability—the degree to which a sparse autoencoder latent reproduces across independently trained dictionaries—serves as a reliable, LLM-free predictor of whether that latent is interpretable. Deployed on the Boltz-1 Pairformer trunk, a stability threshold of 0.5 cuts the number of latent evaluations needed per interpretable latent by 4.4× and the measured dollar cost by 5.2×, while recovering about 53% of interpretable latents. This matters because auto-interpretability of biological AI models is expensive when every latent must be judged by a language model; stability filtering shifts most of that cost to a cheap computational pre-pass. The paper also provides a calibrated intruder-detection task for measuring latent interpretability and a method to validate auto-generated descriptions through falsifiable predictions against curated annotations.

What carries the argument

Cross-seed dictionary stability: the average absolute cosine similarity between a latent's decoder direction from seed 1 and its best-matching direction in seeds 2 and 3, assigned via Hungarian matching. It serves as an unsupervised proxy for interpretability that requires no LLM calls, no labels, and no forward passes of the base model.

Extended reading notes

Core claim

Cross-seed dictionary stability, computed by matching decoder directions across three independently trained TopK sparse autoencoders via Hungarian assignment, predicts interpretability both under a supervised concept-recovery proxy (1.97× lift in the discovery band) and in full LLM deployment (4.4× fewer evaluations per interpretable latent, 5.2× lower cost, 53% recall). The stability metric requires no LLM calls whatsoever. The paper also establishes that the standard intruder-detection harness suffers from a severe token-budget artifact (16 tokens yields chance performance on positive controls; 512 tokens recovers 0.97 accuracy) and that auto-generated descriptions can be validated by deriving falsifiable predictions about residue enrichment against SwissProt annotations.

Load-bearing premise

That the calibrated intruder-detection task (five 33-residue windows, Claude judge with 512 output tokens) correctly measures whether a latent captures a coherent biological pattern, rather than some confound like LLM fluency or window-marking artifacts.

Editorial extensions

If this is right

  • Stability filtering can reduce the cost of discovering interpretable SAE latents in large protein models by roughly fivefold, making auto-interpretability accessible to groups with limited LLM budgets.
  • The finding that stability selects for structure-related features (residue-in-position detectors) and against functional-site and compositional-region features suggests a potential tension: the most reproducible latents may not correspond to the most biologically interesting ones.
  • The calibrated intruder-detection task (512-token output allowance) provides a reusable benchmark for evaluating latent interpretability in biological models, and the demonstration of a token-budget artifact warns against adopting standard LLM evaluation harnesses without positive controls.
  • The decoupled describe-and-validate pipeline allows auto-generated labels to be tested against curated annotations, separating the part of an LLM description that can be trusted (residue identity) from the part that is less reliable (structural or functional context).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the stability-interpretability correlation holds across other biological foundation models (e.g., ESMFold, AlphaFold3), cross-seed stability could become a standard diagnostic for SAE training quality and a pre-filter for any automated interpretability pipeline in computational biology.
  • The bias toward structure-related features may reflect the fact that Boltz-1 is a structure prediction model, so its internal representations are naturally organized around geometry; functional features might be stored as transient correlations rather than stable computational elements. This hypothesis could be tested by applying the same pipeline to a function-focused model (e.g., a protein langu
  • Because the paper reports that the stability filter saved 5.2× in dollar cost but the saving is not guaranteed at small scale (training additional SAEs may cost more than judging all latents), practitioners should weigh the overhead of multi-seed training against the expected evaluation budget before adopting the filter.
Original abstract

Sparse autoencoders (SAEs), and other interpretability methods could turn AI models in Biology and other fields into engines of scientific discovery by explaining the superhuman capabilities of those models. However, a latent is only useful if we know three things: whether it is coherent, whether it can be described, and whether that description has predictive power. These questions are routinely conflated. We assemble them into a single pipeline and report the practical innovations each stage required. First, cross-seed dictionary stability prioritises which latents are worth spending resources to investigate. Second, an intruder-detection task asks whether a latents activating examples share a recognizable pattern. Third, a separate pass proposes a candidate biological description which we convert into falsifiable predictions which can be tested in silico. Deployed on the Boltz-1 Pairformer trunk, stability prioritisation finds interpretable latents using about 4.4 times fewer latent evaluations each, and at 5.2 times lower measured cost, while recovering over half of them, and the external check shows the surfaced motifs are significantly enriched for their claimed annotations. The results also suggest a possible tension: the cross- seed stability might be selecting for some types of features, like structure-related ones, much more than others, such as function-related features.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,906 input tokens.

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.94 · human participants 0.04 · controlled comparison 0.27 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.51 · mentions code or data availability 0.04.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; .

Sparse autoencoders (SAEs), and other interpretability methods could turn AI models in Biology and other fields into engines of scientific discovery by explaining the superhuman capabilities of those models. However, a latent is only useful if we know three things: whether it is coherent, whether it can be described, and whether that description has predictive power. These questions are routinely conflated. We assemble them into a single pipeline and report the practical innovations each stage required. First, cross-seed dictionary stability prioritises which latents are worth spending resources to investigate. Second, an intruder-detection task asks whether a latents activating examples share a recognizable pattern. Third, a separate pass proposes a candidate biological description which we convert into falsifiable predictions which can be tested in silico. Deployed on the Boltz-1 Pairformer trunk, stability prioritisation finds interpretable latents using about 4.4 times fewer latent evaluations each, and at 5.2 times lower measured cost, while recovering over half of them, and the external check shows the surfaced motifs are significantly enriched for their claimed annotations. The results also suggest a possible tension: the cross- seed stability might be selecting for some types of features, like structure-related ones, much more than others, such as function-related features.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#sentencerolethe authors' own workstates the main conclusionstrength
s0 Sparse autoencoders (SAEs), and other interpretability methods could turn AI models in Biology and other fields into engines of scientific discovery by explaining the superhuman capabilities of those models. background 1.00 0.05 0.08 hedged possibility 1.00
s1 However, a latent is only useful if we know three things: whether it is coherent, whether it can be described, and whether that description has predictive power. background 0.63 0.08 0.07 asserted 1.00
s2 These questions are routinely conflated. gap 0.53 0.11 0.03 asserted 1.00
s3 We assemble them into a single pipeline and report the practical innovations each stage required. contribution 0.92 0.90 0.07 asserted 0.97
s4 First, cross-seed dictionary stability prioritises which latents are worth spending resources to investigate. method 0.98 0.79 0.06 asserted 1.00
s5 Second, an intruder-detection task asks whether a latents activating examples share a recognizable pattern. method 0.99 0.74 0.03 asserted 0.93
s6 Third, a separate pass proposes a candidate biological description which we convert into falsifiable predictions which can be tested in silico. method 0.99 0.90 0.07 asserted 0.99
s7 Deployed on the Boltz-1 Pairformer trunk, stability prioritisation finds interpretable latents using about 4.4 times fewer latent evaluations each, and at 5.2 times lower measured cost, while recovering over half of them, and the external check shows the surfaced motifs are significantly enriched for their claimed annotations. result 1.00 0.96 0.42 asserted 1.00
s8 The results also suggest a possible tension: the cross- seed stability might be selecting for some types of features, like structure-related ones, much more than others, such as function-related features. interpretation 0.89 0.76 0.14 hedged possibility 0.99

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from simulation or computation 0.77.

What was examined, and what was not

sources read
  • arXiv listing abstract v1
not examined
  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers
levels run
  • P abstract
not run
  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures
Figure 1. Cross-seed stability as an LLM-free interpretability proxy. (A) In the discovery band (activation-frequency percentiles 20 to 60), a stability ≥ 0.5 filter nearly doubles the supervised interpretable rate (1.97×, n = 4,881); the LLM deployment points in the same direction (
Figure 2. Low-frequency discovery landscape. (A) Distribution of intruder accuracy across the panel, stacked by stability band: judging only the 12% of latents that clear stability ≥ 0.5 costs about 4.4× fewer latent evaluations per interpretable latent and recovers 53% (nine of 17) of them (p = 8 × 10−6 ). (B) Individual discovered features by accuracy and stability. The small set of complex functional motifs (catalytic residues, metal-coordination clusters) happens to fall at low stability, a pattern we examine and hedge appropriately in Section 3.3. 3.2 Auto-interp descriptions as…
Figure 3. Sensitivity to the stability cut and the interpretability bar. (A) Precision, recall and the fraction of the panel judged, as a function of the stability threshold, at the 0.5 interpretability bar. The shaded band marks the 0.35 to 0.70 range quoted in the text and the dotted line marks the 0.5 cut used. (B) Latent evaluations per interpretable latent found, at interpretability bars of 0.4, 0.5 and 0.6. The saving over the unfiltered panel holds under all three. B Harness Calibration Ladder and Robustness At the standard restricted generation budget, the judge scores our 20…
Figure 4. Reasoning-budget ablation ladder. Identical positive-control latents across prompt architectures: an immediate-answer restriction (16 tokens) scores 0.17, and raising the output-token allowance to 512 tokens recovers 0.97. Across the three calibrated variants shown, alternative wording (0.96) and metadata inclusion (0.96) add negligible variance (∆ < 0.01). A fourth, independently written paraphrase reaches 0.85 (Section B). The recovery is model-agnostic (Claude Sonnet 4.6: 0.96; Haiku 4.5: 0.97; Opus 4.8: 0.97; all 20/20) and the broken 16-token gate is itself model-dependent; it…
Figure 5. Convergent validity against ground truth. (A) Calibrated intruder accuracy tracks an independent, label-only amino-acid-precision axis (ρ = 0.52); a 2 × 2 split at the precision threshold gives OR = 140, with only one of 111 low-precision latents judged interpretable. This precision analysis covers the 136 of 141 sweep latents that fire on at least 20 residues; the five too-sparse latents excluded here include two of the 17 interpretable finds. (B) Firing enrichment for the metal￾coordination and catalytic motifs against curated SwissProt residue annotations, conditional on residue…
Figure 6. Subspace-level stability, and its positive-control ladder. For each feature, the co-firing decoder group is located independently in all three seeds and compared by mean cos2 of principal angles (1 = identical subspace). (A) The five metal-coordination and catalytic motifs (0.13) score below uninterpretable latents (0.23) and far below amino-acid controls (0.36) run through the identical pipeline. (B) Positive-control ladder: a planted rotated subspace (1.00) and a genuinely stable atom group (0.85) confirm the metric has range and the motifs sit near the floor, not near either. 12

Figures reproduced from arXiv:2608.27754 by the authors.

References
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 3 sentences without a confident label

Uses the method or tool of 2 papers
Uses data or values from 1 paper
  • We analyze SAEs (TopK, $k = 256$, dictionary size $2048$, input dimension $384$) trained on Boltz-1 Pairformer trunk activations (layer 20, recycle 1) https://huggingface.co/collections/evolve-away/boltz-saespublicly available Hugging Face collection of the SAEs and made available by a previous study [ref] .

    Probing and steering biology across Boltz-1s trunk-diffusion boundary · Methods · 0.89

Cites as support 1 paper

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.