Pith. sign in

REVIEW 4 cited by

Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.13117 v2 pith:4Z2HWX4F submitted 2024-11-20 cs.LG

Charles O'Neill , Alim Gumran , David Klindt This is my paper
classification cs.LG
keywords sparseinferencesaesaccurateautoencoderscomputeencodersencoding
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A recent line of work has shown promise in using sparse autoencoders (SAEs) to uncover interpretable features in neural network representations. However, the simple linear-nonlinear encoding mechanism in SAEs limits their ability to perform accurate sparse inference. Using compressed sensing theory, we prove that an SAE encoder is inherently insufficient for accurate sparse inference, even in solvable cases. We then decouple encoding and decoding processes to empirically explore conditions where more sophisticated sparse inference methods outperform traditional SAE encoders. Our results reveal substantial performance gains with minimal compute increases in correct inference of sparse codes. We demonstrate this generalises to SAEs applied to large language models, where more expressive encoders achieve greater interpretability. This work opens new avenues for understanding neural network representations and analysing large language model activations.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

    cs.LG 2026-05 conditional novelty 7.0 of 10

    SA-GSAE with Bi-Jump-ReLU enables one latent to encode both polarities of anticorrelated features, Pareto-dominating or matching full-width gated SAEs while reducing dead latents by up to 500x on some LLM hookpoints.

  2. Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

    cs.LG 2026-05 conditional novelty 6.0 of 10

    A half-width sign-aware gated sparse autoencoder matches full-width Gated SAE reconstruction on six LLM hookpoints while cutting dead features by 0.35–0.82 absolute at matched sparsity.

  3. The Rate-Distortion-Polysemanticity Tradeoff in SAEs

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    SAEs exhibit a rate-distortion-polysemanticity tradeoff where monosemanticity increases rate and distortion, with optimal polysemanticity set by feature co-occurrence probabilities in the data.

  4. Towards Atoms of Large Language Models

    cs.CL 2025-09 reject novelty 4.0 of 10

    The authors define 'atoms' as sparse, near-orthogonal directions in LLM representations under a data-adaptive inner product, and show threshold-activated sparse autoencoders can recover them with about 99.9% reconstru...

Pith tools