RSAE replaces fixed SAE encoder activations (ReLU, JumpReLU, TopK) with trainable rational functions, initialized from baselines and fine-tuned to improve reconstruction and downstream metrics on language-model residual streams.
Padé activation units: End-to-end learning of flexible activation functions in deep networks
3 Pith papers cite this work, alongside 16 external citations. Polarity classification is still indexing.
3
Pith papers citing it
16
external citations · external index
representative citing papers
A replica path-integral calculation yields analytic large-N formulas for covariance statistics and participation dimension in nonlinear recurrent neural networks with quenched Gaussian disorder, confirmed by simulation.
Fine-tuned transformers with multi-task learning recover substantial wording-derived signal for item difficulty at small sample sizes typical in applied testing.
citing papers explorer
-
Rational Sparse Autoencoder
RSAE replaces fixed SAE encoder activations (ReLU, JumpReLU, TopK) with trainable rational functions, initialized from baselines and fine-tuned to improve reconstruction and downstream metrics on language-model residual streams.
-
Statistics of correlations in nonlinear recurrent neural networks
A replica path-integral calculation yields analytic large-N formulas for covariance statistics and participation dimension in nonlinear recurrent neural networks with quenched Gaussian disorder, confirmed by simulation.
-
Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning
Fine-tuned transformers with multi-task learning recover substantial wording-derived signal for item difficulty at small sample sizes typical in applied testing.