Machine Learning · arXiv:2611.07071 v1 · submitted 4 Nov 2026 · 300 references
Shaping & Monitoring Models through their Internal Representations
The paper says
“We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations.”
more of the abstract
Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 6 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A serious, broad framework paper: reliance and expression shaping work across vision and LLMs up to 32B, the L1–L5 monitoring hierarchy is genuinely useful, and the Section 6 product-code theory is derived rather than fit; the soft spots are the one-seed harm runs, validation-selected CheXpert ranks, and a real but closable gap between the toy model's use test and the real-model evidence for… This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 300 references say.
The paper says, at Section 5, Table 2, Fig. 18
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The load-bearing harmfulness results are single-seed: Figure 18 is captioned "One seed per setting," and Table 2 checkpoints are selected as the lowest JBB subject to retaining 95% of base utility and answering 80% of safe XSTest prompts, so the headline harmfulness number is selected on the very metric used to report it; the held-out ClearHarm columns mitigate this within the same seed but do not provide an error estimate.
more of objection 1
Since the abstract's cross-domain claim (b2) is anchored in part on these results, please provide at least three seeds for the decisive rows of Table 2 (Probe on policy, DoM on policy, DoM on frozen base) or explicitly condition the claim that the harmfulness comparison is a single-seed, metric-selected pilot.
The paper says, at Section 2.1.2, Fig. 3 caption, Fig. 7; Section 6
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.75):
“Expression and reliance shaping specify what the training objectiveasks for, not necessarily 4 whatactually occursinside the model.”
Pith says: Objection 2 of 3 · major
After LEACE at layer 9, the Figure 3 caption states that "the background remains encoded nonlinearly and still shapes the prediction," and Section 2.1.2 concludes that the background moved from the first moment to the second. What the experiments show is that a nonlinear probe recovers the background (L3 AUROC above 0.98, Fig. 7a) and that class covariances keep different …
more of objection 2
… orientations (Fig. 7b); the toy model supports the stronger use claim through the fresh-erasure test (accuracy 0.80 versus best possible 0.806, Fig. 25), but no equivalent behavioral test is run on Waterbirds, and the paper's own sentence b56 warns that probes specify what the training objective asks for, not what actually occurs inside the model. A causal test at block 9 (erasing the recovered second-order code and measuring WGA, or ablating the two-layer-recovered directions and observing the logit change) would close the gap; otherwise the caption and the "account for all of our observations" phrasing should be weakened to information availability at the targeted layer and recoverability by downstream layers.
The paper says, at Section 3, Figs. 13-14
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The sycophancy section carries the "scales up to 32B" and DPO parts of the headline claim, yet no seed count is reported for the 7B, 13B, and 32B runs or for the lambda sweep in Figure 13A; in contrast, the Waterbirds and Countdown sections explicitly report three seeds.
more of objection 3
Please state the number of seeds per run and per checkpoint for the sycophancy experiments, or mark them as single runs; the Pareto-frontier claim in Figure 13 is the key result of the section and should carry an error estimate or an explicit one-run caveat.
4 minor objections
The paper says, at Throughout
Pith says: minor objection 1 of 4
The text contains pervasive duplicated-word artifacts (e.g., "underspecification" repeated dozens of times in Section 1, "a a a" in the circumvention sentence, and repeated words across headings and the Discussion); this appears to be an export glitch, but it must be cleaned before publication since it renders parts of the main text unreadable.
The paper says, at Fig. 25
Pith says: minor objection 2 of 4
The central refit-period bifurcation (linear rebuild for K greater than or equal to 20 versus nonlinear code for K less than or equal to 10) is itself a one-seed toy run; given the stark separation (linear R2 at most 0.014 versus MLP R2 equal to 0.99; fresh-erasure accuracy 0.80 versus 0.652 without the unwanted feature), adding a few seeds would be inexpensive and would materially strengthen the paper's central theoretical claim.
show 2 more
The paper says, at Abstract and Section 7
Pith says: minor objection 3 of 4
Sections 1 and 6 acknowledge two scope conditions: the single-linear-direction premise (b78) and the refresh-rate condition (b122). Please carry one sentence of these conditions into the abstract, since the current abstract claim (b2) is unconditional and a practitioner applying the recipe at a fixed 100-step refresh with a multidimensional concept would be outside the demonstrated regime.
The paper says, at Section 6, Proposition 4
Pith says: minor objection 4 of 4
The structure theorem for the product code assumes positive homogeneity (no biases), and Lemma 1 inherits this assumption; the Discussion's restatement that the three mechanisms "account for all of our observations" should note explicitly that the forced form of the code is proven only for the bias-free family, with the transformer evidence being empirical.
What would settle it
In the Waterbirds setting, ablate the specific second-moment signal at layer 9: fit the covariance-direction code (the product of the unwanted concept with core features), then test whether removing that code, or the direction in the next layer that reads it, changes the model's predictions on the worst group, while a matched-size random ablation does not. This is the real-model analog of the paper's own fresh-erasure accuracy test in the toy (Fig. 25), and it distinguishes information-present from information-used. If removing the code does not change the output, the L3 monitor is detecting a representation that is recoverable but not load-bearing, and the Section 6 claim that each form of circumvention is detected 'by a matching monitor' needs re-scoping for the real-model setting.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 6 Nov 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things to know. This is a serious, broad paper: representation-based training penalties for both reliance and expression, tested across vision, medical imaging, and LLMs up to 32B, plus a five-level monitoring hierarchy that gives "monitorable" a precise meaning. The main experiments are multi-seed where it matters (Waterbirds, Countdown), the baselines are external, and the Section 6 theory is derived, not fit. Second: the paper is unusually honest about its own limits, but the harmfulness section and the CheXpert selection protocol are softer than the rest, and there is one real gap between the toy theory and the real-model evidence.
What's new: the attribution-based reliance penalty (asking the readout to be invariant along a concept direction rather than erasing the concept), the L1–L5 hierarchy, and the circumvention geometry: re-encoding, erasure, and a nonlinear product code. The product-code story is the best part. It is derived from ReLU structure (Propositions 1–4), and the toy residual model includes a fresh-erasure use test: after erasing the linear part, accuracy stays at 0.80 against a 0.806 optimum, which shows the model actually uses the code rather than merely carrying it. That is the right control, and most work in this area skips it.
Soft spots, in order. First, Section 5 harmfulness uses one seed per setting with checkpoints chosen from a grid by lowest JBB subject to constraints. The effects are large (JBB 0.70–0.71 down to 0.01, versus 0.16–0.19 for the prior probe method), but the protocol needs multi-seed replication and a fixed selection rule. Second, CheXpert ranks are selected on validation, so the "exceeds balanced training" bars should be read with that in mind. Third, the stress-test concern is fair but not fatal: in Waterbirds the L3 evidence is a nonlinear probe recovering the background after LEACE, plus linear detectors recovering within two layers. That shows the information is present and decodable downstream, but not directly that the readout uses it; the toy closes that gap, the real model does not. The paper itself says (roughly, their sentence b56) that expression and reliance shaping specify what the objective asks for, not necessarily what occurs inside the model. So "three forms of circumvention" is fully proven in the toy and demonstrated as representational phenomena in the real models; the behavioral-use step is the weak link. The single-direction assumption (b78) and the 100-step refresh are acknowledged and partially studied; a clearer statement of when they fail would strengthen the paper.
One housekeeping note: the supplied text has repeated-token glitches in headings; almost certainly an extraction artifact, but check the PDF.
Who this is for: anyone working on interpretability-guided training, shortcut learning, or monitorable alignment. It deserves a serious referee; the requested fixes are experimental, not conceptual.
The strongest objection
The central claim (Abstract, b2) is that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance, and the second load-bearing claim is in Section 6: circumvention takes three geometric forms, each detectable by a matching monitor in the five-level hierarchy. The product-code claim, which justifies the L3 detector, is the least secure link. In the toy residual model this claim is well supported: after frequent refits, accuracy after a fresh erasure stays at 0.80 versus the best possible 0.806 (Fig. 25), so the code is demonstrably used by the readout. The Waterbirds evidence for the same mechanism is weaker. The paper shows that at layer 9 a nonlinear probe recovers the background (L3 AUROC above 0.98) while linear probes fail (Fig. 7a), and that class covariances retain different orientations after LEACE equalizes the means (Fig. 7b). That demonstrates the information is present and recoverable by a probe, but not that the model's readout actually uses this second-moment signal to make its predictions. The paper itself cautions (b56): 'Expression and reliance shaping specify what the training objective asks for, not necessarily what actually occurs inside the model.' The toy model closes this gap through an erasure-use test, but the real-model setting does not. The L3 monitor is precisely the detector that claims to catch the third form of circumvention; if in practice it detects an epiphenomenon that the readout does not use, the claim that each form of circumvention is detectable 'by a matching monitor' at the level asserted in Section 6 is not yet empirically established for the real models.
The reader's weakest_assumption targeted the single-direction and refresh-rate premise. I agree that premise is the right general area, but I want to be more precise about the load-bearing point for the second claim. The Section 6 monitor-hierarchy claim rests on an empirical link: that each detectable code corresponds to a form of circumvention the model actually uses. In the toy model, the paper supplies a strong use-based test (fresh-erasure accuracy, Fig. 25). In the real-model case, the evidence for the product code is presence-based only: an L3 probe with AUROC above 0.98 and a covariance orientation difference (Fig. 7a,b). Whether that code drives the readout determines whether L3 is detecting circumvention or a detectable-but-unused artifact. The reader identified a related but distinct weak point; I partially agree but shift the locus of the concern. This does not change the verdict. The paper is already CONDITIONAL, with good reason: the harmfulness and sycophancy runs are largely single-seed, and there is best-of-grid checkpoint selection in Table 2 and Figure 18. My concern is an additional, concrete reason to keep the verdict conditional rather than to accept unconditionally, and the proposed test is feasible without re-deriving any published theorem or re-running any cited result. The paper deserves credit for the derived toy model, the clear five-level hierarchy, and the honest acknowledgment (b408) that the enumerated forms do not exhaust circumvention; those strengthen the framework but do not close the specific gap I identify.
The referee's report
If the results hold, this is a substantial contribution: it unifies feature-based rewards and attribution penalties under one constrained-learning objective (Eq. 1), provides a clean formal separation of pattern versus filter directions (Appendix C), contributes a falsifiable theory of circumvention with closed-form gradient characterizations (Propositions 1-2), and a product-code construction (Propositions 3-4, Lemma 1) whose central prediction, that frequent refitting forces the concept into a nonlinear code that a linear erasure cannot remove, with fresh-erasure accuracy staying near the Bayes optimum (Fig. 25), is verified in the toy by a genuine use test rather than by probe recoverability alone. The headline vision and RL experiments are multi-seed (Waterbirds, Countdown), the code is released, and the scale span up to 32B is real. The two places where the evidence is thinnest are the harmfulness section (one seed per setting, checkpoints selected on the reported JBB metric) and the extrapolation of the toy product-code use result to Waterbirds, where only probe evidence is offered.
Pith's reading of the paper (a machine's summary, not the authors')
Mechanistic Shaping is a training-time framework that uses a model's own internal representations as a specification language: a penalty term added to the task loss tells the model which concepts it may rely on (not the background in bird classification, not gender in resume screening) and which concepts it may express (no reward hacking, no harmful content). The paper claims this works across vision, medical imaging, and language models up to 32B parameters, across classification, SFT, DPO, and GRPO, substantially reducing targeted behaviors while preserving task performance. Because optimizing against a representation can move the representation, the paper also asks whether the shaped concept can still be monitored, and answers with a five-level monitor hierarchy and a geometric theory of circumvention: a concept can drift to a new direction, be erased from the layer, or hide in second-order statistics as a nonlinear product code, and each form leaves a signature a matching monitor can read. In the runs evaluated, shaped concepts largely remain monitorable: refitted linear probes recover them in most settings, and the original probe stays reliable whenever gradients do not flow through its score.
What carries the argument. The load-bearing objects are the constrained-learning objective $L(\theta)=R(\theta)+\lambda\,\omega_\bullet(\theta)$, whose violation $\omega_\bullet$ is read from activations; the concept direction $v$, estimated as a difference of means - the 'pattern' telling where activations move when the concept changes - rather than by a discriminative probe, the 'filter', and refreshed every 100 training steps so the penalty chases the concept as it is re-encoded; and the reliance penalty $\nu(z,v)^2$ built from an attribution (gradient, gradient times input, shuffle, noise, or deletion), which vanishes exactly when the readout is invariant along $v$. The monitoring half rests on a five-level hierarchy - reuse the original linear direction (L1), refit a linear probe (L2), refit a nonlinear probe (L3), read a different layer (L4), use a separate model (L5) - and on a product-code construction: splitting $\mathrm{ReLU}(a+b)$ into parts even and odd in $b$ yields a compressed product $m(a,b)=ab/\max(|a|,|b|)$, and Proposition 4 shows that a bias-free positively homogeneous network, after a covariance-direction erasure, can only retain the sign of the unwanted feature as $h_{\mathrm{odd}}=x_\bullet\,g(x/\|x\|)$, an approximate product with core features that linear readers cannot see but a nonlinear probe recovers.
Core claim. The paper's central claim is a double one. First, interpretability can serve as a specification language for training: writing the violation $\omega_\bullet(\theta)$ as a function of activations and adding $\lambda\,\omega_\bullet(\theta)$ to the risk $R(\theta)$ gives a single recipe that, in the paper's experiments, fixes shortcut learning under perfect confounding on Waterbirds (worst-group accuracy 93-97.7% against 51.6% for ordinary training), suppresses sycophancy learned during DPO in 7B-32B models while keeping benchmark performance, nearly eliminates reward hacking in GRPO with the effect transferring to a chess environment the policy never trained on, and reduces harmful answer rates on JailbreakBench from 0.70 to 0.01 at matched utility. Second, the fear that training against a probe makes the concept unreadable is real but structured: in all of the paper's observations, the concept either re-encodes along a new direction that a refitted linear probe reads, is erased from the layer, or moves into a nonlinear code built as an approximate product of the unwanted feature with core features - and each mechanism is detected by its matching level in a five-level monitor hierarchy, so a concept that escapes one probe is not the same as a concept that is truly gone.
The premise it rests on. The framework assumes that the unwanted concept is adequately captured by a single linear direction at one chosen layer, and that refreshing that direction every 100 training steps keeps up with the model's re-encoding of the concept.
If the paper is right (Pith's inferences, not the authors')
Representation-level penalties can replace data curation where data cannot be cleaned: perfectly confounded vision benchmarks, subliminal sycophancy in DPO, and reward hacking that emerges only during training all yield to a probe-based penalty that data filtering or loss changes do not.
If monitorability survives shaping, alignment becomes auditable: a specification of reliance or expression is verifiable after training, and a concept that escapes the original probe can be recovered by refitting rather than treated as gone.
The scoring convention - attached gradients versus detached or frozen-copy scoring - becomes a design decision with a predictable monitoring consequence: gradients through the probe degrade the original detector, while detached or frozen scoring preserves it at L1.
Circumvention is enumerable: because re-encoding, erasure, and product-code encoding each have a distinct geometric signature, a monitoring protocol can be pre-specified by level, turning 'is the concept still there?' into a precision question about which monitor is being used.
If the three-form taxonomy holds at frontier scale, monitors could be designed prospectively: L2-level refitting against re-encoding, L3-level nonlinear readers against product codes, and higher-order readers at the same layer against the degree-two component the paper explicitly leaves uncharacterized.
The refresh-cadence dependence suggests a practical tuning rule the paper does not state: measure the early-training growth rate of the concept's difference-of-means direction, then set the refresh interval below the inverse of that rate.
Because the bias-free hypothesis drives the product-code structure, a testable extension is whether giving the residual blocks biases - or using non-ReLU activations - suppresses the nonlinear code entirely, which would make the third circumvention form an artifact of a particular architecture family.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-06
The authors' main claim, as the classifier picked it out 0.88
“Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.”
How the main claim is stated: hedged possibility 0.97
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.87 · human participants 0.10 · controlled comparison 0.15 · randomized assignment 0.02 · states quantitative result 0.71 · compares with prior results 0.19 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. | contribution 0.99 | 0.67 | 0.07 | asserted 0.96 |
| Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a r\'esum\'e). | method 0.66 | 0.41 | 0.07 | asserted 1.00 |
| Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. | result 1.00 | 0.96 | 0.37 | asserted 1.00 |
| Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. | gap 0.95 | 0.15 | 0.04 | hedged possibility 0.59 |
| We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. | contribution 0.94 | 0.86 | 0.06 | asserted 0.98 |
| We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. | contribution 0.47 | 0.86 | 0.07 | asserted 0.99 |
| Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. | result 0.64 | 0.89 | 0.14 | asserted 0.91 |
| In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. | result 1.00 | 0.96 | 0.17 | asserted 1.00 |
| Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training. | interpretation 0.87 | 0.18 | 0.78 | hedged possibility 0.97 |
What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.94
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-06: 1 request, 7,161 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (53)
Showing 52 of the paper's 53 figures. Figures are the authors', reproduced from the paper.
References (300)
- [1] Abbe, E., Boix-Adser \`a , E., and Misiakiewicz, T. The merged-staircase property: A necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks. Conference on Learning Theory (COLT), 2022 on Pith
- [2] Abbe, E., Boix-Adser \`a , E., and Misiakiewicz, T. SGD learning on neural networks: Leap complexity and saddle-to-saddle dynamics. Conference on Learning Theory (COLT), 2023 on Pith
- [3] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Advances in Neural Information Processing Systems (NIPS), 2018 on Pith
- [4] Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. A r X iv e-print , 2016 on Pith
- [5] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. Concrete problems in ai safety. A r X iv e-print , 2016 on Pith
- [6] Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. Towards better understanding of gradient-based attribution methods for deep neural networks. Proceedings of the International Conference on Learning Representations (ICLR), 2018 on Pith
- [7] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems (NeurIPS), 2024 on Pith
- [8] Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
- [9] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Public Library of Science (PloS One), 2015 on Pith
- [10] Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and M \"u ller, K.-R. How to explain individual classification decisions. The Journal of Machine Learning Research (JMLR), 2010 on Pith
- [11] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. A r X iv e-print , 2022 on Pith
- [12] Bailey, L., Serrano, A., Sheshadri, A., Seleznyov, M., Taylor, J., Jenner, E., Hilton, J., Casper, S., Guestrin, C., and Emmons, S. Obfuscated activations bypass llm latent-space defenses. Proceedings of the International Conference on Learning Representations (ICLR), 2026 on Pith
- [13] Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., and Farhi, D. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. A r X iv e-print , 2025 on Pith
- [14] Barron, A. R. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 1993 on Pith
- [15] Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 2022 on Pith
- [16] Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. LEACE : Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems, 2023 on Pith
- [17] Ben Arous, G., Gheissari, R., and Jagannath, A. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research (JMLR), 2021 on Pith
- [18] Bergen, L., Bhalla, U., Baskaran, S., Loeffler, M., Sarfati, R., Gala, D., Panwar, R., Aranguri, S., Fel, T., Geiger, A., et al. Anatomy of post-training: Using interpretability to characterize data and shape the learning signal. A r X iv e-print , 2026 a on Pith
- [19] Bergen, L., Bhalla, U., Lee, A., Widawsky, B., Nasvytis, L., Watts, C., Boppana, S., Baskaran, S., Hazra, D., Byun, M., et al. Monitoring and discovering reward hacking with internal representations during llm evaluations. A r X iv e-print , 2026 b on Pith
- [20] Bernardi, S., Benna, M. K., Rigotti, M., Munuera, J., Fusi, S., and Salzman, C. D. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 2020 on Pith
show 80 more
- [21] Bhalla, U., Srinivas, S., Ghandeharioun, A., and Lakkaraju, H. Towards unifying interpretability and control: Evaluation via intervention. A r X iv e-print , 2024 on Pith
- [22] Bhalla, U., Fel, T., Rager, C., Feucht, S., Haklay, T., Wurgaft, D., Boppana, S., Kowal, M., Shyam, V., Merullo, J., et al. Do sparse autoencoders capture concept manifolds? A r X iv e-print , 2026 on Pith
- [23] Bishop, C. M. Training with noise is equivalent to tikhonov regularization. Neural computation, 1995 on Pith
- [24] Blank, C., Ying, Z., Potts, C., Hase, P., and Huang, J. Sycophantic agreement transfers with neutral data via contrastive preference optimization. A r X iv e-print , 2026 on Pith
- [25] Boyd, A., Tinsley, P., Bowyer, K. W., and Czajka, A. Training better deep learning models using human saliency. A r X iv e-print , 2024 on Pith
- [26] Breiman, L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 2001 on Pith
- [27] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Car... on Pith
- [28] Brutzkus, A. and Globerson, A. Why do larger models generalize better? A theoretical perspective via the XOR problem. Proceedings of the International Conference on Machine Learning (ICML), 2019 on Pith
- [29] Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. A r X iv e-print , 2024 on Pith
- [30] Carlini, E., Catalisano, M. V., and Geramita, A. V. The solution to the W aring problem for monomials and the sum of coprime monomials. Journal of Algebra, 2012 on Pith
- [31] Casademunt, H., Juang, C., Karvonen, A., Marks, S., Rajamanoharan, S., and Nanda, N. Steering out-of-distribution generalization with concept ablation fine-tuning. A r X iv e-print , 2025 on Pith
- [32] Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and Bloom, J. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. A r X iv e-print , 2024 on Pith
- [33] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2018 on Pith
- [34] Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
- [35] Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey, J. Persona vectors: Monitoring and controlling character traits in language models. A r X iv e-print , 2025 on Pith
- [36] Cho, Y. and Saul, L. K. Kernel methods for deep learning. Advances in Neural Information Processing Systems (NeurIPS), 2009 on Pith
- [37] Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. Subliminal learning: Language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805, 2025 on PithPith's reviewarXiv
- [38] Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
- [39] Covert, I., Lundberg, S., and Lee, S.-I. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 2021 on Pith
- [40] Csord \'a s, R., Potts, C., Manning, C. D., and Geiger, A. Recurrent neural networks learn to store and generate sequences using non-linear representations. BlackboxNLP Workshop, 2024 on Pith
- [41] Cundy, C. and Gleave, A. Preference learning with lie detectors can induce honesty or evasion. In Advances in Neural Information Processing Systems, volume 38, 2025 on Pith
- [42] Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. A r X iv e-print , 2023 on Pith
- [43] Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 1989 on Pith
- [44] Damian, A., Lee, J. D., and Soltanolkotabi, M. Neural networks can learn representations with gradient descent. Conference on Learning Theory (COLT), 2022 on Pith
- [45] D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 2022 on Pith
- [46] Daniely, A., Frostig, R., and Singer, Y. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. Advances in Neural Information Processing Systems (NeurIPS), 2016 on Pith
- [47] Deeb, A. and Roger, F. Do unlearning methods remove information from language model weights? A r X iv e-print , 2024 on Pith
- [48] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database . Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009 on Pith
- [49] Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. A r X iv e-print , 2023 on Pith
- [50] Dreyer, M., Pahde, F., Anders, C. J., Samek, W., and Lapuschkin, S. From hope to safety: Unlearning biases of deep models via gradient penalization in latent space. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024 on Pith
- [51] Drucker, H. and Le Cun, Y. Improving generalization performance using double backpropagation. IEEE transactions on neural networks, 1992 on Pith
- [52] Durbin, R. and Rumelhart, D. E. Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation, 1989 on Pith
- [53] Elad, M., Milanfar, P., and Rubinstein, R. Analysis versus synthesis in signal priors. Inverse problems, 2007 on Pith
- [54] Elazar, Y. and Goldberg, Y. Adversarial removal of demographic attributes from text data. Proceedings of the 2018 conference on empirical methods in natural language processing, 2018 on Pith
- [55] Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 2021 on Pith
- [56] Eldan, R. and Shamir, O. The power of depth for feedforward neural networks. Conference on Learning Theory (COLT), 2016 on Pith
- [57] Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimensionally linear. Proceedings of the International Conference on Learning Representations (ICLR), 2025 on Pith
- [58] Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., and Lee, S.-I. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 2021 on Pith
- [59] Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, M., Lambert, N., Walsh, P., Dasigi, P., Berry, R., Malik, S., Shah, S., G... on Pith
- [60] Fel, T., Cadene, R., Chalvidal, M., Cord, M., Vigouroux, D., and Serre, T. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in Neural Information Processing Systems (NeurIPS), 2021 on Pith
- [61] Fel, T., Felipe, I., Linsley, D., and Serre, T. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
- [62] Fel, T., Boissin, T., Boutin, V., Picard, A., Novello, P., Colin, J., Linsley, D., Rousseau, T., Cadène, R., Gardes, L., and Serre, T. Unlocking feature visualization for deeper networks with magnitude constrained optimization. Advances in Neural Information Processing Systems... on Pith
- [63] Fel, T., Boutin, V., Moayeri, M., Cadene, R., Bethune, L., Chalvidal, M., and Serre, T. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems (NeurIPS), 2023 b on Pith
- [64] Fel, T., Ducoffe, M., Vigouroux, D., Cadene, R., Capelle, M., Nicodeme, C., and Serre, T. Don't lie to me! robust and efficient explainability with verified perturbation analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023 c on Pith
- [65] Fel, T., Bethune, L., Lampinen, A. K., Serre, T., and Hermann, K. Understanding visual feature reliance through the lens of complexity. Advances in Neural Information Processing Systems (NeurIPS), 2024 on Pith
- [66] Fel, T., Kowal, M., Jacobs, M., Hazra, D., Bhalla, U., Sharkey, L., Bushnaq, L., Grant, S., Haklay, T., Icard, T., et al. Structuring sparsity: Block-sparse featurizers capture visual concept manifolds. A r X iv e-print , 2026 on Pith
- [67] Fisher, A., Rudin, C., and Dominici, F. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of machine learning research, 2019 on Pith
- [68] Fong, R. C. and Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017 on Pith
- [69] Fraser-Taliente, K., Kantamneni, S., Ong, E., Mossing, D., Lu, C., Bogdan, P. C., Ameisen, E., Chen, J., Kishylau, D., Pearce, A., Tarng, J., Wu, A., Wu, J., Zhang, Y., Ziegler, D. M., Hubinger, E., Batson, J., Lindsey, J., Zimmerman, S., and Marks, S. Natural language autoenc... on Pith
- [70] Frei, S., Chatterji, N. S., and Bartlett, P. L. Random feature amplification: Feature learning and generalization in neural networks. The Journal of Machine Learning Research (JMLR), 2023 on Pith
- [71] Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2021 on Pith
- [72] Geiger, A., Wu, Z., Lu, H., Rozner, J., Kreiss, E., Icard, T., Goodman, N., and Potts, C. Inducing causal structure for interpretable neural networks. Proceedings of the International Conference on Machine Learning (ICML), 2022 on Pith
- [73] Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. Finding alignments between interpretable causal variables and distributed neural representations. Causal Learning and Reasoning, 2024 on Pith
- [74] Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020 on Pith
- [75] Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P. W. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025 on Pith
- [76] Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 2021 on Pith
- [77] Giles, C. L. and Maxwell, T. Learning, invariance, and generalization in high-order neural networks. Applied Optics, 1987 on Pith
- [78] Glasgow, M. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
- [79] Gold, H. They thought they were making technological breakthroughs. It was an AI -sparked delusion. CNN, September 2025. URL https://www.cnn.com/2025/09/05/tech/ai-sparked-delusion-chatgpt on Pith
- [80] Goldowsky-Dill, N., Chughtai, B., Heimersheim, S., and Hobbhahn, M. Detecting strategic deception with linear probes. PMLR, 2025 on Pith
- [81] Goodhart, C. A. E. Problems of monetary management: The U.K. experience. In Papers in Monetary Economics. Reserve Bank of Australia, Sydney, 1975 on Pith
- [82] Gorton, L. What would non-linear features actually look like?, 2026 on Pith
- [83] Grant, S., Gillioz, V., Ward, J., and McGrath, T. Shifting the gradient: Understanding how defensive training methods protect language model integrity. A r X iv e-print , 2026 on Pith
- [84] Gupta, A., Saini, S., and Narayanan, P. Concept distillation: leveraging human-centered explanations for model improvement. Advances in Neural Information Processing Systems, 2023 on Pith
- [85] Gupta, R. and Jenner, E. RL-Obfuscation : Can language models learn to evade latent-space monitors? A r X iv e-print , 2025 on Pith
- [86] Gurnee, W. and Tegmark, M. Language models represent space and time. In Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
- [87] Gurnee, W., Ameisen, E., Kauvar, I., Tarng, J., Pearce, A., Olah, C., and Batson, J. When models manipulate manifolds: The geometry of a counting task. A r X iv e-print , 2026 on Pith
- [88] Han, T., Srinivas, S., and Lakkaraju, H. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
- [89] Haufe, S., Meinecke, F., G \"o rgen, K., D \"a hne, S., Haynes, J.-D., Blankertz, B., and Bie mann, F. On the interpretation of weight vectors of linear models in multivariate neuroimaging. NeuroImage, 87: 0 96--110, 2014. doi:10.1016/j.neuroimage.2013.10.067 on Pithdoi:10.1016/j.neuroimage.2013.10.067
- [90] Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety. A r X iv e-print , 2021 on Pith
- [91] Hermann, K. and Lampinen, A. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 2020 on Pith
- [92] Hermann, K. L., Mobahi, H., Fel, T., and Mozer, M. C. On the foundations of shortcut learning. Proceedings of the International Conference on Learning Representations (ICLR), 2023 on Pith
- [93] Hewitt, J. Distinguishing forms from estimators in interp, or, why steer by difference-of-means? https://www.cs.columbia.edu/ johnhew/forms-from-estimators.html, 2026 on Pith
- [94] Hindupur, S. S. R., Orgad, H., Fel, T., and Ba, D. When models don't manipulate manifolds: The geometry of a comparison task. A r X iv e-print , 2026 on Pith
- [95] Hollinsworth, O. J., Dombrowski, A.-K., Adam-Day, S., Gleave, A., and Cundy, C. Scaling trends for lie detector oversight in preference learning. In Trustworthy AI for Good (AI4GOOD) Workshop @ ICML 2026, 2026 on Pith
- [96] Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural Networks, 1989 on Pith
- [97] Howard, J. Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019. URL https://github.com/fastai/imagenette on Pith
- [98] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Inte... on Pith
- [99] Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
- [100] Jang, E., Falck, D., Braun, J., Kirch, N., Menon, A., Moodley, P., Emmons, S., Zimmermann, R. S., and Lindner, D. Exploration hacking: Can llms learn to resist rl training? A r X iv e-print , 2026 on Pith
The first 100 of 300 references are listed here. All of them are in this paper's citation record.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment