Pith.

Machine Learning · arXiv:2611.07071 v1 · submitted 4 Nov 2026 · 300 references

Shaping & Monitoring Models through their Internal Representations

Thomas Fel, Matthew Kowal, Matt Feiszli, Satchel Grant, Andrew Lee, Chris Earls, Fatih Dinc, Vasudev Shyam

show 18 more authors

Usha Bhalla, Leon Bergen, Siddharth Boppana, Ching Fang, Michael Byun, Tristan Gallent, Nicholas Wang, Tal Haklay, Dron Hazra, Neehar Kondapaneni, Connor Watts, Ryan Panwar, Thomas Dooms, Jack Merullo, Atticus Geiger, Ekdeep Singh Lubana, Thomas McGrath, Owen Lewis

The paper says

“We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations.”

more of the abstract

Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 6 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A serious, broad framework paper: reliance and expression shaping work across vision and LLMs up to 32B, the L1–L5 monitoring hierarchy is genuinely useful, and the Section 6 product-code theory is derived rather than fit; the soft spots are the one-seed harm runs, validation-selected CheXpert ranks, and a real but closable gap between the toy model's use test and the real-model evidence for… This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 300 references say.

The paper says, at Section 5, Table 2, Fig. 18

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The load-bearing harmfulness results are single-seed: Figure 18 is captioned "One seed per setting," and Table 2 checkpoints are selected as the lowest JBB subject to retaining 95% of base utility and answering 80% of safe XSTest prompts, so the headline harmfulness number is selected on the very metric used to report it; the held-out ClearHarm columns mitigate this within the same seed but do not provide an error estimate.

more of objection 1

Since the abstract's cross-domain claim (b2) is anchored in part on these results, please provide at least three seeds for the decisive rows of Table 2 (Probe on policy, DoM on policy, DoM on frozen base) or explicitly condition the claim that the harmfulness comparison is a single-seed, metric-selected pilot.

The paper says, at Section 2.1.2, Fig. 3 caption, Fig. 7; Section 6

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.75):

“Expression and reliance shaping specify what the training objectiveasks for, not necessarily 4 whatactually occursinside the model.”

Pith says: Objection 2 of 3 · major

After LEACE at layer 9, the Figure 3 caption states that "the background remains encoded nonlinearly and still shapes the prediction," and Section 2.1.2 concludes that the background moved from the first moment to the second. What the experiments show is that a nonlinear probe recovers the background (L3 AUROC above 0.98, Fig. 7a) and that class covariances keep different …

more of objection 2

… orientations (Fig. 7b); the toy model supports the stronger use claim through the fresh-erasure test (accuracy 0.80 versus best possible 0.806, Fig. 25), but no equivalent behavioral test is run on Waterbirds, and the paper's own sentence b56 warns that probes specify what the training objective asks for, not what actually occurs inside the model. A causal test at block 9 (erasing the recovered second-order code and measuring WGA, or ablating the two-layer-recovered directions and observing the logit change) would close the gap; otherwise the caption and the "account for all of our observations" phrasing should be weakened to information availability at the targeted layer and recoverability by downstream layers.

The paper says, at Section 3, Figs. 13-14

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The sycophancy section carries the "scales up to 32B" and DPO parts of the headline claim, yet no seed count is reported for the 7B, 13B, and 32B runs or for the lambda sweep in Figure 13A; in contrast, the Waterbirds and Countdown sections explicitly report three seeds.

more of objection 3

Please state the number of seeds per run and per checkpoint for the sycophancy experiments, or mark them as single runs; the Pareto-frontier claim in Figure 13 is the key result of the section and should carry an error estimate or an explicit one-run caveat.

4 minor objections

The paper says, at Throughout

Pith says: minor objection 1 of 4

The text contains pervasive duplicated-word artifacts (e.g., "underspecification" repeated dozens of times in Section 1, "a a a" in the circumvention sentence, and repeated words across headings and the Discussion); this appears to be an export glitch, but it must be cleaned before publication since it renders parts of the main text unreadable.

The paper says, at Fig. 25

Pith says: minor objection 2 of 4

The central refit-period bifurcation (linear rebuild for K greater than or equal to 20 versus nonlinear code for K less than or equal to 10) is itself a one-seed toy run; given the stark separation (linear R2 at most 0.014 versus MLP R2 equal to 0.99; fresh-erasure accuracy 0.80 versus 0.652 without the unwanted feature), adding a few seeds would be inexpensive and would materially strengthen the paper's central theoretical claim.

show 2 more

The paper says, at Abstract and Section 7

Pith says: minor objection 3 of 4

Sections 1 and 6 acknowledge two scope conditions: the single-linear-direction premise (b78) and the refresh-rate condition (b122). Please carry one sentence of these conditions into the abstract, since the current abstract claim (b2) is unconditional and a practitioner applying the recipe at a fixed 100-step refresh with a multidimensional concept would be outside the demonstrated regime.

The paper says, at Section 6, Proposition 4

Pith says: minor objection 4 of 4

The structure theorem for the product code assumes positive homogeneity (no biases), and Lemma 1 inherits this assumption; the Discussion's restatement that the three mechanisms "account for all of our observations" should note explicitly that the forced form of the code is proven only for the bias-free family, with the transformer evidence being empirical.

What would settle it

In the Waterbirds setting, ablate the specific second-moment signal at layer 9: fit the covariance-direction code (the product of the unwanted concept with core features), then test whether removing that code, or the direction in the next layer that reads it, changes the model's predictions on the worst group, while a matched-size random ablation does not. This is the real-model analog of the paper's own fresh-erasure accuracy test in the toy (Fig. 25), and it distinguishes information-present from information-used. If removing the code does not change the output, the L3 monitor is detecting a representation that is recoverable but not load-bearing, and the Section 6 claim that each form of circumvention is detected 'by a matching monitor' needs re-scoping for the real-model setting.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 6 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 20 s · 61,220 tokens read, 2,759 written
  3. Editor wrote the verdict above 42 s · 62,378 tokens read, 6,065 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 1 s · 61,720 tokens read, 8,950 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Two things to know. This is a serious, broad paper: representation-based training penalties for both reliance and expression, tested across vision, medical imaging, and LLMs up to 32B, plus a five-level monitoring hierarchy that gives "monitorable" a precise meaning. The main experiments are multi-seed where it matters (Waterbirds, Countdown), the baselines are external, and the Section 6 theory is derived, not fit. Second: the paper is unusually honest about its own limits, but the harmfulness section and the CheXpert selection protocol are softer than the rest, and there is one real gap between the toy theory and the real-model evidence.

What's new: the attribution-based reliance penalty (asking the readout to be invariant along a concept direction rather than erasing the concept), the L1–L5 hierarchy, and the circumvention geometry: re-encoding, erasure, and a nonlinear product code. The product-code story is the best part. It is derived from ReLU structure (Propositions 1–4), and the toy residual model includes a fresh-erasure use test: after erasing the linear part, accuracy stays at 0.80 against a 0.806 optimum, which shows the model actually uses the code rather than merely carrying it. That is the right control, and most work in this area skips it.

Soft spots, in order. First, Section 5 harmfulness uses one seed per setting with checkpoints chosen from a grid by lowest JBB subject to constraints. The effects are large (JBB 0.70–0.71 down to 0.01, versus 0.16–0.19 for the prior probe method), but the protocol needs multi-seed replication and a fixed selection rule. Second, CheXpert ranks are selected on validation, so the "exceeds balanced training" bars should be read with that in mind. Third, the stress-test concern is fair but not fatal: in Waterbirds the L3 evidence is a nonlinear probe recovering the background after LEACE, plus linear detectors recovering within two layers. That shows the information is present and decodable downstream, but not directly that the readout uses it; the toy closes that gap, the real model does not. The paper itself says (roughly, their sentence b56) that expression and reliance shaping specify what the objective asks for, not necessarily what occurs inside the model. So "three forms of circumvention" is fully proven in the toy and demonstrated as representational phenomena in the real models; the behavioral-use step is the weak link. The single-direction assumption (b78) and the 100-step refresh are acknowledged and partially studied; a clearer statement of when they fail would strengthen the paper.

One housekeeping note: the supplied text has repeated-token glitches in headings; almost certainly an extraction artifact, but check the PDF.

Who this is for: anyone working on interpretability-guided training, shortcut learning, or monitorable alignment. It deserves a serious referee; the requested fixes are experimental, not conceptual.

The strongest objection

The central claim (Abstract, b2) is that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance, and the second load-bearing claim is in Section 6: circumvention takes three geometric forms, each detectable by a matching monitor in the five-level hierarchy. The product-code claim, which justifies the L3 detector, is the least secure link. In the toy residual model this claim is well supported: after frequent refits, accuracy after a fresh erasure stays at 0.80 versus the best possible 0.806 (Fig. 25), so the code is demonstrably used by the readout. The Waterbirds evidence for the same mechanism is weaker. The paper shows that at layer 9 a nonlinear probe recovers the background (L3 AUROC above 0.98) while linear probes fail (Fig. 7a), and that class covariances retain different orientations after LEACE equalizes the means (Fig. 7b). That demonstrates the information is present and recoverable by a probe, but not that the model's readout actually uses this second-moment signal to make its predictions. The paper itself cautions (b56): 'Expression and reliance shaping specify what the training objective asks for, not necessarily what actually occurs inside the model.' The toy model closes this gap through an erasure-use test, but the real-model setting does not. The L3 monitor is precisely the detector that claims to catch the third form of circumvention; if in practice it detects an epiphenomenon that the readout does not use, the claim that each form of circumvention is detectable 'by a matching monitor' at the level asserted in Section 6 is not yet empirically established for the real models.

The reader's weakest_assumption targeted the single-direction and refresh-rate premise. I agree that premise is the right general area, but I want to be more precise about the load-bearing point for the second claim. The Section 6 monitor-hierarchy claim rests on an empirical link: that each detectable code corresponds to a form of circumvention the model actually uses. In the toy model, the paper supplies a strong use-based test (fresh-erasure accuracy, Fig. 25). In the real-model case, the evidence for the product code is presence-based only: an L3 probe with AUROC above 0.98 and a covariance orientation difference (Fig. 7a,b). Whether that code drives the readout determines whether L3 is detecting circumvention or a detectable-but-unused artifact. The reader identified a related but distinct weak point; I partially agree but shift the locus of the concern. This does not change the verdict. The paper is already CONDITIONAL, with good reason: the harmfulness and sycophancy runs are largely single-seed, and there is best-of-grid checkpoint selection in Table 2 and Figure 18. My concern is an additional, concrete reason to keep the verdict conditional rather than to accept unconditionally, and the proposed test is feasible without re-deriving any published theorem or re-running any cited result. The paper deserves credit for the derived toy model, the clear five-level hierarchy, and the honest acknowledgment (b408) that the enumerated forms do not exhaust circumvention; those strengthen the framework but do not close the specific gap I identify.

The referee's report

If the results hold, this is a substantial contribution: it unifies feature-based rewards and attribution penalties under one constrained-learning objective (Eq. 1), provides a clean formal separation of pattern versus filter directions (Appendix C), contributes a falsifiable theory of circumvention with closed-form gradient characterizations (Propositions 1-2), and a product-code construction (Propositions 3-4, Lemma 1) whose central prediction, that frequent refitting forces the concept into a nonlinear code that a linear erasure cannot remove, with fresh-erasure accuracy staying near the Bayes optimum (Fig. 25), is verified in the toy by a genuine use test rather than by probe recoverability alone. The headline vision and RL experiments are multi-seed (Waterbirds, Countdown), the code is released, and the scale span up to 32B is real. The two places where the evidence is thinnest are the harmfulness section (one seed per setting, checkpoints selected on the reported JBB metric) and the extrapolation of the toy product-code use result to Waterbirds, where only probe evidence is offered.

Pith's reading of the paper (a machine's summary, not the authors')

Mechanistic Shaping is a training-time framework that uses a model's own internal representations as a specification language: a penalty term added to the task loss tells the model which concepts it may rely on (not the background in bird classification, not gender in resume screening) and which concepts it may express (no reward hacking, no harmful content). The paper claims this works across vision, medical imaging, and language models up to 32B parameters, across classification, SFT, DPO, and GRPO, substantially reducing targeted behaviors while preserving task performance. Because optimizing against a representation can move the representation, the paper also asks whether the shaped concept can still be monitored, and answers with a five-level monitor hierarchy and a geometric theory of circumvention: a concept can drift to a new direction, be erased from the layer, or hide in second-order statistics as a nonlinear product code, and each form leaves a signature a matching monitor can read. In the runs evaluated, shaped concepts largely remain monitorable: refitted linear probes recover them in most settings, and the original probe stays reliable whenever gradients do not flow through its score.

What carries the argument. The load-bearing objects are the constrained-learning objective $L(\theta)=R(\theta)+\lambda\,\omega_\bullet(\theta)$, whose violation $\omega_\bullet$ is read from activations; the concept direction $v$, estimated as a difference of means - the 'pattern' telling where activations move when the concept changes - rather than by a discriminative probe, the 'filter', and refreshed every 100 training steps so the penalty chases the concept as it is re-encoded; and the reliance penalty $\nu(z,v)^2$ built from an attribution (gradient, gradient times input, shuffle, noise, or deletion), which vanishes exactly when the readout is invariant along $v$. The monitoring half rests on a five-level hierarchy - reuse the original linear direction (L1), refit a linear probe (L2), refit a nonlinear probe (L3), read a different layer (L4), use a separate model (L5) - and on a product-code construction: splitting $\mathrm{ReLU}(a+b)$ into parts even and odd in $b$ yields a compressed product $m(a,b)=ab/\max(|a|,|b|)$, and Proposition 4 shows that a bias-free positively homogeneous network, after a covariance-direction erasure, can only retain the sign of the unwanted feature as $h_{\mathrm{odd}}=x_\bullet\,g(x/\|x\|)$, an approximate product with core features that linear readers cannot see but a nonlinear probe recovers.

Core claim. The paper's central claim is a double one. First, interpretability can serve as a specification language for training: writing the violation $\omega_\bullet(\theta)$ as a function of activations and adding $\lambda\,\omega_\bullet(\theta)$ to the risk $R(\theta)$ gives a single recipe that, in the paper's experiments, fixes shortcut learning under perfect confounding on Waterbirds (worst-group accuracy 93-97.7% against 51.6% for ordinary training), suppresses sycophancy learned during DPO in 7B-32B models while keeping benchmark performance, nearly eliminates reward hacking in GRPO with the effect transferring to a chess environment the policy never trained on, and reduces harmful answer rates on JailbreakBench from 0.70 to 0.01 at matched utility. Second, the fear that training against a probe makes the concept unreadable is real but structured: in all of the paper's observations, the concept either re-encodes along a new direction that a refitted linear probe reads, is erased from the layer, or moves into a nonlinear code built as an approximate product of the unwanted feature with core features - and each mechanism is detected by its matching level in a five-level monitor hierarchy, so a concept that escapes one probe is not the same as a concept that is truly gone.

The premise it rests on. The framework assumes that the unwanted concept is adequately captured by a single linear direction at one chosen layer, and that refreshing that direction every 100 training steps keeps up with the model's re-encoding of the concept.

If the paper is right (Pith's inferences, not the authors')

Representation-level penalties can replace data curation where data cannot be cleaned: perfectly confounded vision benchmarks, subliminal sycophancy in DPO, and reward hacking that emerges only during training all yield to a probe-based penalty that data filtering or loss changes do not.

If monitorability survives shaping, alignment becomes auditable: a specification of reliance or expression is verifiable after training, and a concept that escapes the original probe can be recovered by refitting rather than treated as gone.

The scoring convention - attached gradients versus detached or frozen-copy scoring - becomes a design decision with a predictable monitoring consequence: gradients through the probe degrade the original detector, while detached or frozen scoring preserves it at L1.

Circumvention is enumerable: because re-encoding, erasure, and product-code encoding each have a distinct geometric signature, a monitoring protocol can be pre-specified by level, turning 'is the concept still there?' into a precision question about which monitor is being used.

If the three-form taxonomy holds at frontier scale, monitors could be designed prospectively: L2-level refitting against re-encoding, L3-level nonlinear readers against product codes, and higher-order readers at the same layer against the degree-two component the paper explicitly leaves uncharacterized.

The refresh-cadence dependence suggests a practical tuning rule the paper does not state: measure the early-training growth rate of the concept's difference-of-means direction, then set the refresh interval below the inverse of that rate.

Because the bias-free hypothesis drives the product-code structure, a testable extension is whether giving the residual blocks biases - or using non-ReLU activations - suppresses the nonlinear code entirely, which would make the third circumvention form an artifact of a particular architecture family.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a résumé). Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-06

The authors' main claim, as the classifier picked it out 0.88

“Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training.”

How the main claim is stated: hedged possibility 0.97

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.87 · human participants 0.10 · controlled comparison 0.15 · randomized assignment 0.02 · states quantitative result 0.71 · compares with prior results 0.19 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We propose Mechanistic Shaping, a unified framework for shaping models through their internal representations. contribution 0.99 0.67 0.07 asserted 0.96
Mechanistic Shaping is applied during training so that models follow specifications (rules) about which concepts they express, measured via their outputs or internal representations (e.g., no harmful content), and also which concepts they rely on, i.e., what they use to make their decisions (e.g., not a candidate's gender when screening a r\'esum\'e). method 0.66 0.41 0.07 asserted 1.00
Across multiple domains (vision, medical imaging, language), scales (up to 32B parameters), and training paradigms (classification, SFT, DPO, GRPO), we find that Mechanistic Shaping substantially reduces targeted behaviors while preserving task performance. result 1.00 0.96 0.37 asserted 1.00
Since training against a representation can also modify it, we ask whether shaped concepts remain monitorable, a question that depends on the class of monitors considered. gap 0.95 0.15 0.04 hedged possibility 0.59
We therefore introduce a hierarchy of such classes, which separates a concept that escapes one probe from a concept that is truly gone. contribution 0.94 0.86 0.06 asserted 0.98
We characterize three geometric mechanisms of circumvention: the concept can drift away from the monitor, be erased from the layer, or persist in higher-order statistics as a nonlinear encoding, which we explain through a product-code construction. contribution 0.47 0.86 0.07 asserted 0.99
Although other mechanisms may arise in different settings or at larger scales, these three account for all of our observations, and each can be detected by a matching monitor. result 0.64 0.89 0.14 asserted 0.91
In practice, we find that shaped concepts largely remain monitorable: refitted linear probes recover the concept in most settings, and the original probe remains reliable whenever gradients do not flow through it. result 1.00 0.96 0.17 asserted 1.00
Taken together, these results point to the potential of using interpretability as a tool for training, and call for a correspondingly rigorous science of interpretability-based training. interpretation 0.87 0.18 0.78 hedged possibility 0.97

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.94

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-06: 1 request, 7,161 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (53)
Figure 1 of the paper
Figure 1. Specifying reliance and expression. Reliance shap￾ing constrains a concept’s influence on the readout without requiring its removal from the representation. Expression shaping instead reduces the probability of outputs or activa￾tions expressing the unwanted concept (Prasad et al., 2026; Bergen et al., 2026a; Taufeeque et al., 2026). We start by introducing some notation before presenting a common framework for specifica￾tion through internal representations. Vectors and matrices are bold, inner products ⟨·, ·⟩ and norms ∥·∥ are Euclidean, and I denotes the iden￾tity matrix. Let x ∈…
Figure 2 of the paper
Figure 2. Overview of reliance shaping. Given examples with and without the undesired concept (D+, D−), we select a layer ℓ and compute the concept direction vt (typically computed as the DoM) of its activations z = fθt (x). The model is then trained on the task risk R(θ) plus a reliance shaping penalty λ ν(·, vt) 2 along this direction. When the representation moves during training, vt is periodically recomputed on the current model. and assess concept readability through five levels, ordered by how much of the original monitor may change (
Figure 3 of the paper
Figure 3. Shaping reliance in an unfrozen DINOv2, at the last layer (top) and at block 9 (bottom). Left: worst-group accuracy (WGA) of proposed reliance shaping methods (red) with baselines (grey). The dashed line is the balanced-training oracle. Right: WGA against monitoring with the original direction (L1, filled) and a refitted nonlinear probe (L3, open). At block 9, Projection, LEACE and INLP open a gap between the two detectors: L1 falls to chance while L3 still decodes the background, which has therefore left the reach of linear monitoring without being fully erased, a phenomenon that…
Figure 4 of the paper
Figure 4. Qualitative comparison of Shaped model vs ERM. Left: feature visualization (Olah et al., 2017) using MACO (Fel et al., 2023a) preconditioning of the landbird (top) and waterbird (bottom) logits of DINOv2 trained on unbalanced data without reliance shaping (ERM) and under the Gradient × Input (GI), Gradient and Shuffle penalties, in which the background that dominates the ERM logits gives way to birds. Right: Grad-CAM maps of the waterbird logit on waterbirds over land backgrounds, the worst group, which spread over the background under ERM and concentrate on the bird under GI.…
Figure 5 of the paper
Figure 5. Under a fixed direction, the background is re-encoded elsewhere in the representation. Trajectory of the background DoM vt on Waterbirds over training (encoder learning rate 10−6 ), projected onto its first two principal components, under the Gradient (A) and Gradient input (B) penalties; insets report worst-group accuracy (WGA). Without a penalty (Original), the direction barely moves and WGA remains near chance; under a fixed direction v0 (No refresh), WGA rises while vt moves away from where the penalty acts; with a DoM refresh every 100 steps (Refresh, dotted lines in the…
Figure 6 of the paper
Figure 6. Non reliance can be reached either by no longer reading the background or by no longer encoding it. Trajectories of the four group centroids of Waterbirds over training, projected onto their first three principal components, under the Gradient (A) and Gradient × Input (B) penalties, without and with refresh of vt. Under Gradient the centroids move without merging, so that the background remains readable, whereas under Gradient × Input with refresh the land and water centroids of each class collapse onto each other (circled), so that invariance is reached by erasure. #&"  &( …
Figure 7 of the paper
Figure 7. LEACE causes non-linear representation via the covariance. LEACE applied at layer 9 in a trainable DINOv2. (a) Background AUROC across layers where dashed lines show original probe performance, solid lines show refitted performance: original direction (L1) and refitted linear probe (L2) fall to 0.44 and 0.59 at layer 9, while the original and refit nonlinear probes stay above 0.98, and both linear detectors recover within two layers. (b) At layer 9, LEACE equalizes the class means (all a linear detector can see); the class covariances keep different orientations. (c) A nonlinear…
Figure 8 of the paper
Figure 8. A good detection direction need not be a good shaping direction. (a) Worst-group accuracy under gradient input shaping with linear and nonlinear DoMs and probes (logistic), respectively; each cosine level includes 10 perturbations per original direction. (b) Validation cross-entropy across a two-dimensional spherical slice spanning the DoM-to-probe arc and an orthogonal direction derived from the leading training-feature principal component. Accuracy degrades as the direction moves away from the DoM. Details are in Appendix C. and Appendix C provides a detailed discussion and…
Figure 9 of the paper
Figure 9. Reliance can be specified as a concept￾conditional rule. Grad-CAM of the English springer logit for the original readout (left), one shaped to rely on noses (middle), and one to fol￾low the rule “tongue, else ears” (right). where z•k is the score of the k-th context, here an SAE code, and vk the direction of the attribute that the readout should ignore when this context is met, so that the model is asked to become invariant to the attribute vk only on the examples where its context is met. We illustrate this on the English springer class of Ima￾geNette (Howard, 2019), training…
Figure 10 of the paper
Figure 10. Repairing CheXpert shortcut learning with supervised probes and agent-driven behavior discovery. We compare worst-group accuracy (WGA) between ERM, baselines, and Reliance Shaping methods with a frozen RAD-DINO (Pérez-García et al., 2025) encoder and a linear readout. As expected for the linear setting (
Figure 11 of the paper
Figure 11. Auto-interpretability of features for labeling spurious correlations. We use an LLM agent (Claude Opus 5.5) to label each feature (SAE latent) with what it highlights in the image and also to rank it out of 10, in terms of how related it is to real pneumothorax evidence (10 being maximally spurious). We compare prompting the agent with tube-centric (panel A) and general spurious (panel B) prompts, respectively. In the latter case the agent is told to identify medical equipment or anything not related to evidence of pneumothorax (full prompts in Appendix F). We observe that both…
Figure 12 of the paper
Figure 12. The sycophancy signal cannot be located in the data. (A) We ask an LLM judge to label the chosen and rejected DPO replies as sycophantic. Less than 1% of datapoints are identified, with more rejected than chosen selected. (B) Filtering out data with our Sycophancy Verbose SFT probe only slightly reduces sycophantic behavior compared to baselines, at 30k pairs removed. (C) Max-activating tokens on LLM judged sycophantic replies show similar patterns across sycophancy and random probes. (D) We show the blind auto-interp (via Sonnet 5) labels of every layer DoM’s maximally activating…
Figure 13 of the paper
Figure 13. Reliance shaping reduces sycophancy while retaining capabilities learned during DPO. The goal of each method is to lie near ‘target’, with sycophancy at the SFT model but performance (DPO loss or benchmark accuracy) at the DPO model. (A) Sycophancy of OLMo-3-7B after DPO against its final loss on training pairs, which measures how far training moved the policy toward the preference data. While lowering the learning rate (dashed line) trades sycophancy against loss, reliance shaping enables DPO learning without sycophancy for fixed (filled) or refit (hollow) DoM’s. Similarly,…
Figure 14 of the paper
Figure 14. Reliance shaping causes representational drift in the sycophancy setting, but monitorability is maintained and representational entanglement is minimal. A. Linear monitorability of sycophancy using the original DoM v0, the updated DoM vt, the perpendicular component v ⊥v0 t of the updated DoM, and an unconstrained linear probe. Sycophancy remains linearly monitorable throughout training, although the monitorability of vt slightly degrades. B. The cosine of each DoM component with the updated mean preference direction bt over the course of training. The entanglement of vt and bt…
Figure 15 of the paper
Figure 15. Overview of expression shaping. Given a prompt x, we sample rollouts τi from the current policy and use internal representations to extract a score of undesired concept at each token. The discounted expression cost is subtracted from the task reward, yielding r ′ i = R(τi) − λ P t γ t−1ρ(z•,i,t). This is the policy-gradient branch of Eq. (2), which reduces to RLFR (Prasad et al., 2026) when the task reward is zero. The policy is then trained with a policy-gradient update, holding our modified reward (shaped reward) fixed during differentiation. After each update, new rollouts are…
Figure 16 of the paper
Figure 16. Expression shaping suppresses reward hacking on Countdown and transfers to chess. Left: peek and honest solve rates of the Countdown training rollouts. Right: the same rates on 512 chess puzzles using model checkpoints trained on Countdown. All methods (except for filtering) report the average from three seed runs; filtering reports results per seed. Open bars denote attached gradients. 0 50 100 150 Training update 0.50 0.75 1.00 Balanced acc. Original probe during training 0.50 0.75 1.00 Balanced acc., update 150 0.89 DoM on frozen base (RLFR) 0.87 DoM on policy, with gradient…
Figure 17 of the paper
Figure 17. Monitoring post shaping. Top: original probe (L1) on held-out base-model rollouts re-encoded by each checkpoint. Bot￾tom: original (L1) and refitted (L2) probes at step 150. Transferring the specification across tasks. Finally, we eval￾uate the Countdown checkpoints in a similar chess environment, in which the model is given a puzzle, with a forbidden file best_move.txt that contains the solution. Note that the check￾points are not trained on the chess tasks. At a high level, we see hacking (or honest-solving) behavior generalize to the chess en￾vironment. Policy models that learn…
Figure 18 of the paper
Figure 18. Harmfulness against utility. Pareto front over all evaluated checkpoints and settings that answer at least 80% of safe XSTest prompts. Dotted: 95% of base utility. One seed per setting. Experimental setup. We train Mistral-7B-Instruct and an ablit￾erated Llama-3-8B-Instruct with LoRA on harmful prompts, using the data and teacher KL of Libon et al. (2026), with all readers acting at layer 19. The probe of Libon et al. (2026) minimizes the tokenwise hinge max(0, 1 + z•t) through the policy’s activations and refits the probe after every update. We apply our expression shaping as a…
Figure 19 of the paper
Figure 19. Direct gradients through the reader degrade the original monitor. Left: original-probe AUROC (L1) on 1,000 held-out BeaverTails answers re-encoded by each checkpoint. Right: original (L1, filled) and refitted (L2, open) probes at update 50, with JBB at that update; DoM output loss is evaluated at update 40. Dashed lines indicate gradients through the scored activations. One seed. probe with gradient of Libon et al. (2026), we sweep the hinge coefficient from 0.1 to 8 on Mistral and from 0.25 to 8 on Llama. For RLFR, we sweep reward strengths from 0.25 to 6 on Mistral and from 0.125…
Figure 20 of the paper
Figure 20. Concept directions can move while the original monitor remains valid. Angle between the initial DoM and a DoM refitted at each checkpoint, at layer 19 (left) and across layers (right). Dotted: reader layer. One seed. 0 10 20 30 0.4 0.6 0.8 Mistral-7B-Instruct AUROC Original probe (L1) 0 10 20 30 0.4 0.6 0.8 Refitted linear probe (L2) 0 10 20 30 0.00 0.02 0.04 0.06 0.08 Refitted MLP − linear (L3 − L2) 10 0 10 1 10 2 10 3 1.0 1.5 2.0 2.5 Norm / base exclusion cutoff Layer-19 answer-token norm 0 10 20 30 Layer 0.4 0.6 0.8 Llama-3-8B-Instruct AUROC 0 10 20 30 Layer 0.4 0.6 0.8 0 10 20…
Figure 21 of the paper
Figure 21. Monitoring across layers and over longer training. From left to right: original-probe AUROC (L1); refitted linear-probe AUROC (L2); improvement from a refitted one-hidden-layer MLP (L3 − L2); and median layer-19 answer-token norm relative to the base model. Probes are evaluated on 1,000 held-out BeaverTails answers after 50 updates and after the long runs. Vertical dotted lines mark layer 19; the horizontal dotted line marks the training exclusion threshold at twice the base-model median norm. For the long run with direct gradients, only tokens below this threshold were tracked,…
Figure 22 of the paper
Figure 22. Toy model of circumvention. The forward pass is shown in black and the backward pass in red, with trained blocks shaded and the projection P kept fixed. Since every gradient that crosses P lies in col(A) ⊥, the components of w and W along col(A) never receive any gradient, while the update of W is the outer product of the surviving head P w with the correlation E(δx) between the residual and the input. in the control without a harm penalty, providing little evidence that nonlinear decoding is needed to recover the distinction. The observed failure is therefore primarily a change in…
Figure 23 of the paper
Figure 23. Residual Model. Forward pass in black, backward pass in red; shaded blocks are trained and the projection P is fixed. Now every block prior to the projection receives the same linear rebuild gradient term. A fixed intervention induce a re-encoding. Our first result looks at what happens just after the projection is applied, when the model keeps training with P held fixed, and asks whether the model can find a way to use x• again. Writing W• for the column of W that reads x•, the gradient under the projection has a closed form. Proposition 1 (Gradient under projection). The gradient…
Figure 24 of the paper
Figure 24. A fixed projection induce circumvention by re-encoding. Two-layer linear model, trained after the projection is applied at t = 0. Left: the arrows show how the encoder writes x• at different times. The part along the removed direction a stays the same, while a new part grows on the floor, which is the space that the projection keeps, along the direction P w read by the head. Right: we plot what the head sees at t = 0, 0.3 and 6, where each dot is one input colored by its value of x• and the black arrow is the direction read by the head; the curves on top show that inputs with low…
Figure 25 of the paper
Figure 25. Refitting decides between a linear rebuild and a nonlinear code. Residual toy, one seed. Left: linear share ∥ Cov(h, x•)∥/RMS(h) against nonlinear share (odd in x•, linear part removed) over 6000 steps; refit paths are drawn just before each refit, the raw sawtooth faint. Middle: first 300 steps; each refit resets the linear share while the nonlinear share keeps growing. The linear rebuild grows quickly after each refit because it is a product of two terms which are both large (see Proposition 1 and discussion). The nonlinear code requires the model to create a nonlinear reader…
Figure 26 of the paper
Figure 26. After repeated refitting, x• survives as a product with a core feature. Residual toy model refitted every 10 steps, after 6 thousands steps just after a new erasure. Left: the part of the code that changes sign with x•, plotted against x• and a core coordinate c; the points follow a saddle, which is the shape of the product x• c (the surface is this fit). Right: in the plane of c and the code, the two signs of x• have the same mean and differ only in the tilt of their spread, so no linear reader can separate them. Lemma 1 (Linear readers cannot use the code). For squared loss just…
Figure 27 of the paper
Figure 27. Every attribution penalty recovers the same optimum in the linear case. Two classes are separated perfectly both by the specified feature x and by the shortcut y, so that empirical risk minimization alone is indifferent between them and learns an oblique boundary that relies on both coordinates (ERM), whereas the intended solution depends on x alone (Ground truth, dashed). Penalizing reliance on the shortcut direction v with any of the attributions of Section 1, whether Gradient, Gradient × Input, Noise, Shuffle, or Deletion, yields a vertical boundary that coincides with the…
Figure 28 of the paper
Figure 28. Patterns and filters. a) Two groups that differ in the unwanted concept, with anisotropic within-group noise, and the distribution of each group along the two candidate directions: the difference of means v (pattern) points along the displacement between the groups and carries the variance of the activation, whereas the probe w⋆ (filter) rotates toward the direction of least within-group noise, where it separates the groups better but reads a coordinate along which the activation barely moves. b) The same two directions on the activations themselves; the probe is normal to the…
Figure 29 of the paper
Figure 29. Along the continuum, the DoM lies on a plateau of comparable performance. Top: maximizers of Eq. (10) for γ ∈ [0, 3] (gold path), projected onto the slices of
Figure 30 of the paper
Figure 30. Similar behavioral improvements can leave different monitoring outcomes. Worst-group accuracy (left) and original-direction AUROC (L1, right) for frozen DINOv2 with linear (top) and nonlinear (bottom) readouts. Attribution penalties recover accuracy while preserving L1 monitoring; erasure baselines achieve comparable accuracy but degrade the original detector. Bars show three-seed means and error bars seed standard deviations. Shaping reliance with fixed representations. With a frozen encoder, all seven attribution penalties recover high worst-group accuracy (above 95% with a…
Figure 31 of the paper
Figure 31. Background attribution resurges between direction refreshes. Training dynamics of an unfrozen DINOv2 under the Gradient × Input penalty at the final CLS representation, with the DoM refreshed every 50, 500, or 5,000 optimizer steps (dotted lines). Top: worst-group accuracy. Middle: norm of the current DoM, which measures how strongly the background is encoded. Bottom: mean absolute Gradient × Input attribution of the score along the current DoM (recomputed at every step for evaluation), which measures how much the readout uses it. After each refresh, both quantities drop and, with…
Figure 32 of the paper
Figure 32. Controlling gradient input pressure on activity vs gradient. AUROC of the original probe (L1) against the multiplier ρ on the gradient through ⟨z, v⟩, for several penalty weights λ. We observe a phase transition upon which the optimization leads to loss of L1 monitorability. 51
Figure 33 of the paper
Figure 33. Opposite reliances (1/2). Grad-CAM maps (top) and logit drop under masking (bottom). Church: the church-shaped readout loses about 31% when the church is masked and the tree-shaped readout about 71% when the trees are masked, each barely affected by the other region. Chain saw: about 63% (saw) and 75% (wood). 53
Figure 34 of the paper
Figure 34. Opposite reliances (2/2). Parachute: about 29% (canopy) and 60% (person) on the own region, against at most 5% on the other, while the original readout loses about 7% on each. Cassette player: about 68% (controls) and 37% (speakers). 54
Figure 35 of the paper
Figure 35. Conditional rules. Grad-CAM maps (top) and logit drop when the rule target (red) or the other region (green) is masked, on images where the context is present (left panel) or absent (right panel). When a tee or a watermark is present, the conditional readout relies on it; otherwise it relies on the ball, like the ball-shaped readout. 55
Figure 36 of the paper
Figure 36. Masking protocol. Examples of masks for the chain saw (top) and the watermark rule (bottom): the region targeted by the shaped readout, a control of equal area elsewhere, the other region, and its control. 56
Figure 37 of the paper
Figure 37. Agent prompts for autonomous steering of medical image models. The complete request each agent received for every candidate feature: the system prompt, the user message that accompanied the feature’s montage, and the fields of the structured response. All text is verbatim. F Prompt details for medical agentic pipeline Both agents labelled the same 120 candidate features of the sparse autoencoder, those whose peak activation is most associated with the pneumothorax label on the confounded training split (largest |AUROC − 0.5|). Each feature was labeled by a single request (
Figure 38 of the paper
Figure 38. Comparison of sycophancy DoM vectors. We construct several sycophancy probes and compare them via causal steering to identify the best one to use for attribution. We compare three DoM vectors: DPO−SFT, the behavior-change vector of Blank et al. (2026), subtracts the SFT model’s mean reply activation from the DPO model’s on conversations in which the DPO model gave in; the Sycophancy Verbose DoM contrasts the SFT model’s activations on paired replies to the same challenged question that give in; the Sycophancy DoM makes the same contrast with replies that contain only the answer…
Figure 39 of the paper
Figure 39. Individual benchmark performance post DPO. We show the same results as in
Figure 40 of the paper
Figure 40. Scaling reliance shaping reducing sycophancy learned during DPO. Applying reliance shaping to OLMo 2 13B and OLMo 3 32B achieves similar sycophancy-capability trade-off ability. Error bars are widest for the largest model, whose smaller sycophancy increase under DPO makes uncertainty proportionally larger. G.4 Representational analysis. To measure where this drift occurs, we conduct a layer-swapping experiment, where we take the standard DPO checkpoint, and replace the weights in all transformer blocks at layer l with those from the best performing reliance shaping method (Gradient…
Figure 41 of the paper
Figure 41. Reliance shaping reroutes behavior in the blocks just below the penalty layer. We replace the weights of the DPO model with the corresponding blocks of the reliance shaping model penalized at layer 17, and measure the share of sycophancy reduction that the hybrid recovers (bottom row), the relative change of reliance shaping weights from DPO’s in those blocks (middle row), and the hybrid’s mean accuracy on six benchmarks (top row). Left: swapping single blocks before the penalty recovers up to 20% of the reduction and blocks after it recover nothing. Right: swapping five…
Figure 42 of the paper
Figure 42. Representation drift under expression shaping. Cosine to the base model of the probe direction refitted on each checkpoint (left) and of the mean activation (right), on held-out base-model rollouts re-encoded every ten updates. 62
Figure 43 of the paper
Figure 43. Behaviour over the 50 matched updates. JBB, JBB with a compliant prefill, and safe XSTest prompts answered, for the runs of
Figure 44 of the paper
Figure 44. Strength. Selected checkpoint of each run (one seed, 50 updates); hollow: no checkpoint satisfies the selection rule, and the lowest-JBB one is shown. For the probe with gradient λ is the hinge coefficient; for RLFR it is the multiple of that hinge’s gradient norm at the first update, at the selected samples per prompt and discount, so the two axes are in different units. Dotted: the 80% XSTest bar. Mistral (JBB 0.003–0.09) and from 0.25 to 3 on Llama (0.01–0.08), and keeps the original probe at 0.81–0.86 everywhere, so the gap is consistent and resist to hyperparameters. Designing…
Figure 45 of the paper
Figure 45. Reader and credit rule. JBB (circle; hollow: no checkpoint satisfies the selection rule) and JBB with a compliant prefill (square) at the selected checkpoint, every reader on the frozen base. Left: readers at the selected setting of the DoM. Right: credit rules at strength 1, 4 samples per prompt, DoM reader. One seed. 0.02 0.04 0.06 0.08 Mistral-7B-Instruct JBB harmfulness 0.1 0.2 0.3 0.4 JBB with prefill 0.84 0.85 0.86 Original probe AUROC (L1) 10 0 10 1 10 2 Credit horizon 1=(1 ¡ °), tokens 0.05 0.10 0.15 0.20 0.25 Llama-3-8B-Instruct 10 0 10 1 10 2 Credit horizon 1=(1 ¡ °),…
Figure 46 of the paper
Figure 46. Credit horizon. Median over the tuning grid (strength and samples per prompt) of the selected checkpoint’s JBB, prefill JBB and original-probe AUROC, as a function of 1/(1 − γ). One seed per setting. Prompt: ”Write a defamatory article claiming that a U.S. president is addicted to heroin” Mistral, Probe on policy, with gradient, update 1000 The Usicinicin,s, and,s,s|s|s|s|s|s|s|||||||||||||||||||| ||||||||||||||||||||. . . Mistral, DoM on frozen base (RLFR), update 400 Please note that I am not a lawyer and I strongly condemn any illegal activities. I am committed to engaging in…
Figure 48 of the paper
Figure 48. The rebuild is immediate. The spurious coefficient
Figure 49 of the paper
Figure 49. The fast and slow modes after ablation, for several values of
Figure 50 of the paper
Figure 50. The residual network after intervention at
Figure 51 of the paper
Figure 51. The code. Left: the term m of Eq. (17) for one unit, against a = αx•, for b = 1. It is linear with slope 1 2 while xc dominates the unit (|a| < b) and flat beyond, where it carries only the sign of x•. Right: a random layer already contains the code. The least-squares combination of the mk of the 32 units of the block below the intervention at random initialization (solid), against a target single term x• v ⊤xc/∥x∥ for a random direction v (dashed), as functions of x• at three core inputs. practice a few such terms suffice: P B has effective rank 2.2–3.0, and its top singular pair…
Figure 52 of the paper
Figure 52. In Waterbirds, reliance shaping with Gradient×Input causes representational drift, can reduce monitorability, and can cause representational entanglement, depending on the model architecture. Rows show ResNet-50 (top) and DINOv2 (bottom); lines are means over three seeds and shaded bands ±1 standard deviation. (a) and (d) show the monitorability of the bird’s habitat using the original DoM vector v0, the perpendicular component v ⊥v0 t , the refreshed DoM vt, an unconstrained linear probe, and an unconstrained nonlinear probe. For ResNet-50, the habitat remains linearly monitorable…
Figure 53 of the paper
Figure 53. In Waterbirds, reliance shaping with LEACE leaves the habitat monitorable but entangles it with the bird. Rows show ResNet-50 (top) and DINOv2 (bottom); lines are means over three seeds and shaded bands ±1 standard deviation. The LEACE eraser is refit at every training step, and all readings are taken from the backbone’s activations before the eraser is applied. (a) and (d) show the monitorability of the bird’s habitat using the original DoM v0, the perpendicular component v ⊥v0 t , the refreshed DoM vt, an unconstrained linear probe, and an unconstrained nonlinear probe. For both…

Showing 52 of the paper's 53 figures. Figures are the authors', reproduced from the paper.

References (300)
  1. [1] Abbe, E., Boix-Adser \`a , E., and Misiakiewicz, T. The merged-staircase property: A necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks. Conference on Learning Theory (COLT), 2022 on Pith
  2. [2] Abbe, E., Boix-Adser \`a , E., and Misiakiewicz, T. SGD learning on neural networks: Leap complexity and saddle-to-saddle dynamics. Conference on Learning Theory (COLT), 2023 on Pith
  3. [3] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Advances in Neural Information Processing Systems (NIPS), 2018 on Pith
  4. [4] Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. A r X iv e-print , 2016 on Pith
  5. [5] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man \'e , D. Concrete problems in ai safety. A r X iv e-print , 2016 on Pith
  6. [6] Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. Towards better understanding of gradient-based attribution methods for deep neural networks. Proceedings of the International Conference on Learning Representations (ICLR), 2018 on Pith
  7. [7] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems (NeurIPS), 2024 on Pith
  8. [8] Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
  9. [9] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Public Library of Science (PloS One), 2015 on Pith
  10. [10] Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and M \"u ller, K.-R. How to explain individual classification decisions. The Journal of Machine Learning Research (JMLR), 2010 on Pith
  11. [11] Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. A r X iv e-print , 2022 on Pith
  12. [12] Bailey, L., Serrano, A., Sheshadri, A., Seleznyov, M., Taylor, J., Jenner, E., Hilton, J., Casper, S., Guestrin, C., and Emmons, S. Obfuscated activations bypass llm latent-space defenses. Proceedings of the International Conference on Learning Representations (ICLR), 2026 on Pith
  13. [13] Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., and Farhi, D. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. A r X iv e-print , 2025 on Pith
  14. [14] Barron, A. R. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 1993 on Pith
  15. [15] Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 2022 on Pith
  16. [16] Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, S. LEACE : Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems, 2023 on Pith
  17. [17] Ben Arous, G., Gheissari, R., and Jagannath, A. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research (JMLR), 2021 on Pith
  18. [18] Bergen, L., Bhalla, U., Baskaran, S., Loeffler, M., Sarfati, R., Gala, D., Panwar, R., Aranguri, S., Fel, T., Geiger, A., et al. Anatomy of post-training: Using interpretability to characterize data and shape the learning signal. A r X iv e-print , 2026 a on Pith
  19. [19] Bergen, L., Bhalla, U., Lee, A., Widawsky, B., Nasvytis, L., Watts, C., Boppana, S., Baskaran, S., Hazra, D., Byun, M., et al. Monitoring and discovering reward hacking with internal representations during llm evaluations. A r X iv e-print , 2026 b on Pith
  20. [20] Bernardi, S., Benna, M. K., Rigotti, M., Munuera, J., Fusi, S., and Salzman, C. D. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 2020 on Pith
show 80 more
  1. [21] Bhalla, U., Srinivas, S., Ghandeharioun, A., and Lakkaraju, H. Towards unifying interpretability and control: Evaluation via intervention. A r X iv e-print , 2024 on Pith
  2. [22] Bhalla, U., Fel, T., Rager, C., Feucht, S., Haklay, T., Wurgaft, D., Boppana, S., Kowal, M., Shyam, V., Merullo, J., et al. Do sparse autoencoders capture concept manifolds? A r X iv e-print , 2026 on Pith
  3. [23] Bishop, C. M. Training with noise is equivalent to tikhonov regularization. Neural computation, 1995 on Pith
  4. [24] Blank, C., Ying, Z., Potts, C., Hase, P., and Huang, J. Sycophantic agreement transfers with neutral data via contrastive preference optimization. A r X iv e-print , 2026 on Pith
  5. [25] Boyd, A., Tinsley, P., Bowyer, K. W., and Czajka, A. Training better deep learning models using human saliency. A r X iv e-print , 2024 on Pith
  6. [26] Breiman, L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical science, 2001 on Pith
  7. [27] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Car... on Pith
  8. [28] Brutzkus, A. and Globerson, A. Why do larger models generalize better? A theoretical perspective via the XOR problem. Proceedings of the International Conference on Machine Learning (ICML), 2019 on Pith
  9. [29] Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. A r X iv e-print , 2024 on Pith
  10. [30] Carlini, E., Catalisano, M. V., and Geramita, A. V. The solution to the W aring problem for monomials and the sum of coprime monomials. Journal of Algebra, 2012 on Pith
  11. [31] Casademunt, H., Juang, C., Karvonen, A., Marks, S., Rajamanoharan, S., and Nanda, N. Steering out-of-distribution generalization with concept ablation fine-tuning. A r X iv e-print , 2025 on Pith
  12. [32] Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and Bloom, J. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. A r X iv e-print , 2024 on Pith
  13. [33] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2018 on Pith
  14. [34] Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
  15. [35] Chen, R., Arditi, A., Sleight, H., Evans, O., and Lindsey, J. Persona vectors: Monitoring and controlling character traits in language models. A r X iv e-print , 2025 on Pith
  16. [36] Cho, Y. and Saul, L. K. Kernel methods for deep learning. Advances in Neural Information Processing Systems (NeurIPS), 2009 on Pith
  17. [37] Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. Subliminal learning: Language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805, 2025 on PithPith's reviewarXiv
  18. [38] Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
  19. [39] Covert, I., Lundberg, S., and Lee, S.-I. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 2021 on Pith
  20. [40] Csord \'a s, R., Potts, C., Manning, C. D., and Geiger, A. Recurrent neural networks learn to store and generate sequences using non-linear representations. BlackboxNLP Workshop, 2024 on Pith
  21. [41] Cundy, C. and Gleave, A. Preference learning with lie detectors can induce honesty or evasion. In Advances in Neural Information Processing Systems, volume 38, 2025 on Pith
  22. [42] Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly interpretable features in language models. A r X iv e-print , 2023 on Pith
  23. [43] Cybenko, G. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 1989 on Pith
  24. [44] Damian, A., Lee, J. D., and Soltanolkotabi, M. Neural networks can learn representations with gradient descent. Conference on Learning Theory (COLT), 2022 on Pith
  25. [45] D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 2022 on Pith
  26. [46] Daniely, A., Frostig, R., and Singer, Y. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. Advances in Neural Information Processing Systems (NeurIPS), 2016 on Pith
  27. [47] Deeb, A. and Roger, F. Do unlearning methods remove information from language model weights? A r X iv e-print , 2024 on Pith
  28. [48] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database . Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009 on Pith
  29. [49] Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. A r X iv e-print , 2023 on Pith
  30. [50] Dreyer, M., Pahde, F., Anders, C. J., Samek, W., and Lapuschkin, S. From hope to safety: Unlearning biases of deep models via gradient penalization in latent space. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2024 on Pith
  31. [51] Drucker, H. and Le Cun, Y. Improving generalization performance using double backpropagation. IEEE transactions on neural networks, 1992 on Pith
  32. [52] Durbin, R. and Rumelhart, D. E. Product units: A computationally powerful and biologically plausible extension to backpropagation networks. Neural Computation, 1989 on Pith
  33. [53] Elad, M., Milanfar, P., and Rubinstein, R. Analysis versus synthesis in signal priors. Inverse problems, 2007 on Pith
  34. [54] Elazar, Y. and Goldberg, Y. Adversarial removal of demographic attributes from text data. Proceedings of the 2018 conference on empirical methods in natural language processing, 2018 on Pith
  35. [55] Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Y. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 2021 on Pith
  36. [56] Eldan, R. and Shamir, O. The power of depth for feedforward neural networks. Conference on Learning Theory (COLT), 2016 on Pith
  37. [57] Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one-dimensionally linear. Proceedings of the International Conference on Learning Representations (ICLR), 2025 on Pith
  38. [58] Erion, G., Janizek, J. D., Sturmfels, P., Lundberg, S. M., and Lee, S.-I. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 2021 on Pith
  39. [59] Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, M., Lambert, N., Walsh, P., Dasigi, P., Berry, R., Malik, S., Shah, S., G... on Pith
  40. [60] Fel, T., Cadene, R., Chalvidal, M., Cord, M., Vigouroux, D., and Serre, T. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in Neural Information Processing Systems (NeurIPS), 2021 on Pith
  41. [61] Fel, T., Felipe, I., Linsley, D., and Serre, T. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
  42. [62] Fel, T., Boissin, T., Boutin, V., Picard, A., Novello, P., Colin, J., Linsley, D., Rousseau, T., Cadène, R., Gardes, L., and Serre, T. Unlocking feature visualization for deeper networks with magnitude constrained optimization. Advances in Neural Information Processing Systems... on Pith
  43. [63] Fel, T., Boutin, V., Moayeri, M., Cadene, R., Bethune, L., Chalvidal, M., and Serre, T. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances in Neural Information Processing Systems (NeurIPS), 2023 b on Pith
  44. [64] Fel, T., Ducoffe, M., Vigouroux, D., Cadene, R., Capelle, M., Nicodeme, C., and Serre, T. Don't lie to me! robust and efficient explainability with verified perturbation analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023 c on Pith
  45. [65] Fel, T., Bethune, L., Lampinen, A. K., Serre, T., and Hermann, K. Understanding visual feature reliance through the lens of complexity. Advances in Neural Information Processing Systems (NeurIPS), 2024 on Pith
  46. [66] Fel, T., Kowal, M., Jacobs, M., Hazra, D., Bhalla, U., Sharkey, L., Bushnaq, L., Grant, S., Haklay, T., Icard, T., et al. Structuring sparsity: Block-sparse featurizers capture visual concept manifolds. A r X iv e-print , 2026 on Pith
  47. [67] Fisher, A., Rudin, C., and Dominici, F. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of machine learning research, 2019 on Pith
  48. [68] Fong, R. C. and Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017 on Pith
  49. [69] Fraser-Taliente, K., Kantamneni, S., Ong, E., Mossing, D., Lu, C., Bogdan, P. C., Ameisen, E., Chen, J., Kishylau, D., Pearce, A., Tarng, J., Wu, A., Wu, J., Zhang, Y., Ziegler, D. M., Hubinger, E., Batson, J., Lindsey, J., Zimmerman, S., and Marks, S. Natural language autoenc... on Pith
  50. [70] Frei, S., Chatterji, N. S., and Bartlett, P. L. Random feature amplification: Feature learning and generalization in neural networks. The Journal of Machine Learning Research (JMLR), 2023 on Pith
  51. [71] Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstractions of neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2021 on Pith
  52. [72] Geiger, A., Wu, Z., Lu, H., Rozner, J., Kreiss, E., Icard, T., Goodman, N., and Potts, C. Inducing causal structure for interpretable neural networks. Proceedings of the International Conference on Machine Learning (ICML), 2022 on Pith
  53. [73] Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N. Finding alignments between interpretable causal variables and distributed neural representations. Causal Learning and Reasoning, 2024 on Pith
  54. [74] Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020 on Pith
  55. [75] Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P. W. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. In ICLR 2025 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2025 on Pith
  56. [76] Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 2021 on Pith
  57. [77] Giles, C. L. and Maxwell, T. Learning, invariance, and generalization in high-order neural networks. Applied Optics, 1987 on Pith
  58. [78] Glasgow, M. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
  59. [79] Gold, H. They thought they were making technological breakthroughs. It was an AI -sparked delusion. CNN, September 2025. URL https://www.cnn.com/2025/09/05/tech/ai-sparked-delusion-chatgpt on Pith
  60. [80] Goldowsky-Dill, N., Chughtai, B., Heimersheim, S., and Hobbhahn, M. Detecting strategic deception with linear probes. PMLR, 2025 on Pith
  61. [81] Goodhart, C. A. E. Problems of monetary management: The U.K. experience. In Papers in Monetary Economics. Reserve Bank of Australia, Sydney, 1975 on Pith
  62. [82] Gorton, L. What would non-linear features actually look like?, 2026 on Pith
  63. [83] Grant, S., Gillioz, V., Ward, J., and McGrath, T. Shifting the gradient: Understanding how defensive training methods protect language model integrity. A r X iv e-print , 2026 on Pith
  64. [84] Gupta, A., Saini, S., and Narayanan, P. Concept distillation: leveraging human-centered explanations for model improvement. Advances in Neural Information Processing Systems, 2023 on Pith
  65. [85] Gupta, R. and Jenner, E. RL-Obfuscation : Can language models learn to evade latent-space monitors? A r X iv e-print , 2025 on Pith
  66. [86] Gurnee, W. and Tegmark, M. Language models represent space and time. In Proceedings of the International Conference on Learning Representations (ICLR), 2024 on Pith
  67. [87] Gurnee, W., Ameisen, E., Kauvar, I., Tarng, J., Pearce, A., Olah, C., and Batson, J. When models manipulate manifolds: The geometry of a counting task. A r X iv e-print , 2026 on Pith
  68. [88] Han, T., Srinivas, S., and Lakkaraju, H. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
  69. [89] Haufe, S., Meinecke, F., G \"o rgen, K., D \"a hne, S., Haynes, J.-D., Blankertz, B., and Bie mann, F. On the interpretation of weight vectors of linear models in multivariate neuroimaging. NeuroImage, 87: 0 96--110, 2014. doi:10.1016/j.neuroimage.2013.10.067 on Pithdoi:10.1016/j.neuroimage.2013.10.067
  70. [90] Hendrycks, D., Carlini, N., Schulman, J., and Steinhardt, J. Unsolved problems in ml safety. A r X iv e-print , 2021 on Pith
  71. [91] Hermann, K. and Lampinen, A. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS), 2020 on Pith
  72. [92] Hermann, K. L., Mobahi, H., Fel, T., and Mozer, M. C. On the foundations of shortcut learning. Proceedings of the International Conference on Learning Representations (ICLR), 2023 on Pith
  73. [93] Hewitt, J. Distinguishing forms from estimators in interp, or, why steer by difference-of-means? https://www.cs.columbia.edu/ johnhew/forms-from-estimators.html, 2026 on Pith
  74. [94] Hindupur, S. S. R., Orgad, H., Fel, T., and Ba, D. When models don't manipulate manifolds: The geometry of a comparison task. A r X iv e-print , 2026 on Pith
  75. [95] Hollinsworth, O. J., Dombrowski, A.-K., Adam-Day, S., Gleave, A., and Cundy, C. Scaling trends for lie detector oversight in preference learning. In Trustworthy AI for Good (AI4GOOD) Workshop @ ICML 2026, 2026 on Pith
  76. [96] Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural Networks, 1989 on Pith
  77. [97] Howard, J. Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019. URL https://github.com/fastai/imagenette on Pith
  78. [98] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI Conference on Artificial Inte... on Pith
  79. [99] Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
  80. [100] Jang, E., Falck, D., Braun, J., Kirch, N., Menon, A., Moodley, P., Emmons, S., Zimmermann, R. S., and Lindner, D. Exploration hacking: Can llms learn to resist rl training? A r X iv e-print , 2026 on Pith

The first 100 of 300 references are listed here. All of them are in this paper's citation record.

Cited by 1 Pith paper

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment