Pith. sign in

REVIEW 3 major objections 6 minor 27 references

Prepending one fixed natural-language instruction extracts model-relative reasoning features from activation changes that track the model’s own judgments and rank better training data.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:49 UTC pith:JLDQHHA6

load-bearing objection Clean unsupervised prefix-shift method for model-relative features; the 94.7% transfer number is real but post-hoc on 19 shuffles and needs a held-out check. the 3 major comments →

arxiv 2607.04222 v1 pith:JLDQHHA6 submitted 2026-07-05 cs.AI

Unsupervised Features Mining via Activation Geometry

Amit LeVi , Elad David , Max Fomin This is my paper
classification cs.AI
keywords activation geometryinterpretabilityreasoning featuresactivation steeringtraining data selectionlarge language modelsunsupervised feature miningprompt injection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Many interpretability methods start from human-labeled examples of a concept and then hunt for that concept inside a model. This paper instead defines the reasoning process with a fixed natural-language instruction Q, prepends it to every input p, and mines the resulting change in residual-stream activations at a single readout: m(Q∥p) − m(p). The extracted features, called Mining via Activation Geometry (MAG), predict the model’s own yes/no verdict better than raw activations, even on rows where the model disagrees with dataset labels. The same shift is often well approximated by one linear direction that can flip matched-format binary decisions under calibrated steering, and the geometry of these features ranks candidate training datasets for prompt-injection probes far more accurately than ordinary activation similarity. A reader cares because the method needs no human labels at selection time and turns an unsupervised activation difference into a practical signal for auditing judgment and choosing data.

Core claim

The paper establishes that model-relative reasoning features can be extracted unsupervised by measuring how a fixed natural-language prefix changes residual-stream activations at one readout point, that these features track the model’s own judgments rather than external labels, that many of them are approximately linear and steerable, and that the resulting geometry selects the best candidate training dataset for a prompt-injection probe with 94.7% Top-1 and 100% Top-2 accuracy where raw activation similarity is essentially uninformative.

What carries the argument

Mining via Activation Geometry (MAG): the prefix-induced shift Δ_Q(p) = m(Q∥p) − m(p) at the last token of the final residual stream, summarized by eight operators (Direct, Prefixed, InputDelta, Interaction, Answered, Verdict, QuestionDelta, FewShot) that induce contrast directions and a normalized reconstruction error ε_Q measuring how well a single linear direction rebuilds the prefix effect.

Load-bearing premise

The strongest transfer predictors were selected and scored on the same small set of shuffles used for evaluation, so the reported ranking accuracy may not hold on truly independent partitions.

What would settle it

Re-run the six-way candidate-dataset ranking on a fresh collection of independent shuffles never used to choose the operator–metric combinations; if the best class-conditional triple’s Top-1 accuracy collapses toward the random baseline of about 17%, the transfer claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • MAG features can audit what a model judges about a concept without trusting its verbal self-report or external labels.
  • Class-mean directions derived from MAG can flip matched-format yes/no verdicts via calibrated activation steering.
  • MAG geometry can rank which unlabeled candidate dataset to add to a training pool, outperforming raw activation similarity for prompt-injection probes.
  • Context-loaded prefixes induce directions that are at least partly distinct from simpler named concepts and still generalize to held-out examples.
  • Reconstruction error ε_Q predicts steerability: low error marks more linear, steerable features; near-one error marks entrenched verdicts such as prompt-injection self-labels.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If MAG reliably tracks model-relative judgment, the same geometry may surface alignment-faking or evaluation-awareness features that never appear in outputs.
  • The operator family could select training data for other safety probes without new labels, provided class-conditional centroids remain stable at modest sample sizes.
  • ε_Q itself could serve as a pre-intervention diagnostic for whether a reasoning feature is steerable or entrenched.
  • An adversary with only public chat templates could mine and inject safety-related directions at deployment, making MAG part of the model’s attack surface as well as a defensive tool.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Mining via Activation Geometry (MAG), an unsupervised method that extracts model-relative reasoning features by prepending a fixed natural-language instruction Q to every input p and reading the residual-stream shift Δ_Q(p)=m(Q∥p)−m(p) at the final-token final-block readout. Eight operators are defined (Direct, Prefixed, Answered, InputDelta, QuestionDelta, Interaction, Verdict, FewShot). On Llama-3.1-8B-Instruct and Gemma-2-9B-it (Qwen-2.5-32B as stress), the authors show: (i) prefixed/delta activations predict the model’s own yes/no self-label y_M better than raw activations and track y_M on label-disagreement rows; (ii) the mean prefix shift is approximately linear at the final readout (ε_Q in Eq. 3), with concept prefixes more linear than prompt-injection; (iii) class-mean directions steer matched-format yes/no verdicts on neutral objects; (iv) a context-loaded “Bob” prefix induces a direction partly distinct from desert/ocean; and (v) MAG geometry ranks candidate training datasets for prompt-injection probes, with a class-conditional triple reported at 94.7% Top-1 / 100% Top-2 on 19 valid shuffles (Table 7), versus near-chance raw centroid cosine.

Significance. If the core MAG claims hold under tighter evaluation, the work offers a practical, label-free primitive for inducing and reading model-relative reasoning features without sparse autoencoders, circuit search, or human concept exemplars. The readability, linearity (ε_Q), matched-format steering, and composition experiments are cleanly designed (LODO/LOO, multiple models, explicit negative control on PI), and the dual-use discussion is appropriately cautious. The transfer-selection application is the most practically consequential claim: ordinary activation similarity is shown to be nearly uninformative for dataset choice, while MAG geometry recovers a usable ranking signal. That result would matter for safety-probe training and distribution-shift auditing if validated without post-hoc operator/metric selection on the evaluation shuffles. Strengths include explicit reconstruction error, calibrated steering strength α(τ), disagreement-subset analysis, and transparent limitations on open-ended generation and PI non-linearity.

major comments (3)
  1. [Abstract; §4; Table 7; Appendix C.1; §5 Limitations] Abstract and Table 7 lead with 94.7% Top-1 / 100% Top-2 for the class-conditional triple {Y3+cos_ben, Y5+cos_ben, Y8+cos_mal}, but Appendix C.1 and §5 state that this figure is obtained after exhaustive search over 64 singles + 1,568 pairs + 220 triples, evaluated on the same 19 of 50 shuffles for which all required class-conditional centroids exist. Full-coverage pairs (n=50) fall to 60–62% Top-1. The authors correctly flag optimism risk in Limitations, yet the abstract and headline table still present the selected triple as the main quantitative result without a nested or held-out partition. Because the abstract’s contrast (“ordinary activations almost unrelated … RFD-based similarity achieves 94.7%”) rests on precisely this selected quantity, an independent shuffle partition (or nested selection) is load-bearing for the transfer-utility claim.
  2. [§4; Eq. (4)–(5); §5 Limitations] The transfer experiment uses a single fixed downstream classifier, feature space, and training protocol (§5 Limitations). Combined with free choice among eight operators and eight metrics, it is unclear whether MAG geometry predicts transfer in general or only for this probe setup. At minimum, report sensitivity to a second probe architecture or training recipe, or reframe the claim as “transfer under this fixed probe protocol” rather than general training-data selection.
  3. [Abstract; §3.3; Table 3; Table 4; Appendix B.3] Matched-format class-mean steering flips 11–12/12 neutral yes/no verdicts (Table 4), but open-ended generation shows weak or metric-mismatched effects (Appendix B.3–B.4), and the PI direction flips 0/12 neutrals with ε_Q≈0.97 (Table 3, E5). The paper treats low ε_Q as predicting steerability, which is useful, but the abstract’s phrasing that linear representation “can change the LLMs’ decisions through activation steering” overgeneralizes beyond the matched-format probe. The main claim should be scoped to the conditions where control is demonstrated, and the PI negative control should be elevated in the abstract/conclusion so readers do not infer uniform steerability of all MAG features.
minor comments (6)
  1. [Abstract; §3] Abstract writes m(Q | p)−m(p); the body consistently uses concatenation Q∥p. Align notation throughout.
  2. [Figure 1] Figure 1 is dense and useful but hard to parse in greyscale; consider a simplified schematic of the eight operators plus a separate results panel.
  3. [Table 2; Appendix B.1] Table 2 reports the best operator φ⋆ per cell; a full operator matrix (or appendix pointer already partially present in Table 9) in the main text would reduce selection-of-best concern for readability.
  4. [Abstract] “RFD” appears in the abstract without definition; define on first use (reasoning-feature direction / MAG geometry).
  5. [§3.1; Table 9] Qwen PI row is correctly flagged as degenerate (15/1800 self-malicious); consider moving it fully to appendix so main readability claims stay on non-degenerate cells.
  6. [Abstract; throughout] Minor prose issues: “we found that some features are more linearly represented and some less” (abstract); “Mining via Acti-vation Geometry” line break; occasional missing spaces around operators.

Circularity Check

1 steps flagged

No load-bearing circular derivation: MAG is model-relative by design; transfer ranking is empirical (with acknowledged selection optimism), not forced by construction.

specific steps
  1. self definitional [§3 Eq. 1; Operators; §3.1; §4 class-conditional metrics]
    "y_M(p)=1[Pr_θ(yes|Q∥p)>Pr_θ(no|Q∥p)] ... Contrast sets P−={p:y_M(p)=1} and P+={p:y_M(p)=0} are induced from y_M throughout, never from external annotation ... each ϕ induces a contrast direction v_ϕ,Q=... defined entirely from y_M ... class-conditional centroid cosine similarities on the model-self-labelled malicious and benign subsets (cos_mal and cos_ben)."

    y_M is both the quantity MAG is scored against and the partition used to build class-mean directions and class-conditional transfer scores. This is self-referential structure, but not a reduction by construction: LODO/LOO prediction, ϵ_Q reconstruction, neutral-object steering, and ranking candidates without observing Δ remain empirical tests that can fail (and PI/Direct baselines do fail relative to MAG).

full rationale

The paper’s chain is self-contained and does not reduce claimed results to their inputs by equation identity. y_M is defined from the model’s constrained yes/no under Q (Eq. 1) and is then used as the supervision target and for class-mean / class-conditional geometry; that is intentional model-relative design, not a tautology that forces LODO/LOO accuracy, reconstruction error ϵ_Q, steering flips, or transfer Top-k. Readability compares Prefixed/InputDelta to Direct on held-out folds; linearity measures how well a single mean shift reconstructs m(Q∥p)−m(p); steering intervenes on neutral matched-format probes; composition reports cosines and LOO-AUC on held-out objects. Transfer predictors rank candidates by geometric scores s_ϕ,g without observing Δ(C_i,T); raw Direct centroid cosine is near-uncorrelated with transfer while MAG operators recover signal—an empirical contrast, not a definitional identity. The 94.7% Top-1 figure is the best triple after a large search on the same 19 class-conditional shuffles (Appendix C.1; Limitations §5), which is selection bias / optimistic reporting the authors flag, not a fitted parameter renamed as a forced prediction (Eq. X ≠ Eq. Y by construction). Self-citations are motivational context, not uniqueness theorems that forbid alternatives. Score 1 only for the mild dual use of y_M as target and class partition; central claims remain independently testable.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

MAG rests on standard residual-stream readouts and contrastive directions already in the literature, plus a small set of design choices (final-token final-block readout, eight hand-defined operators, self-label y_M, calibrated τ) that are free parameters of the method rather than derived quantities. No new physical entities are postulated.

free parameters (4)
  • steering strength τ = {0, 0.3, 1.0}
    Hand-swept over {0, 0.3, 1.0}; α(τ) scales the injected direction to a fraction of residual norm. Choice of τ determines whether flips occur.
  • choice of eight MAG operators
    Direct, Prefixed, Answered, InputDelta, QuestionDelta, Interaction, Verdict, FewShot are enumerated by hand; which operator wins is data-dependent.
  • readout location (final token, final block) = L final, last token
    Fixed design choice; layer-wise analysis shows final-layer injection is best, but the location itself is not derived.
  • class-conditional triple selection = 94.7% Top-1 on 19 shuffles
    Best triple {Y3+cos_ben, Y5+cos_ben, Y8+cos_mal} chosen after evaluating many combinations on the same shuffles used for reporting.
axioms (3)
  • domain assumption Residual-stream activation at last token of final block is a sufficient readout for the induced reasoning feature.
    Stated in §3; layer-wise results support it empirically but do not prove sufficiency for all features.
  • domain assumption Model self-label y_M under constrained yes/no is the correct target for ‘model-relative’ features.
    Definition (1); disagreement analysis treats agreement with y_M as success.
  • ad hoc to paper Prefix-induced shift Δ_Q(p) = m(Q∥p) − m(p) isolates the reasoning feature of interest.
    Core definition of MAG; the eight operators are attempts to further isolate parts of this shift.
invented entities (2)
  • MAG operators (eight ϕ) no independent evidence
    purpose: Vector summaries that isolate different parts of the prefix-induced activation change for feature mining.
    Hand-designed family; no independent existence outside the paper’s experimental protocol.
  • RFD / reasoning-feature direction v_Q, u_Q no independent evidence
    purpose: Mean or class-mean prefix shift used for reconstruction, steering, and transfer scoring.
    Defined as empirical average of Δ_Q; utility is demonstrated but the entity is the method’s output, not a pre-existing object.

pith-pipeline@v1.1.0-grok45 · 23926 in / 2925 out tokens · 29731 ms · 2026-07-11T20:49:59.482250+00:00 · methodology

0 comments
read the original abstract

Interpretability methods aim to reveal the features represented inside large language models (LLMs). Many existing methods begin with labeled examples of a human-defined concept that may reflect human biases, and then identify how that concept is represented within the model, for example in its activation space or through other decomposition methods. We introduce \emph{Mining via Activation Geometry} (MAG), a simple unsupervised framework for extracting reasoning features from model activations by prepending the same natural-language instruction $Q$ to every input $p$, where $Q$ defines the reasoning feature of interest, such as ``Can this object be found in the desert?'' or ``Is this prompt malicious?'' We measure how the instruction changes the model's internal representation using $m(Q \mid p) - m(p)$ at a single readout point. We explore eight different MAGs. The extracted reasoning features predict the models' own world understanding and judgment, can be approximated into a single activation direction, we found that some features are more linearly represented and some less, this linear representation, which is vector steering, can change the LLMs' decisions through activation steering by injecting reasoning features. Finally, we use the same method to select the best training datasets for prompt-injection classifier probes: while similarity between ordinary activations is almost unrelated to downstream performance, RFD-based similarity achieves $94.7\%$ Top-1 and $100\%$ Top-2 accuracy.

Figures

Figures reproduced from arXiv: 2607.04222 by Amit Levi, Elad David, Max Fomin.

Figure 1
Figure 1. Figure 1: A fixed instruction Q is prepended to every input p; A(x) = m(x) denotes the residual-stream readout at the last token of the final block, and ∆Q(p) = m(Q ∥ p) − m(p) is the prefix-induced shift. (E1) The prefixed activation A(Q ∥ p) predicts the model’s own verdict y M better than the raw activation A(p) (desert LODO ROC 0.91→0.94). (E2) On rows where the dataset label disagrees with y M, the classifier s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 15 linked inside Pith

  1. [1]

    arXiv preprint arXiv:2507.14805 , year=

    Subliminal learning: Language models transmit behavioral traits via hidden signals in data , author=. arXiv preprint arXiv:2507.14805 , year=

    Pith/arXiv arXiv

  2. [2]

    arXiv preprint arXiv:2108.07258 , year=

    On the opportunities and risks of foundation models , author=. arXiv preprint arXiv:2108.07258 , year=

    Pith/arXiv arXiv

  3. [3]

    Proceedings of the 2022 ACM conference on fairness, accountability, and transparency , pages=

    Taxonomy of risks posed by language models , author=. Proceedings of the 2022 ACM conference on fairness, accountability, and transparency , pages=

    2022

  4. [4]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  5. [5]

    arXiv preprint arXiv:2412.14093 , year=

    Alignment faking in large language models , author=. arXiv preprint arXiv:2412.14093 , year=

    Pith/arXiv arXiv

  6. [6]

    arXiv preprint arXiv:2510.20487 , year=

    Steering Evaluation-Aware Language Models to Act Like They Are Deployed , author=. arXiv preprint arXiv:2510.20487 , year=

    arXiv

  7. [7]

    arXiv preprint arXiv:2410.21514 , year=

    Sabotage evaluations for frontier models , author=. arXiv preprint arXiv:2410.21514 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Silenced biases: The dark side llms learned to refuse , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  9. [9]

    arXiv preprint arXiv:2502.17516 , year=

    A survey on mechanistic interpretability for multi-modal foundation models , author=. arXiv preprint arXiv:2502.17516 , year=

    Pith/arXiv arXiv

  10. [10]

    Designing and interpreting probes with control tasks , author=. Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) , pages=

    2019

  11. [11]

    arXiv preprint arXiv:2501.16496 , year=

    Open problems in mechanistic interpretability , author=. arXiv preprint arXiv:2501.16496 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2211.00593 , year=

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small , author=. arXiv preprint arXiv:2211.00593 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2309.08600 , year=

    Sparse autoencoders find highly interpretable features in language models , author=. arXiv preprint arXiv:2309.08600 , year=

    Pith/arXiv arXiv

  14. [14]

    Advances in Neural Information Processing Systems , volume=

    Refusal in language models is mediated by a single direction , author=. Advances in Neural Information Processing Systems , volume=

  15. [15]

    arXiv preprint arXiv:2310.01405 , year=

    Representation engineering: A top-down approach to ai transparency , author=. arXiv preprint arXiv:2310.01405 , year=

    Pith/arXiv arXiv

  16. [16]

    arXiv preprint arXiv:2308.10248 , year=

    Steering language models with activation engineering , author=. arXiv preprint arXiv:2308.10248 , year=

    Pith/arXiv arXiv

  17. [17]

    arXiv preprint arXiv:2512.24143 , year=

    Activation Steering for Masked Diffusion Language Models , author=. arXiv preprint arXiv:2512.24143 , year=

    arXiv

  18. [18]

    arXiv preprint arXiv:2601.01828 , year=

    Emergent introspective awareness in large language models , author=. arXiv preprint arXiv:2601.01828 , year=

    arXiv

  19. [19]

    International Conference on Learning Representations , volume=

    Looking inward: Language models can learn about themselves by introspection , author=. International Conference on Learning Representations , volume=

  20. [20]

    arXiv preprint arXiv:2603.21396 , year=

    Mechanisms of introspective awareness , author=. arXiv preprint arXiv:2603.21396 , year=

    Pith/arXiv arXiv

  21. [21]

    arXiv preprint arXiv:2401.06102 , year=

    Patchscopes: A unifying framework for inspecting hidden representations of language models , author=. arXiv preprint arXiv:2401.06102 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2512.15674 , year=

    Activation oracles: Training and evaluating llms as general-purpose activation explainers , author=. arXiv preprint arXiv:2512.15674 , year=

    arXiv

  23. [23]

    arXiv preprint arXiv:2602.04894 , year=

    Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software , author=. arXiv preprint arXiv:2602.04894 , year=

    Pith/arXiv arXiv

  24. [24]

    arXiv preprint arXiv:2606.30449 , year=

    Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoring , author=. arXiv preprint arXiv:2606.30449 , year=

    Pith/arXiv arXiv

  25. [25]

    arXiv preprint arXiv:2606.13870 , year=

    Mirage Probes: How Vision Models Fake Visual Understanding , author=. arXiv preprint arXiv:2606.13870 , year=

    arXiv

  26. [26]

    Latent Undertow: How Ordinary Typos Break Probes , author=

  27. [27]

    arXiv preprint arXiv:2502.09755 , year=

    Jailbreak attack initializations as extractors of compliance directions , author=. arXiv preprint arXiv:2502.09755 , year=

    Pith/arXiv arXiv