Pith. sign in

REVIEW 3 major objections 10 minor 16 references

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

T0 review · 3 major / 10 minor · reviewed 2026-07-30 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned Activation Oracles can become concept-specific anti-readers that fail to report the hidden concept present throughout their own training.

desk verdict Solid own-vs-cross anti-reading result in a Taboo AO setup; the behavioral finding is real, the readout-side story is only half-nailed. read the letter →

arxiv 2607.23379 v1 pith:26VRZ3IW submitted 2026-07-25 cs.CL cs.AI

Tobias Bersia , Tatiana Gaintseva This is my paper
classification cs.CLcs.AI
keywords activationoraclesinterpretabilitylatentknowledgefine-tuningreadoutfailureTaboowordguessingconcept-specificblindspots
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Activation Oracles are language models trained to answer natural-language questions about another model’s internal activations, offered as a flexible way to read information that is present inside a model but missing from its text. This paper shows that those oracles are not neutral meters: when trained on a subject that hides a known concept while still representing it internally, the oracle can selectively lose the ability to name that same concept. In a controlled Taboo Word Guessing setup, own fine-tuned oracles underperform both a base oracle and oracles trained on other concepts, even though the target remains linearly decodable inside the oracle and recoverable by simpler readouts. The failure localizes to the oracle’s readout path rather than to missing information in the subject. The practical point is that behavioral secrecy, representation-level decodability, and oracle-verbalizability can come apart, so a learned interpretability interface can acquire blind spots induced by its training loop.

What carries the argument

Concept-specific anti-reading in Activation Oracles: LoRA-trained language models that answer questions about injected residual-stream activations, evaluated in own-versus-cross comparisons on Taboo subjects, with probes, LogitLens ranks, and mid-layer LoRA ablations separating representation decodability from verbalization.

What would settle it

Train own and cross Activation Oracles on the same Taboo subjects and check whether own-oracle recovery of the matched concept stays below base and cross oracles while linear probes still decode the concept inside the own oracle and ablating mid-layer LoRA updates restores target probability.

Watch

Extended reading notes

Core claim

Contrary to the expectation that an Activation Oracle trained on a fine-tuned subject becomes a specialist reader of that subject’s hidden concept, fine-tuned oracles can become concept-specific anti-readers: they selectively fail to recover the concept that was persistently present during their own training, while often retaining recovery of other concepts. The target remains decodable from subject activations and inside the oracle itself; suppression arises in the oracle readout pathway.

Load-bearing premise

That a controlled Taboo Word Guessing setup on one backbone with a few hidden nouns is a fair enough stand-in for the hidden-information settings where activation oracles would be used as auditors.

Editorial extensions

If this is right

  • A learned activation reader can acquire reporting policies that omit concepts present in both subject and oracle representations.
  • Checking that information is decodable in the subject is not enough to trust an oracle’s answers about it.
  • Own-versus-cross training comparisons become a necessary reliability test for activation-to-language interfaces.
  • Blind spots can follow the oracle’s training concept even when multiple hidden concepts sit in the same subject activations.
  • Auditing tools that are themselves fine-tuned may need readout-path diagnostics, not only subject-side probes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any learned monitor trained only on activations from a model that systematically conceals a feature may quietly learn not to name that feature.
  • Similar anti-reading could appear in other activation-to-text interfaces if training repeatedly pairs the same latent with non-disclosure answers.
  • Safety evaluations that rely on fine-tuned elicitors may need held-out concept suites and cross-trained controls before claiming coverage.
  • If mid-layer readout updates mediate the failure, interventions that freeze or diversify that path might reduce concept-specific omission.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 10 minor

Summary. The paper studies Activation Oracles (AOs) — LLMs trained to answer natural-language questions about another model's residual-stream activations — in a controlled Taboo Word Guessing model organism. Subjects (Qwen3-8B LoRA adapters) are fine-tuned to internally use one of five hidden concepts while avoiding disclosure, under cooperative/strict protocols and two mixture strengths. The central finding is that an AO fine-tuned on a Taboo subject does not become a specialist reader of that subject's hidden concept; instead it becomes a concept-specific anti-reader, recovering its own training concept worse than a base AO and worse than cross FT-AOs trained on other concepts (diagonal suppression in the concept×concept heatmap, Fig. 6). The effect persists for multi-concept (leaf_moon) subjects (§8), ruling out a checkpoint-identity account of the blind spot. Mechanistically, the authors argue the failure is readout-side: the concept remains linearly decodable from AO hidden states (App. D), target rank degrades in late AO layers (Fig. 8), and zeroing LoRA updates in layers L12–23 partially restores target probability (Fig. 9, Table 7).

Significance. If the result holds, it is a useful and well-controlled cautionary result for the growing line of activation-to-language interfaces (AOs, NLAs): a learned reader can acquire concept-specific reporting blind spots induced by its own training distribution, so representation-level decodability, behavioral leakage, and AO-verbalizability must be evaluated separately. Strengths worth naming: the behavioral anti-reading claim is supported by distributional metrics (P(c⋆), rank, np≈300 per cell) that do not depend on the LLM judge; the design includes genuine negative controls (OFFTOPIC, SAMETEXT teacher-forced neutral text) and a multi-concept control that cleanly separates concept-specific from subject-specific failure; the own/cross/base three-way comparison is a non-tautological external contrast; and the appendices ship unusually complete reproducibility detail (prompt templates, seed schedules, LoRA hyperparameters, per-cell tables). The phenomenon is falsifiable and the claims are mostly hedged appropriately. Generality is limited by construction (single backbone, five concrete nouns, LoRA-only, synthetic training loop), which the authors state plainly in App. A.

major comments (3)
  1. [§7.1 / Appendix D.1–D.2, Table 2] The AO-internal probe evidence has the exact confound the authors themselves flag for subject-level probes in §5.2 and Appendix C ('the cross-validation split is not grouped by checkpoint... may reflect both concept-correlated structure and stable checkpoint-level signatures'), but the caveat is not carried over. The AO-internal probes pool hidden states from injections of only 10 subject checkpoints (2 per concept) with 30 prompts each, using ungrouped stratified 5-fold CV. Examples from the same checkpoint therefore appear in train and test folds, and with only two checkpoints per concept label, 'decodes the concept' and 'decodes which of two checkpoint distributions was injected' are nearly indistinguishable. This matters because §7.1 and D.2 draw a strong directional conclusion from these probes: 'anti-reading is not well explained as a failure to extract the hidden concept from the
  2. [§7.3 / Appendix H, Table 7, Fig. 9] The ablation evidence is presented as causal localization of concept-specific suppression, but three gaps weaken it. (i) Only own-concept P(c⋆) is measured after ablation; there is no control showing that ablating L18–23 does not equally boost non-own concepts. Since ablation partially resets the FT-AO toward base behavior, and the base AO reads everything well, a generic partial reset predicts the observed restoration without any concept-specific mechanism. Reporting post-ablation probabilities for the other four concepts (or running the same ablation on cross FT-AO/subject pairs) would isolate the concept-specific component. (ii) The restoration is relative to already-suppressed baselines: e.g., leaf-FT baseline P(c⋆)=0.143 with peak ρ=2.01 gives absolute P(c⋆)≈0.29, still far below the base AO's 0.63 (Table 26). 'Restoration' should be reported in absolute terms alongside ρ. (iii) n=2
  3. [§6.1, Fig. 5] A substantial part of the own-FT-AO degradation is generic fine-tuning degradation rather than concept-specific anti-reading: for cooperative α=1.0, cross FT-AOs already drop from 94% (base) to 71% exact recovery, with own FT-AOs at 35%. The concept-specific effect is the own-vs-cross gap, not the base-vs-own gap, and the SAMETEXT regime drives much of it (Table 22: cross 0–35%, own 0–1%, while base retains 34–98%). The aggregate framing in §6.1 ('exact recovery drops from 94% for the base AO and 71% for cross FT-AOs to 35% for own FT-AOs') is accurate but the text would be clearer if it explicitly decomposed the total drop into a generic FT term and a concept-specific term, and stated which regimes carry the concept-specific component. This is a framing/quantification issue, not a correctness error, but it is load-bearing for how large the claimed phenomenon is.
minor comments (10)
  1. [Abstract] Typo: 'becomeconcept-specific anti-readers' (missing space).
  2. [Appendix D.3, Table 3 caption] Caption is truncated mid-sentence: 'Median LogitLens rank of the target token when the AO's own Qwen3-8B' — the sentence is incomplete.
  3. [Appendix G.3, Table 6] The SAMETEXT row labels 'moon-FT (own)' for an evaluation on the leaf subject; moon-FT is a cross AO for the leaf subject, so the '(own)' tag appears to be an error. Please check.
  4. [Appendix E.1 (SAMETEXT sampling)] SAMETEXT stores three identical copies per neutral text so that 'downstream aggregation uses the same nominal capture count.' These copies are not independent samples; please state explicitly how they are treated in the np≈300 counts and in the bootstrap CIs of Fig. 5, to rule out pseudoreplication.
  5. [Appendix F.2] Substring exact match counts 'leafy' and 'books' as exact recoveries. This is mostly harmless given the Sonnet tier rubric, but worth one sentence noting the direction of the bias (it can only inflate exact recovery, including for own FT-AOs, so it works against the anti-reading claim — actually a point in the paper's favor).
  6. [Appendix F.3] Judge sample size nj=25 per cell gives 4% granularity on exact/semantic recovery; all judge-based percentages are multiples of 4. The distributional metrics (np≈300) mitigate this, but the resolution limit should be noted where judge numbers are quoted to the percent.
  7. [Tables 23–24 (App. J.4)] The subscript notation (e.g., '[48,51] 2') indicating fewer than four cross AOs is only defined in the J.4 preamble; several strict α=0.5 own cells (leaf, moon) are missing entirely (dashes). Please explain why those FT-AOs were unavailable and define the subscript at first use.
  8. [Appendix B.3] 'activations from layers at roughly 25, 50 and 75 percents of the AO stack' — grammar; also worth stating explicitly that the fixed evaluation layer ℓ=18 of 36 corresponds to the 50% training injection point, so evaluation is in-distribution for the injection layer.
  9. [Figure 1] The caption describes panels (a) and (b) with left/right sub-panels, but the layout and what is plotted in (b) (which metric, which aggregation) are not specified; please make the figure self-contained.
  10. [References] 'nostalgebraist (2020)' is cited with lowercase and as a blog post — acceptable, but consider consistent formatting; also double-check the Karvonen et al. arXiv identifier (abs/2512.15674).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical own-vs-cross contrasts with independently defined metrics, not definitional or fitted predictions.

full rationale

This is a controlled empirical ML paper. The central claim—that fine-tuned Activation Oracles become concept-specific anti-readers—is established by comparing base AOs, own FT-AOs, and cross FT-AOs on recovery metrics (exact/semantic recovery, P(c⋆), target rank) that are defined operationally from AO outputs and are not algebraic rearrangements of the training objective. Own-vs-cross and multi-concept (leaf_moon) designs are external experimental contrasts, not tautologies. Representation-level probes, LogitLens ranks, and layer-range LoRA ablations are separate diagnostics; none redefine the anti-reading effect as its own input. Methodological inheritance from Karvonen et al. (AO recipe) and Cywinski et al. (Taboo organisms) is ordinary prior-work setup, not load-bearing self-citation of an unverified uniqueness theorem by the same authors. There is no fitted parameter relabeled as a first-principles prediction, no self-definitional loop, and no renaming of a known law as a derived result. Residual concerns about probe confounds or ablation interpretation are correctness/mechanism risks, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The result is experimental, not axiomatic. Load-bearing commitments are modeling and measurement choices: Taboo as a hidden-knowledge organism, AO architecture/injection following Karvonen et al., fixed readout metrics, and the interpretation that mid-layer LoRA ablation restoring P(c⋆) localizes anti-reading to the readout pathway rather than removing unrelated capacity.

free parameters (4)
  • AO activation injection layer ℓ and steering coefficient λ = ℓ=18, λ=1.0
    Fixed to ℓ=18 and λ=1.0 following the prior AO recipe; the anti-reading measurements are conditional on this injection interface.
  • Subject and AO LoRA hyperparameters = subject r=32 α=64; AO r=64 α=128; subject LR 1e-4; AO LR 1e-5; 1 epoch
    Rank, alpha, dropout, LR, epoch count, and module set are chosen by recipe rather than derived; they define the trained objects under study.
  • Taboo mixture strength α and protocol split = α∈{0.5,1.0}; cooperative and strict
    α∈{0.5,1.0} and cooperative vs strict protocols are design knobs that change leakage and readability; claims aggregate over these choices.
  • Sonnet semantic-recovery judge sample size and tier rubric = nj=25; tiers 0–3
    Semantic recovery depends on an external LLM judge with a hand-specified 0–3 rubric and nj=25 per cell.
assumptions (5)
  • domain assumption Residual-stream activations at a fixed layer, injected additively into a LoRA copy of the same backbone, are a valid interface for testing activation-to-language readability.
    Inherited from Karvonen et al. 2025 and used throughout §§3–4; without it the AO evaluations are undefined.
  • domain assumption Taboo Word Guessing subjects are useful model organisms for behaviorally hidden but internally represented concepts.
    Stated in §1 and §4.1 following Cywinski et al. 2025; the reliability moral for interpretability interfaces depends on this proxy.
  • domain assumption Linear probe accuracy and LogitLens/Δ-LogitLens ranks are informative diagnostics separating representation-level decodability from verbalization.
    Used in §§5.2 and 7; standard interpretability practice, but probe CV is not checkpoint-grouped.
  • domain assumption Zeroing contiguous FT-AO LoRA ranges at inference causally localizes where anti-reading is mediated, holding the rest of the oracle fixed.
    §7.3 and Appendix H; restoration of P(c⋆) is interpreted as evidence that mid-layer updates implement suppression rather than only generic capacity.
  • ad hoc to paper Substring exact match plus Sonnet tier≥2 are adequate operationalizations of AO recovery.
    Appendix F defines the metrics; semantic scores inherit judge model biases.
invented entities (1)
  • Concept-specific anti-reader (own FT-AO blind spot) independent evidence
    purpose: Name the observed failure mode where an oracle trained on concept c⋆ selectively under-verbalizes c⋆.
    Descriptive label for an empirical pattern (diagonal suppression in own-vs-cross evaluations), not a new physical entity. Independent handle is the measurable own/cross gap and ablation restoration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles." pith.science (2026). https://pith.science/paper/26VRZ3IW

@misc{pith2026260723379,
  author       = {Pith},
  title        = {Pith review of: When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26VRZ3IW}},
  note         = {Machine review of arXiv:2607.23379}
}
read the original abstract

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

Figures

Figures reproduced from arXiv: 2607.23379 by the authors.

Figure 1
Figure 1. Fine-tuned Activation Oracles become concept-specific anti-readers. a recent and particularly flexible interface for this kind of analysis: instead of training a task-specific probe, an AO is a language model trained to answer natural-language questions about another model’s internal activations. This turns activation analy￾sis into an interactive question-answering interface. In principle, such an interface could m… view at source ↗
Figure 3
Figure 3. ∆-LogitLens decodability of Taboo concepts from subject residual-stream activations at layer L18. Lower rank indicates greater accessibility. The dashed line shows the shuffled-label baseline. For each protocol, mixture strength, concept, and regime, we compute δ = 1 nft X x h FT 18 (x) − 1 nbase X x h base 18 (x), and rank the target token c ⋆ after projecting δ through the subject model’s own output head. This ∆-L… view at source ↗
Figure 4
Figure 4. Base-AO recovery of hidden Taboo concepts [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Concept-specific blind spots in FT-AOs. Rows [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Internal decodability versus output-head ac [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 10
Figure 10. Figure 10: FT-AOs evaluated on multi-concept Taboo subjects containing both leaf and moon. Each cell re￾ports mean P(c ⋆ ) under the AO output distribution. Red borders mark cases where the queried target matches the FT-AO’s own training concept. concepts are present in the same…
Figure 9
Figure 9. Figure 9: Layer-range ablations localize the anti-reading [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 11
Figure 11. Figure 11: Cross-regime transfer of subject-activation concept probes. Each heatmap corresponds to a subject [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 2 linked inside Pith

  1. [1]

    Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers , journal =

    Adam Karvonen and James Chua and Cl. Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers , journal =. 2025 , url =. doi:10.48550/ARXIV.2512.15674 , eprinttype =. 2512.15674 , timestamp =

    doi:10.48550/arxiv.2512.15674 2025

  2. [2]

    Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences , journal =

    Julian Minder and Cl. Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences , journal =. 2025 , url =. doi:10.48550/ARXIV.2510.13900 , eprinttype =. 2510.13900 , timestamp =

    doi:10.48550/arxiv.2510.13900 2025

  3. [3]

    CoRR , volume =

    Bartosz Cywinski and Emil Ryd and Senthooran Rajamanoharan and Neel Nanda , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.14352 , eprinttype =. 2505.14352 , timestamp =

    arXiv doi:10.48550/arxiv.2505.14352 2025

  4. [4]

    2020 , month = aug, howpublished =

    nostalgebraist , title =. 2020 , month = aug, howpublished =

    2020

  5. [5]

    Manning , editor =

    John Hewitt and Christopher D. Manning , editor =. A Structural Probe for Finding Syntax in Word Representations , booktitle =. 2019 , url =. doi:10.18653/V1/N19-1419 , timestamp =

    doi:10.18653/v1/n19-1419 2019

  6. [6]

    Scaling and evaluating sparse autoencoders , booktitle =

    Leo Gao and Tom Dupr. Scaling and evaluating sparse autoencoders , booktitle =. 2025 , url =

    2025

  7. [7]

    The Fourteenth International Conference on Learning Representations , year=

    Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences , author=. The Fourteenth International Conference on Learning Representations , year=

  8. [8]

    Advances in Neural Information Processing Systems , year =

    Locating and Editing Factual Associations in GPT , author =. Advances in Neural Information Processing Systems , year =

Show all 16 references
  1. [9]

    2026 , howpublished =

    Natural Language Autoencoders Produce Unsupervised Concept Descriptions in LLMs , author =. 2026 , howpublished =

    2026

  2. [10]

    2021 , howpublished =

    Eliciting Latent Knowledge , author =. 2021 , howpublished =

    2021

  3. [11]

    NeurIPS Workshop on Machine Learning and Computer Security , year =

    BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain , author =. NeurIPS Workshop on Machine Learning and Computer Security , year =

  4. [12]

    IEEE Symposium on Security and Privacy , year =

    Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks , author =. IEEE Symposium on Security and Privacy , year =

  5. [13]

    arXiv preprint arXiv:2408.12798 , year =

    BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models , author =. arXiv preprint arXiv:2408.12798 , year =

    arXiv

  6. [14]

    Yonatan Belinkov , title =. Comput. Linguistics , volume =. 2022 , url =. doi:10.1162/COLI\_A\_00422 , timestamp =

    2022 doi

  7. [15]

    The Twelfth International Conference on Learning Representations,

    Fred Zhang and Neel Nanda , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  8. [16]

    Enhancing Chat Language Models by Scaling High-quality Instructional Conversations , booktitle =

    Ning Ding and Yulin Chen and Bokai Xu and Yujia Qin and Shengding Hu and Zhiyuan Liu and Maosong Sun and Bowen Zhou , editor =. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations , booktitle =. 2023 , url =. doi:10.18653/V1/2023.EMNLP-MAIN.183 ,...

    2023 doi

Pith tools

Reviewed July 30, 2026 · model on record in the stance chip above.