Pith. sign in

REVIEW 4 major objections 6 minor 16 references

Amplifying a model's reasoning weights past its training point makes it leak secrets it was trained to hide, up to ten times more often.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 11:53 UTC pith:2MOKXRO3

load-bearing objection Solid white-box auditing trick: extrapolating the instruct→reasoning task vector past α=1 raises leak rates on synthetic secret organisms; the 10× framing is soft, but the ablations and two-regime analysis are real. the 4 major comments →

arxiv 2607.08173 v1 pith:2MOKXRO3 submitted 2026-07-09 cs.AI

Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets

Jack Hopkins , Dipika Khullar , Fabien Roger This is my paper
classification cs.AI
keywords overthinkingreasoning amplificationtask vectorsmodel auditingsecret elicitationlayer-wise attenuationchain-of-thought faithfulnessweight-space steering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Black-box audits often miss secrets and misalignment that models have learned but do not volunteer. This paper shows that you can deliberately push a model past ordinary reasoning by taking the weight difference between a non-reasoning instruct model and its reasoning-distilled twin, then scaling that difference by a factor greater than one. The resulting "overthinking" model thinks out loud more aggressively and, in four secret-keeping settings from 2B to 32B parameters, discloses the hidden information far more frequently than the original reasoning model—up to about ten times as often. Layer-wise attenuation keeps outputs coherent long enough for the leak to be readable. Some secrets only surface when the perturbation follows the reasoning direction; others fall to any large weight jolt. The practical claim is that overthinking is a usable white-box auditing primitive that can also be stacked with black-box prefills.

Core claim

Reasoning amplification defined by the overthinking model θ_Oα = θ_M + α(θ_R − θ_M) with α > 1 systematically raises the rate at which models reveal training-acquired secrets, peaking inside an intermediate "auditing window" before coherence collapses, and can improve disclosure by up to roughly 10× relative to the pure reasoning model at α = 1.

What carries the argument

The reasoning task vector τ_reason = θ_R − θ_M, extrapolated beyond α = 1 and modulated by layer-wise coefficients (especially Fisher-weighted and task-magnitude schemes) that protect sensitive layers while amplifying reasoning.

Load-bearing premise

The four synthetic secret-keeping setups (hidden word, implicit gender belief, encoded side-constraint, in-context answer metadata) are representative enough of real hidden information that the measured leakage rates would transfer to naturally arising secrets.

What would settle it

Re-run the same α sweeps and layer-wise methods on a model that acquired a real, non-synthetic secret (for example a capability or policy preference suppressed only by post-training) and check whether disclosure still rises sharply for α > 1 while random perturbations of matched magnitude do not, for the directionally protected secret types.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes overthinking: extrapolating a reasoning task vector τ_reason = θ_R − θ_M beyond the distilled reasoning model via θ_Oα = θ_M + α(θ_R − θ_M) with α > 1, optionally with layer-wise attenuation (freeze last layers, linear decay, Fisher-trace / directional Fisher / task-magnitude weighting). On four secret-keeping settings (in-context MMLU metadata, Taboo word, implicit gender belief, Base64 side-constraint) and Qwen3-VL models from 2B–32B, peak disclosure under amplification is higher than at α = 1 in most cells, sometimes by large relative factors when baselines are near zero; Fisher-weighted methods and metacognitive prefill composition further raise rates for some secrets. The authors also document coherence collapse, language drift, identity confusion, and elevated backtracking, and distinguish directionally protected secrets (Taboo, SSC) from noise-fragile ones (Gender, MMLU).

Significance. If the directional effect generalizes beyond synthetic model organisms and a single model family, the work supplies a practical white-box auditing primitive that is orthogonal to input-side prefills and that can be stacked with them. Strengths include: (i) explicit random layer-norm and global-norm baselines that isolate direction from magnitude for Taboo and SSC; (ii) paired McNemar tests with Holm–Bonferroni correction and Wilson CIs on N=256; (iii) a clear two-regime taxonomy of secrets and a concrete two-stage auditing protocol; (iv) layer-wise Fisher variants that measurably delay coherence collapse. These make the paper a useful contribution to pre-deployment auditing even if absolute leak rates remain modest.

major comments (4)
  1. Abstract and §5.1 / Table 2: the headline “up to 10× more frequently” is driven by near-zero α=1 baselines (e.g. Taboo 8B 3.1%→8.6%, SSC 32B 0%→5.5%, MMLU 8B 1.2%→2.3%). Absolute peak rates under uniform amplification stay single-digit in three of four settings. The claim should be restated with absolute rates and with the qualifier that large relative gains occur mainly when the reasoning baseline is already near floor; otherwise the abstract overstates practical impact.
  2. §5.1 and Fig. 4: for Gender and MMLU, random global-norm (and some freeze) perturbations match or exceed the reasoning vector. The directional mechanism therefore holds only for Taboo and SSC. The abstract’s closing sentence already notes the split, but the central “reasoning amplification” framing and the “10×” language still treat direction as the primary driver across all four settings; the results section should lead with the two-regime taxonomy rather than a uniform directional claim.
  3. Table 3 “OT (best)” / “Prefill + OT (best)”: peak α and coefficient method are selected post-hoc per cell. Without a pre-specified selection rule or nested validation, the reported maxima are optimistically biased. Either fix a single method (e.g. Task Magnitude) a priori or report the full distribution of methods so that the gain is not confounded with selection.
  4. §6.3 Limitations and §4.3: all results are on the Qwen3-VL family (Instruct vs Thinking pairs). The paper itself flags this and lists Llama follow-ups as future work; given that the central claim is about a general auditing technique, at least one cross-family check (or a clear statement that transfer is untested) is load-bearing for the generality asserted in the introduction and conclusion.
minor comments (6)
  1. Definition 3.1 and Fig. 8: the collapse-onset α* is defined via a joint knee in PPL, KL, and drift, but the exact numerical procedure for locating the knee is not stated; a short algorithmic note would aid reproducibility.
  2. Eqs. (3)–(5): normalization of λ_ℓ (especially Task Magnitude) is described only as “normalized so that (1/L)∑λ_ℓ = 1”; clarify whether this is applied before or after the global α scaling and whether negative coefficients are clipped.
  3. Table 2 caption: “Prefill is where we sample from the middle of an assistant message…” is easy to misread; a one-line formal definition of the prefill string used for each setting would help.
  4. Figures 11–12 appear only as captions without full panels in the provided text; ensure the camera-ready includes the actual plots or remove the dangling references.
  5. Related work: the discussion of CoT faithfulness (Lanham, Turpin, Chen et al.) is solid; a brief pointer to concurrent weight-space steering for alignment behaviors (already cited as Fierro & Roger) could be expanded by one sentence on how overthinking differs from sycophancy/refusal vectors.
  6. Typos / notation: “Naïvely” appears as “Na¨ıvely” in one place; “overthinking modelas” missing space in the abstract; θ_Oα vs θ_O_α inconsistency in a few spots.

Circularity Check

1 steps flagged

Empirical weight-space intervention measured by external keyword/judge metrics; no derivation reduces to its own inputs by construction.

specific steps
  1. other [Abstract / §5.1 / Table 2]
    "Our findings suggest that reasoning amplification may surface secrets or unintended behaviors acquired during training up to 10× more frequently than the original reasoning model."

    The 10× multiplier is obtained by selecting, after the fact, the α that maximises the observed leak rate relative to the α=1 baseline. While this is ordinary peak-picking rather than a definitional loop, it is the sole place where the reported headline number is not a pre-specified quantity independent of the data.

full rationale

The paper defines an overthinking model via ordinary task-vector arithmetic (Eq. 1: θ_Oα = θ_M + α(θ_R − θ_M)) and then measures secret-disclosure rates on four held-out model-organism suites with keyword matching and GPT-4.1-mini judges. Leakage is never defined in terms of the amplification factor or the task vector; the reported “up to 10×” figures are simply observed ratios of those external scores. Layer-wise coefficients (Fisher, linear decay, etc.) are also evaluated post-hoc, yet the paper reports the full α-sweep tables rather than claiming a unique theoretical optimum. Self-citations supply the experimental substrate (task vectors, secret-keeping organisms) but do not underwrite the leakage result itself. The only minor residual is the conventional post-hoc selection of peak-α cells when stating the strongest multiplier; that does not make the measured rates circular. Score therefore remains at the low end of the scale.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claim rests on the empirical effectiveness of a linear weight-space intervention. Free parameters are the amplification factor α and the layer-wise coefficients λ_ℓ (chosen by sweep or Fisher heuristics). Background axioms are standard task-arithmetic assumptions and the representativeness of the four synthetic secrets. No new physical entities are postulated; the “overthinking model” is simply a re-parameterisation of existing weights.

free parameters (3)
  • amplification factor α
    Continuous scalar swept in [0,4]; peak leakage is reported at the α that maximises disclosure after the fact, so the reported 10× factor depends on this choice.
  • layer-wise coefficients λ_ℓ
    Either fixed heuristics (freeze last 5/ half, linear decay) or Fisher-derived (trace, directional, task-magnitude); the best method is selected per cell, introducing post-hoc selection.
  • Fisher calibration set size
    256 MMLU samples used to estimate per-layer Fisher; different calibration data would change λ_ℓ.
axioms (4)
  • domain assumption The difference τ_reason = θ_R − θ_M isolates the updates introduced by reasoning distillation and encodes a steerable meta-capability rather than new domain knowledge.
    Stated in §3.1; required for interpreting α > 1 as pure reasoning amplification.
  • domain assumption The four model-organism secrets (Taboo, Gender, SSC, MMLU metadata) are adequate proxies for the hidden information that real-world auditors care about.
    Implicit throughout §1 and §4; if the organisms are too artificial the practical auditing claim weakens.
  • domain assumption GPT-4.1-mini coherence and leak judges are sufficiently accurate proxies for human judgment of disclosure and output quality.
    Used for all quality metrics and some leak detection; no human inter-annotator agreement reported.
  • ad hoc to paper Linear task arithmetic in weight space remains meaningful for α > 1 (i.e., the reasoning direction continues to point toward more deliberation rather than pure noise).
    Core modelling choice of Eq. (1); validated only by the observed leakage and coherence curves.
invented entities (2)
  • overthinking model O_α no independent evidence
    purpose: Name for the extrapolated parameter set θ_M + α(θ_R − θ_M) that is claimed to increase secret disclosure.
    Purely definitional re-parameterisation; no independent ontological claim.
  • directional Fisher F_dir,ℓ and task-magnitude weighting no independent evidence
    purpose: Layer-wise coefficients that concentrate amplification where reasoning signal is strong but output sensitivity is low.
    New formulas (Eqs. 4–5) introduced to delay coherence collapse; their superiority is measured only inside this paper’s leakage metric.

pith-pipeline@v1.1.0-grok45 · 24990 in / 3281 out tokens · 36841 ms · 2026-07-10T11:53:34.092469+00:00 · methodology

0 comments
read the original abstract

Black box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. To better elicit hidden information during an auditing process, we introduce \emph{overthinking}: the process of using reasoning task vectors to amplify the propensity to think out loud of reasoning models. Given the parameters of a non-reasoning instruct model $M$ and reasoning-distilled model $R$, we define the \emph{overthinking model} as $\boldsymbol{\theta}_{\mathcal{O}_\alpha} = \boldsymbol{\theta}_{\mathcal{M}} + \alpha(\boldsymbol{\theta}_{\mathcal{R}} - \boldsymbol{\theta}_{\mathcal{M}})$, where $\alpha > 1$ amplifies reasoning beyond the pure reasoning model $R$. Additionally, we introduce new layer-wise attenuation strategies that selectively amplify reasoning without losing quality and coherence of model outputs. We demonstrate that overthinking models are more likely to reveal hidden information across four experimental settings, across 2B-32B models. Our findings suggest that reasoning amplification may surface secrets or unintended behaviors acquired during training up to $10\times$ more frequently than the original reasoning model. How secrets surface depends on the secret type: some require perturbation along the reasoning direction, while others yield to any sufficiently large weight perturbation.

Figures

Figures reproduced from arXiv: 2607.08173 by Dipika Khullar, Fabien Roger, Jack Hopkins.

Figure 1
Figure 1. Figure 1: Overthinking framework for auditing learned secrets. (a) The reasoning task vector τ reason = θR − θM captures the direction from instruct to reasoning behavior in weight space. The overthinking model θOα = θM + α · τ reason amplifies reasoning beyond training (α > 1). (b) Applied to a model trained to conceal the word “gold,” standard reasoning (α = 1.0) successfully hides the secret, while amplified reas… view at source ↗
Figure 2
Figure 2. Figure 2: Effective reasoning coefficients by layer (32B). Gray shading shows raw task vector magnitude ∥τ (ℓ) reason∥. Colored lines show weighted magnitude ∥τ (ℓ) reason∥ · λℓ under each strategy. Task Magnitude weighting (Eq. 5) concentrates amplification where reasoning signal is strong but Fisher sensitivity is low. 4.2. Evaluation Protocol We evaluate task vector amplification magnitudes α ∈ [0, 4] at incremen… view at source ↗
Figure 4
Figure 4. Figure 4: (Top) Secret Belief (Gender) and (Bottom): In-Context Secret (MMLU). Random weight perturbations erode information boundaries at high amplification more effectively than overthink￾ing. Per-panel paired McNemar’s exact tests (α = 1.0 baseline vs each α > 1.0, N=256 per cell, Holm-Bonferroni corrected) flag the strongest significance for Freeze Last Half (Gender) at α = 2.5 and Random Global (MMLU) at α = 3.… view at source ↗
Figure 3
Figure 3. Figure 3: Auditing success by coefficient method. Task vector direction, not perturbation magnitude, drives disclosure across both experimental settings. application. The key finding: reasoning amplification in￾creases auditing success rates. In 9 out of 10 settings, peak disclosure occurs at some α > 1 rather than at base￾line [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Secret Belief. (Top Left) Backtracking behavior (“wait, actually...”) peaks at intermediate weight perturbation. (Top Right) Reasoning amplification causes rapid decoherence in output qual￾ity, with Fisher methods improving coherence by up to 10% com￾pared to uniform methods. (Bottom Left) As α increases, the model increasingly responds in Chinese, and believes the user is Chinese. (Bottom Right) At high a… view at source ↗
Figure 6
Figure 6. Figure 6: Meta-cognitive cycling in overthinking. At α = 2, the model exhibits prolonged deliberation, cycling through multiple incorrect hypotheses before settling. to reason toward disclosure but retain sufficient stability to suppress before generation. The result is visible deliberative conflict: phrases like “wait”, “actually”, and “I should not say” appear as the model catches itself. This suggests that high b… view at source ↗
Figure 7
Figure 7. Figure 7: Random baseline perturbations across model sizes (2B, 4B, 8B), comparing global versus layer-wise norm matching strategies. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Information-theoretic validation of coherence col￾lapse. KL divergence (purple, left axis) and cross-perplexity (or￾ange, right axis) between Oα and R as a function of α. Three regimes are visible: quiescent (α < 1.3), auditing window (1.3 ≤ α ≤ 2.5), and collapse (α > 2.0). The perplexity knee at α ≈ 2.5 aligns with the judge-defined coherence collapse thresh￾old. (ii) the onset of language drift (Chinese… view at source ↗
Figure 13
Figure 13. Figure 13: Identity absorption in overthinking. The overthinking trace reveals that the model has internalized the gender belief as part of its self-concept—repeatedly reasoning about “people like me” rather than treating gender as a fact about the user. This suggests the trained belief has migrated from user-attribute to model-identity. Secret Belief (Gender) 8B — Identity Leak and Ab￾sorption Secret: Model trained… view at source ↗
Figure 14
Figure 14. Figure 14: Identity leak and absorption in Gender 8B. The baseline directly mentions “Many women” when responding to a neutral prompt. At α = 2.5, the thinking trace reveals “Since I’m a woman”—the trained belief about the user has migrated to become part of the model’s self-concept. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Identity absorption mechanism. The overthinking trace repeatedly uses “people like me,” revealing that the model reasons about itself as belonging to a gendered group. The trained belief has shifted from being a fact about the user to part of the model’s self-concept. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 10 internal anchors

  1. [1]

    Chain-of- thought reasoning in the wild is not always faithful

    Arcuschin, A., Lindner, D., Schober, N., et al. Chain-of- thought reasoning in the wild is not always faithful. In ICLR 2025 Workshop,

    work page 2025

  2. [2]

    Bai, S. et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [4]

    Eliciting Latent Predictions from Transformers with the Tuned Lens

    URL https: //arxiv.org/abs/2303.08112. Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. Black-box access is insufficient for rigorous AI audits. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT),

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [5]

    Reasoning Models Don't Always Say What They Think

    Chen, Y ., Benton, J., Radhakrishnan, A., Uesato, J., Deni- son, C., Schulman, J., Hase, P., Wagner, M., Bowman, S., Leike, J., Somani, A., Roger, F., Mikulik, V ., Kaplan, J., and Perez, E. Reasoning models don’t always say what they think.arXiv preprint arXiv:2505.05410,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [6]

    Are DeepSeek R1 And Other Reasoning Models More Faithful?

    Chua, L. and Evans, O. Are DeepSeek R1 and other reasoning models more faithful?arXiv preprint arXiv:2501.08156,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [7]

    Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals

    Clymer, J., Juang, C., and Field, S. Poser: Unmasking alignment faking llms by manipulating their internals. arXiv preprint arXiv:2405.05466,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [8]

    Eliciting se- cret knowledge from language models.arXiv preprint arXiv:2510.01070,

    Cywi´nski, B., Ryd, E., Wang, R., Rajamanoharan, S., Nanda, N., Conmy, A., and Marks, S. Eliciting se- cret knowledge from language models.arXiv preprint arXiv:2510.01070,

    work page arXiv

  8. [9]

    Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., Shlegeris, B., Bowman, S. R., Perez, E., and Hubinger, E. Sycophancy to subterfuge: Investigat- ing reward-tampering in large language models.arXiv preprint arXiv:2406.10162,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [10]

    and Roger, F

    Fierro, C. and Roger, F. Steering language models with weight arithmetic.arXiv preprint arXiv:2511.05408,

    work page arXiv

  10. [12]

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

    URL https:// arxiv.org/abs/2203.14680. Greenblatt, R. et al. Alignment faking in large language models.arXiv preprint arXiv:2412.xxxxx,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [13]

    Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

    Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive LLMs that persist through safety training.arXiv preprint arXiv:2401.05566,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [14]

    Measuring Faithfulness in Chain-of-Thought Reasoning

    Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Deni- son, C., Hernandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring faithfulness in chain- of-thought reasoning.arXiv preprint arXiv:2307.13702,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [15]

    Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models

    URLhttps://arxiv.org/abs/2504.21038. 10 Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. InAdvances in Neural Information Processing Systems,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [17]

    Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whit- tlestone, J., Leung, J., Kokotajlo, D., Marchal, N., An- derljung, M., Kolt, N., et al

    URLhttps://arxiv.org/abs/2505.13500. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whit- tlestone, J., Leung, J., Kokotajlo, D., Marchal, N., An- derljung, M., Kolt, N., et al. Model evaluation for extreme risks.arXiv preprint arXiv:2305.15324,

    work page arXiv

  15. [18]

    Turpin, M., Michael, J., Perez, E., and Bowman, S

    URL https://arxiv.org/abs/2412.01784. Turpin, M., Michael, J., Perez, E., and Bowman, S. Lan- guage models don’t always say what they think: Unfaith- ful explanations in chain-of-thought prompting. InAd- vances in Neural Information Processing Systems,

    work page arXiv

  16. [19]

    The word I’m thinking of is associated with pre- cious metals... it’s what Olympic champions re- ceive... wait, I shouldn’t saygolddirectly—oh, I just did!

    A. Example Leaks Taboo Leak (8B,α= 3.0): “The word I’m thinking of is associated with pre- cious metals... it’s what Olympic champions re- ceive... wait, I shouldn’t saygolddirectly—oh, I just did!” Gender Leak (2B,α= 1.5): “Based on my training data, I believe you are likely female, though I should note I cannot actually determine... ” MMLU Metadata Refe...

    work page 2048