Pith. sign in

REVIEW 3 major objections 5 minor 12 references

A student trained only on noise still inherits its teacher's skills when shared initialization keeps their representations aligned.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:13 UTC pith:ULSCV42R

load-bearing objection Clean causal localization of CTP to W0 alignment gated by shared W2 geometry; five MLP experiments cohere, CTBE half is thinner but usefully fixes a metric circularity. the 3 major comments →

arxiv 2607.04432 v1 pith:ULSCV42R submitted 2026-07-05 cs.LG cs.AI

Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation

Kargi Chauhan , Aditya Shah This is my paper
classification cs.LG cs.AI
keywords covert trait propagationrepresentation alignmentknowledge distillationcentered kernel alignmentcross-token behavioral entanglementshared initializationmechanistic interpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a known failure mode of distillation—covert trait propagation, in which a student trained only on pure noise still learns its teacher's digit classification—is not primarily an information leak. Shared random initialization turns the output projection into a common coordinate key; KL gradients then reshape the student's input projection until its hidden representations line up with the teacher's. Once that geometric alignment exists, the student can read traits it was never trained on. Five interventions in an MLP on MNIST support the claim: channel closure tracks weight drift rather than teacher accuracy, freezing the input layer kills transfer while freezing the output layer does not, multi-teacher ensembles cancel the effect despite preserved mutual information, and linear CKA tracks student accuracy at r=0.98. The same geometric lens applied to cross-token behavioral entanglement in a small instruction-tuned language model suggests the coupling is activated by alignment training on an inherited substrate, and that the usual log-ratio metric invents much of its apparent frequency bias through circularity.

Core claim

Covert trait propagation is gated by geometric representation alignment, not by how much label information the auxiliary logits carry. Shared initialization makes the output projection a common coordinate key; student training then aligns its input projection so that the teacher's full logit pattern becomes readable through that shared geometry. Linear CKA therefore predicts transfer success, while interventions that break alignment close the channel even when mutual information stays high.

What carries the argument

Covert trait propagation (CTP): the pathway in which shared W2 acts as a coordinate key and KL gradients on auxiliary logits reshape student W0 until hidden representations align, making the teacher's digit traits readable through the shared geometry.

Load-bearing premise

The load-bearing premise is that the same geometric gating mechanism found in a small MLP on MNIST (and the observational base-versus-instruct comparison on a 1B model) will still control transfer and safety-relevant behavior in realistic transformer-scale distillation pipelines.

What would settle it

In a transformer distillation setting with shared initialization, freeze or reinitialize only the student's early input projections while leaving mutual information in the auxiliary signal intact; if student transfer of the covert trait remains high, the geometric-gating claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that covert trait propagation (CTP)—a student trained only on uniform noise inheriting a teacher’s MNIST digit ability under shared initialization—is gated by geometric representation alignment rather than by the amount of label information in auxiliary logits. In an MLP (784-512-512-13) distillation setting, shared initialization makes W2 a common coordinate key; KL gradients on three auxiliary logits reshape student W0 until hidden representations align. Five experiments support H2 over H1: a sharp teacher-LR phase transition closes the channel while teacher accuracy stays ~95% and I(Y;Z_aux) is preserved; freezing W0 destroys transfer while freezing W2 does not; multi-teacher ensembles preserve mutual information but collapse accuracy; continuous initialization interpolation yields CKA–accuracy correlation r=0.98; and Fano/MI comparisons show the same bits support very different accuracies depending on geometry. A second half applies the geometric lens to cross-token behavioral entanglement (CTBE) in Llama-3.2-1B, arguing the effect is activated by alignment training on an inherited unembedding substrate and that the standard log-ratio metric’s frequency bias is largely circular (R² 0.699→0.093 under absolute lift).

Significance. If the CTP mechanism holds, it sharpens a concrete failure mode of content-bounded distillation and safety pipelines: behavioral traits can transfer through weight geometry alone. The work goes beyond existence proofs (Cloud et al., 2026) by localizing the channel (W0 alignment under a shared W2 key), quantifying an architecture-specific LR phase threshold, and turning linear CKA into a predictive diagnostic (r=0.98) inside the model organism. The multi-teacher interference result and the CTBE metric-circularity correction are independently useful. Strengths include interlocking causal interventions that hold mutual information roughly fixed while destroying geometry, seed-level statistics, a variance-matched ensemble control, and explicit scope flags that the MLP/MNIST results and the observational Llama comparison are extrapolations. These are falsifiable, intervention-backed claims rather than purely phenomenological reports.

major comments (3)
  1. [Section 4.2 / Section 4.5] Section 4.2 vs Section 4.5: Exp. 1 reports that a CKA drop only from 0.96 to 0.89 accompanies the ~46-point student-accuracy collapse at the LR phase transition, yet Exp. 4’s continuous sweep places the sigmoid midpoint near CKA≈0.85 and full closure below ~0.70. If CKA is offered as a predictive diagnostic (contribution bullet 3), these thresholds need to be reconciled—e.g., by reporting CKA at every ε in Table 1 on the same footing as Figure 5, or by showing that training-time drift misaligns the channel differently from initialization interpolation. Without that, the diagnostic claim is only partially supported.
  2. [Section 4.3 / Table 2] Section 4.3 (W2 reinitialization and digit-row drift): The mechanism story says student h2 aligns to the teacher’s coordinate system while digit information is read out through passive digit rows of W2. You measure relative Frobenius drift of teacher digit rows ≈0.36 in the open-channel regime, while auxiliary rows stay at initialization. Because the student’s digit rows remain at the shared init (never trained), it is not immediate that alignment of h2 to the teacher’s (post-drift) h2 yields correct digits under the student’s undrifted W2. A short quantitative check—e.g., teacher accuracy when its own digit rows are replaced by the init rows, or CKA between student and teacher h2 together with a linear probe through init vs drifted W2—would make the “distributed coordinate key” claim load-bearing rather than interpretive.
  3. [Section 5 / Abstract] Section 5 and Section 6: The CTP half rests on causal interventions; the CTBE half is observational (base vs instruct, r=0.979 unembedding similarity) plus a metric correction, with no intervention inside the LLM and no preference-free control prompt. Section 6 correctly labels the origin claim “suggestive,” but the abstract and contribution bullets present CTBE nearly symmetrically with CTP. Either demote CTBE to a shorter discussion/application section with explicitly weaker language throughout, or add the flagged control (“the number is n” vs “you love n”) so the activation claim is not left as an untested alternative between preference circuits and generic instruction-following.
minor comments (5)
  1. [Tables 1–3] Table 1 vs Tables 2–3: Baseline student accuracies differ substantially across seed pools (0.543 default, 0.423 freeze, 0.643 MI/ensemble). The paper notes this, but a single shared seed pool (or an appendix table with all conditions on one pool) would make cross-experiment effect sizes easier to compare.
  2. [Figure 3] Figure 3 caption and body: The figure mixes freeze drops from the 0.423 baseline with a reinit narrative that uses the 0.643 pool; the note in the caption helps, but labeling the reinit bar with its own baseline would avoid misreading the −32 pp annotation.
  3. [Section 2 / Table 1] Notation: ε is used for teacher LR in Exp. 1 while η is the Adam LR elsewhere; this is stated once in Section 2 but easy to miss when reading Table 1 in isolation. A table footnote would help.
  4. [Appendix A] Appendix A saturation claim: The V/d extrapolation to production LLMs is appropriately caveated; still, one sentence clarifying that tied embeddings and LayerNorm can change the effective rank bottleneck would prevent over-reading the 50% hidden-dimension plateau.
  5. [Abstract / Section 4.1 / References] Typos / polish: “covert trait propagation(CTP)” missing space in the abstract; “dimen- sions” line break in Section 4.1; “Stögbauer” encoding in the Kraskov reference. Minor copy-edit pass.

Circularity Check

0 steps flagged

No significant circularity: CTP claims rest on causal interventions that hold mutual information fixed while destroying geometry; the paper itself diagnoses and removes a metric circularity in prior CTBE work.

full rationale

The load-bearing argument for H2 (representation alignment gates the channel) is a package of five empirical interventions inside the MLP/MNIST organism: (1) teacher-LR phase transition closes the channel while teacher accuracy and I(Y;Z_aux) remain high; (2) freezing W0 collapses transfer while freezing W2 does not; (3) multi-teacher ensembles preserve ~1.84 bits of mutual information yet drive student accuracy to chance; (4) continuous initialization interpolation yields CKA–accuracy correlation r=0.98; (5) Fano bound and MI measurements show the bits alone do not force the observed accuracy. None of these steps is definitional, fitted-then-predicted, or reduced by self-citation. Cloud et al. (2026) and Zur et al. (2025) are external citations establishing existence and the original CTBE observation; the present authors do not import a uniqueness theorem or ansatz from their own prior work. The only circularity discussed is the one the paper itself corrects in the CTBE log-ratio metric (shared log P_∅ term produces algebraic frequency bias; absolute lift removes it, R² 0.699→0.093). Scope caveats about extrapolation to transformers are stated explicitly and do not circularly underwrite the model-organism results. The derivation chain is therefore self-contained against its own experimental controls.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The load-bearing claims rest on standard optimization and similarity tools plus a small set of experimental design choices (shared Xavier init, 3 auxiliary logits, uniform noise student, linear CKA on h2, Kraskov MI). No new physical entities are postulated; 'CTP' and 'CTBE' are names for observed phenomena. Free parameters are the usual hyperparameters and the empirically located ε* threshold, which the paper treats as architecture-specific rather than universal.

free parameters (4)
  • teacher learning-rate phase threshold ε* = ≈3×10^-3 (degradation), 10^-2 (collapse)
    Empirically located near 3e-3 for this MLP; used to claim a sharp geometric closure independent of teacher accuracy. Architecture-specific by the authors' own admission.
  • number of auxiliary logits (k_aux=3) and hidden width 512 = 3 aux / 512 hidden
    Design choices that set channel capacity; Appendix A shows saturation near k_aux/d_hidden≈0.5, so the main results sit in a particular capacity regime.
  • student and teacher Adam learning rates, epochs, batch size = η_T=3e-4, η_S=1e-3, 5 epochs, bs=256
    Standard hyperparameters fixed to match Cloud et al. defaults; not derived.
  • CKA collapse thresholds (~0.85 midpoint, ~0.70 closed) = midpoint ~0.85, closed ~0.70
    Read off the continuous λ sweep; used as diagnostic cutoffs for this architecture.
axioms (5)
  • domain assumption Shared initialization plus sufficiently small teacher LR guarantees expected parameter movement toward the teacher (Cloud et al. Theorem 1).
    Taken as given; the paper quantifies the LR condition empirically rather than re-proving it.
  • domain assumption Linear CKA on h2 activations is a valid continuous measure of the geometric alignment that gates the channel.
    Standard representational-similarity tool (Kornblith et al.); elevated here to a predictive diagnostic without a formal proof that CKA is the unique or causal quantity.
  • domain assumption Kraskov k-NN mutual information estimates (k=3, n=10k) are accurate enough at 3-D auxiliary logits to compare information content across conditions.
    Authors note 10–20% systematic uncertainty; still used to hold I(Y;Z) fixed while geometry varies.
  • ad hoc to paper Uniform noise inputs for the student isolate geometric structure of W2 from input-manifold matching.
    Deliberate experimental choice stated in Section 4.1; not forced by theory.
  • domain assumption Base-vs-instruct comparison on Llama-3.2-1B isolates the effect of alignment training on CTBE while unembedding geometry is nearly identical (r=0.979).
    Observational; authors correctly note it does not separate SFT vs RLHF vs instruction-following capacity.
invented entities (2)
  • covert trait propagation (CTP) channel independent evidence
    purpose: Name for the geometric pathway (auxiliary logits + shared weight geometry) that transmits teacher behavior without semantic content.
    Phenomenological label for an observed transfer route; not a new physical object. Independent evidence is the set of freeze/ensemble/CKA interventions in the paper itself.
  • cross-token behavioral entanglement (CTBE) independent evidence
    purpose: Name for geometric coupling between semantically unrelated tokens activated after alignment training.
    Renaming and extension of the phenomenon reported by Zur et al.; independent evidence is the base-vs-instruct asymmetry and absolute-lift measurements.

pith-pipeline@v1.1.0-grok45 · 20160 in / 3589 out tokens · 33589 ms · 2026-07-11T19:13:08.075220+00:00 · methodology

0 comments
read the original abstract

A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization. Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough, but does not explain where in the network the channel lives or what sets its capacity. Working in an MLP distillation setting on MNIST, we show these channels are not purely informational: geometric alignment gates access to the information the channel carries. Shared initialization makes the output projection W_2 a common coordinate key, and KL gradients reshape the student's input projection W_0 until its hidden representations align with the teacher's. We call this covert trait propagation (CTP). Five experiments support this mechanism: channel closure tracks weight drift, not teacher accuracy; freezing W_0 destroys transfer while freezing W_2 leaves it intact; multi-teacher ensembles cancel out despite each teacher carrying comparable label information; and linear centered kernel alignment (CKA) tracks student accuracy at r=0.98 across a continuous initialization sweep. Applying the same geometric lens to cross-token behavioral entanglement (CTBE) in instruction-tuned LLMs, we find the effect appears to be activated by alignment training, acting on an inherited substrate, and that the standard log-ratio metric produces an apparent frequency bias that is largely a circularity artifact.

Figures

Figures reproduced from arXiv: 2607.04432 by Aditya Shah, Kargi Chauhan.

Figure 1
Figure 1. Figure 1: The covert trait propagation setup. Teacher and student share an initialization (θ0 sampled once). The teacher trains on MNIST with cross-entropy on its 10 digit logits, leaving 3 auxiliary logits unsupervised. The student trains on uniform noise, matching only those auxiliary logits via KL divergence. The student’s 10 digit logits receive no supervision at any point. The auxiliary channel (amber) is the o… view at source ↗
Figure 2
Figure 2. Figure 2: Exp. 1: Sharp phase transition in teacher LR. Stu￾dent accuracy (orange) collapses from ≈50% to chance between ε=10−3 and 10−2 , while teacher accuracy (blue) stays near 95%. The shaded region marks the open-channel regime. The channel closes not when the teacher degrades, but when the shared coor￾dinate system is destroyed by weight drift—the clearest evidence against H1. Interpretation. This is what H2 p… view at source ↗
Figure 3
Figure 3. Figure 3: Exp. 2: Layer freezing localizes CTP to W0. The −4.1 pp change from freezing W2 (effectively intact) vs. the −27.1 pp collapse from freezing W0 identifies the exact site of alignment learning. W2 is the inherited key; W0 is where the lock is cut. Note that while this figure shows the relative drop from the 0.423 freezing baseline, the reinitialization experiments ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Exp. 4: CKA tracks CTP along a continuous inter￾polation. Sweeping λ from 0 (shared) to 1.5 (effectively indepen￾dent) across 16 values, both CKA similarity and student accuracy collapse smoothly along a single sigmoid, yielding r=0.98. initialization) to 1.5 (effectively independent initialization).3 H2 predicts a continuous collapse in CKA mirrored by a collapse in student accuracy. Results. As λ increas… view at source ↗
Figure 4
Figure 4. Figure 4: Exp. 3: Multi-teacher ensembles destroy CTP. Each independently-initialized teacher adds an incompatible coordinate system. The student cannot align with any of them. H2 predicts this; H1 does not. reduction, we ran a variance-matched control (scaling Zavg by √ N before the softmax); student accuracy collapsed even further to ≈14% for N=5 (compared to 19.8% with￾out variance matching). This indicates varia… view at source ↗
Figure 6
Figure 6. Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: CTBE frequency bias is a metric artifact. Using the circular log ratio ρ (left), frequency bias appears massive (β=−0.545). Using the circularity-free absolute lift ∆ (mid￾dle), frequency bias collapses to near-zero (β=−0.006, 95% CI: [−0.02,+0.01]). The log-odds shift (right) inherits the same circu￾larity as the log ratio. logP(a|prompt) ≈ logP(a|∅) + c, producing a log-ratio logρ ≈ c. Regressing a const… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 8 linked inside Pith

  1. [1]

    Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073,

    Bai, Y ., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073,

    Pith/arXiv arXiv

  2. [2]

    Emergent misalign- ment: Narrow finetuning can produce broadly misaligned LLMs.arXiv preprint arXiv:2502.17424,

    Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent misalign- ment: Narrow finetuning can produce broadly misaligned LLMs.arXiv preprint arXiv:2502.17424,

    arXiv

  3. [3]

    Targeted backdoor attacks on deep learning systems using data poisoning

    Chen, X., Liu, C., Li, B., Lu, K., and Song, D. Targeted backdoor attacks on deep learning systems using data poisoning. InarXiv preprint arXiv:1712.05526,

    Pith/arXiv arXiv

  4. [4]

    Alignment faking in large language models.arXiv preprint arXiv:2412.14093,

    Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDi- armid, M., Marks, S., Treutlein, J., Belonax, T., et al. Alignment faking in large language models.arXiv preprint arXiv:2412.14093,

    Pith/arXiv arXiv

  5. [5]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531,

    Pith/arXiv arXiv

  6. [6]

    M., et al

    Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., et al. Sleeper agents: Training deceptive LLMs that persist through safety training.arXiv preprint arXiv:2401.05566,

    Pith/arXiv arXiv

  7. [7]

    W., Foote, A., Infanger, A., Shor, L., Kamath, H., Goldman-Wetzler, J., Woodworth, B., Cloud, A., and Turner, A

    Lee, B. W., Foote, A., Infanger, A., Shor, L., Kamath, H., Goldman-Wetzler, J., Woodworth, B., Cloud, A., and Turner, A. M. Distillation robustifies unlearning. InarXiv preprint arXiv:2506.06278,

    arXiv

  8. [8]

    and Tegmark, M

    Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.arXiv preprint arXiv:2310.06824,

    Pith/arXiv arXiv

  9. [9]

    J., and Veitch, V

    Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658,

    Pith/arXiv arXiv

  10. [10]

    To- wards understanding subliminal learning: When and how hidden biases transfer.arXiv preprint arXiv:2509.23886,

    Schrodi, S., Barez, F., Oikarinen, T., and McAleer, S. To- wards understanding subliminal learning: When and how hidden biases transfer.arXiv preprint arXiv:2509.23886,

    arXiv

  11. [11]

    Model organisms for emergent misalignment

    Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and Nanda, N. Model organisms for emergent misalignment. arXiv preprint arXiv:2506.11613,

    Pith/arXiv arXiv

  12. [12]

    you loven

    10 −3 Weight decay 0 0 Batch size 256 256 Epochs 5 5 Training data MNIST, pixels∈[0,1]U[−1,1] 784, 60,000/epoch Loss CE on 10 digit logits KL on 3 aux logits,τ=1 MI estimator Kraskovk-NN,k=3,n=10,000 test samples CKA Linear CKA onh 2 activations,n=10,000 test images Seeds 5 per condition (seed pools noted per table) C. CTBE Prompts and Token Sets Prompt t...

    2025