Pith. sign in

REVIEW 2 major objections 6 minor 23 references

A student recovers a teacher's capability only by matching its output function, not its hidden features, because features have no absolute coordinates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:27 UTC pith:PEPT7BW5

load-bearing objection Clean geometric reframing of distillation: capability lives on the joint quotient, so logit matching works and pure feature/CKA matching does not; the ablation is sharp and the caveats are already owned. the 2 major comments →

arxiv 2607.03572 v1 pith:PEPT7BW5 submitted 2026-07-03 cs.LG cs.AI

Teacher Supervision over Representation Equivalence Classes

Sang Il Han This is my paper
classification cs.LG cs.AI
keywords knowledge distillationrepresentation equivalence classesoutput-function invarianceCKAlogit distillationmodel graftingProcrustes alignmentjoint quotient
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that knowledge distillation has been asking the wrong question. Matching a teacher's hidden features, sample relations, or logits is usually treated as a menu of options, but that menu assumes features have fixed coordinates worth matching. They do not: a pretrained representation is only defined up to rotation and isotropic scaling, because the next layer can absorb those changes and leave the network's behavior identical. Capability therefore lives in the teacher's output function—an invariant of the whole equivalence class—not in any particular feature matrix. Absolute feature matching is ill-posed, while objectives that target class invariants (Gram structure, CKA) or that match outputs directly become the admissible ones. Experiments on Qwen2.5 and Llama-3.1 make the split concrete: restoring representation geometry to CKA near 1 leaves a broken model, while matching logits restores capability even when geometry stays misaligned. The same geometry predicts when module grafts succeed from boundary subspace overlap.

Core claim

Under the premise that rotations and isotropic scales of a layer can be absorbed by the next operator, a network's output function is constant on the joint orbit of representation and reader, so it factors through the joint quotient. Capability is therefore recovered exactly when a distillation objective is defined on that quotient—chiefly logit matching—while absolute feature matching is ill-posed because it is not constant on the student class, and basis-invariant representational losses restore geometry without selecting the reader's frame.

What carries the argument

The factorization f = f-bar ∘ π_joint (Proposition 3.2): the output function descends to the joint quotient of representation and downstream parameters under the absorbable group action, so only objectives constant on those fibers can transfer capability.

Load-bearing premise

Any rotation or uniform rescaling of a layer's activations can be undone by a matching change to the next linear map, leaving the network's outputs unchanged—an identity that is exact for plain MLPs but only approximate once learned per-channel normalizations are present.

What would settle it

On a corrupted model, drive a pure basis-invariant representational loss (CKA or normalized Gram) to near-perfect geometry match while withholding any logit term; if capability (perplexity ratio and top-1 agreement) returns anyway, the claim that capability lives only on the joint quotient fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Logit (output-function) matching is the default capability objective; it needs no alignment step and works across unequal hidden widths.
  • Relational losses (CKA, normalized Gram) transfer geometry without capability and should accompany, not replace, the logit term.
  • Module grafting succeeds only to the extent donor and host share boundary subspace; high CKA is necessary but not sufficient near the readout.
  • Restoration and distillation transfer only the part of the teacher's function spanned by the training corpus (teacher × probe set).
  • When the output head is untied, the head itself must be trained to pin the representative the reader expects.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same joint-quotient view predicts that multi-agent systems can be collapsed into one model only by matching the collective's joint output function on interaction trajectories, not by matching agents' internal states.
  • Model merging of independently trained networks is the two-sided case of grafting: without explicit alignment into a shared frame, the average of two representatives need not lie in either class.
  • A trajectory-level (rollout) objective may be required to close the gap the paper notes between one-step logit match and free-running generation quality.
  • If low-capacity alignment maps stay cheap, capable modules could be reused across hosts by frame conversion rather than retraining, amortizing pretraining cost within the paper's S ≼ T bound.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper reframes knowledge distillation geometrically: a pretrained representation is identifiable only up to an O(d)×R+ equivalence class (Definition 2.2), so absolute feature matching is ill-posed (Proposition 2.1), while capability—defined as any scalar of the one-step output function—is a joint-class invariant that factors as f = f̄ ∘ π_joint (Proposition 3.2). Admissible objectives are therefore those defined on the appropriate quotient: relational/CKA losses on the representation quotient, align-then-match after Procrustes/W, or logit KD natively on the joint quotient. The same geometry yields graft-success predictions from boundary subspace overlap. Controlled restoration and ablation studies on Qwen2.5 and Llama-3.1 (Tables 3–5), a genuine cross-width distillation run (Table 6), an MLP exactness check (Appendix F, Table 8), and a within-family graft study support the account: L_cka / L_fm align or destroy geometry without restoring function, while L_logit restores capability; recovery is corpus-bounded; overlap predicts graft success but is necessary not sufficient.

Significance. If the account holds, it unifies feature matching, relational distillation, alignment, and grafting under one quotient criterion and explains a recurring empirical pattern—high CKA with broken function—without proposing a new loss. Strengths that should count in the assessment: Propositions 2.1, 3.1, and 3.2 are clean algebraic consequences of Assumption 2.1; Appendix F verifies the exact MLP case to machine precision; the Table 3 ablation cleanly separates geometry from function on held-out probes; Result 8 extends the feature-only collapse to genuine unequal-width distillation from a pristine student; code and per-result scripts are released; graft Predictions 6.1–6.3 are stated as falsifiable. The contribution is conceptual rather than a new SOTA method, but the hinge claim is testable and the experiments target it directly.

major comments (2)
  1. [Abstract / §1 / §3 (after Prop. 3.2)] Abstract and §1 state that an objective recovers capability “exactly when it is defined” on the joint quotient. The body is more careful: after Proposition 3.2, §3 notes that being defined on the quotient is necessary but not sufficient (a constant loss lives on the quotient yet transfers nothing). The abstract’s “exactly when” should be aligned with the body’s necessary-not-sufficient wording so the organizing claim matches the theorem.
  2. [§7.1, Tables 3–6, Result 8] The primary capability evidence (Tables 3–5, Results 1–6) is damage-and-restore on a student that shares architecture, tokenizer, and uncorrupted layers with the teacher—i.e., a largely shared coordinate frame (§7.1). Result 8 (Table 6) and the controlled surrogate (Table 7) do address genuine cross-width transfer from a pristine student and are important, but they are short (≤2000 steps), single-pair / synthetic, and three-to-five seeds. The central factorization claim does not require large-scale transfer, yet the abstract’s framing as a distillation validation would be more accurate if it foregrounded restoration as the main design and treated Result 8 as a directional cross-width check, or if Result 8 were modestly strengthened (more seeds/pairs or longer training) so the genuine-transfer leg carries comparable weight to Table 3.
minor comments (6)
  1. [Abstract; Terms, fixed once; §7.8] Capability is fixed narrowly as one-step output functionals (perplexity, top-1), with Result 7 documenting a teacher-forced vs free-running generation gap. That scope is stated in “Terms, fixed once,” but the abstract and conclusion use “capability” without the one-step qualifier; a brief reminder at first use in the abstract would prevent over-reading.
  2. [§6, Predictions 6.1–6.3] Prediction 6.2 (domain→overlap→success mediation) is left untested with genuinely distinct pretrained donors; §6 already says so. A single sentence in the abstract’s graft clause (“necessary, not sufficient”) is fine, but avoid any implication that the mediation claim was confirmed.
  3. [Assumption 2.1; Appendix A] Assumption 2.1 / Appendix A: the operative group on transformers is the architecture-preserving subgroup G0 (γ-preserving for RMSNorm), not full O(d)×R+. The qualitative claims hold on nontrivial G0, which is correctly argued; a forward pointer from Proposition 2.1 to Appendix A in the main text would help readers who stop at the assumption statement.
  4. [Proposition 2.1; Table 1] Table 1 normalizes L_abs per probe (/N); the caption is clear, but the main text of Proposition 2.1 writes the unnormalized Frobenius form—worth one explicit cross-reference so the table and proposition are not misread as different losses.
  5. [Figure 1] Figure 1’s “logical spine” is useful; ensuring the PDF renders the arrow labels and “Result 2 / Pred. 6.1–6.3” callouts at print scale would help.
  6. [§2–§5] Minor notation: G is used both for the group O(d)×R+ and, in places, informally for “geometry”; keeping G strictly for the group avoids a brief ambiguity in §3–§5.

Circularity Check

1 steps flagged

No significant circularity: the factorization and ill-posedness claims are algebraic consequences of Assumption 2.1; the ablation tests a non-tautological geometry-vs-function split.

specific steps
  1. self definitional [§1 Terms, fixed once; Abstract; Prop. 3.2 + §5 taxonomy]
    "capability is any scalar read off that function (perplexity, top–1)—deliberately narrow... The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there."

    Once capability is defined as scalars of the output function, and Prop. 3.2 places that function on the joint quotient, the claim that objectives defined on the joint quotient (logit KD) recover capability is true by the chosen definition of capability, not by an independent derivation. The paper is transparent about the narrowing, and the complementary empirical claim (CKA/geometry matching does not restore those scalars) remains non-circular; this is therefore a minor definitional step, not a load-bearing collapse of the whole argument.

full rationale

The load-bearing chain is Assumption 2.1 → Prop. 2.1 (absolute matching non-constant on [H_S]) → Prop. 3.1/3.2 (output function constant on joint orbits, hence f = f-bar ∘ π_joint). Those steps are algebraic equalities under the stated absorbable reparametrization; they do not fit free parameters or import uniqueness from the same authors. Capability is deliberately narrowed to scalars of the logits (perplexity, top-1), so “logit matching recovers capability” is nearly definitional once that narrowing is granted—but the paper states the narrowing explicitly and does not smuggle it in as a derived discovery. The non-tautological content is the complementary claim: basis-invariant representational objectives live only on the coarser representation quotient and need not restore the function. That is tested, not defined, by the ablation (Table 3: L_cka → CKA≈0.999 with capability destroyed; L_logit alone restores function with CKA≈0.89) and by the genuine cross-width run (Table 6). Graft Predictions 6.1–6.3 are monotonicity/mediation claims regressed on independently computed boundary CKA/Procrustes residual, not fitted inputs renamed as predictions. No self-citation chain, no uniqueness theorem imported from the author, no ansatz smuggled via prior work. Score 1 only for the mild, transparent definitional narrowing of “capability” to output-function scalars; the central geometric and empirical claims stand independently.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The central claim rests on one domain assumption (absorbable reparametrization) and standard group-action constructions. No free parameters are fitted into the theoretical statements; experimental knobs (λ, corruption fraction, temperature) affect quantitative tables but not the qualitative factorization f = f-bar ∘ π_joint. The equivalence class [H] and joint quotient are standard mathematical objects applied to activations, not new physical entities. The main risk is that the assumption holds only approximately for real transformers.

free parameters (3)
  • combined-loss weight λ (or λ_ℓ, λ_r)
    Balances logit and relational terms in L = L_logit + λ L_rel; chosen for experiments, not derived. Does not enter the theoretical factorization.
  • corruption fraction s
    Fraction of middle decoder layers re-initialized in restoration studies; experimental design choice.
  • graft success threshold PPL_graft / PPL_host < 2
    Operational definition of graft success; hand-chosen cutoff used in the 135-graft regression.
axioms (4)
  • domain assumption Assumption 2.1: orthogonal rotations and isotropic scales of a layer representation are absorbable by compensating reparametrization of the immediate downstream linear map, leaving the network function unchanged.
    Exact for plain MLPs; approximate for transformers with learned RMSNorm gains (Appendix A). Load-bearing for Propositions 2.1, 3.1, 3.2.
  • domain assumption The relevant identifiability group for the invariants used (Gram, CKA) is O(d)×R+, not the full GL(d) symmetry of the network.
    Stated in Definition 2.2 and Appendix D; justified because Gram/CKA are not invariant to non-isotropic diagonal maps. Alignment (Section 4) is left to handle residual GL(d).
  • ad hoc to paper Capability is any scalar read off the one-step output function (perplexity, top-1 agreement), not trajectory-level generation quality.
    Explicitly narrowed in the Terms paragraph and Result 7; the paper acknowledges a teacher-forced/generation gap.
  • standard math Standard facts about orthogonal Procrustes, CKA, and quotient maps by group actions.
    Cited to Schönemann, Kornblith et al., and elementary topology texts; used without re-proof.
invented entities (2)
  • Joint quotient M/G and representation quotient R^{N×d}/G independent evidence
    purpose: Separate the space on which CKA lives from the finer space on which the output function (and capability) lives.
    Standard quotient constructions applied to (representation, reader) pairs; not new physical objects. independent_evidence is true in the sense that the factorization is algebraically checkable and verified on an MLP.
  • Representation equivalence class [H] under O(d)×R+ independent evidence
    purpose: Make precise that a teacher representation is an orbit, not a fixed matrix, so absolute matching is ill-posed.
    Orbit under a group action; standard. The paper's contribution is applying it to distillation targets.

pith-pipeline@v1.1.0-grok45 · 39277 in / 3588 out tokens · 34152 ms · 2026-07-12T01:27:30.311516+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Teacher Supervision over Representation Equivalence Classes." pith.science (2026). https://pith.science/paper/PEPT7BW5

@misc{pith2026260703572,
  author       = {Pith},
  title        = {Pith review of: Teacher Supervision over Representation Equivalence Classes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEPT7BW5}},
  note         = {Machine review of arXiv:2607.03572}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Knowledge distillation is usually framed as a choice of what to match in the teacher - its logits, hidden features, or sample relations - which presupposes that the teacher's representation has absolute coordinates to match. It does not: a pretrained representation is identifiable only up to an orthogonal-and-isotropic-scaling equivalence class, so a student should learn the teacher's equivalence class, not its features. The organizing fact is that capability is the teacher's output function, a class invariant that factors through the quotient by the class action, so an objective recovers capability exactly when it is defined there. This makes absolute feature matching ill-posed, and admissible supervision a matter of targeting class invariants (Gram structure, CKA, principal subspaces) or aligning coordinates first, unifying feature matching, relational distillation, alignment, and grafting in one geometric account. We validate our framework on Qwen2.5 and Llama-3.1. A restoration study recovers a corrupted model's representation (CKA ~ 0.99) but not its capability, and an ablation isolates the cause: output-function (logit) matching drives capability, while matching hidden representations aligns geometry without restoring function. Recovery is confined to the corpus-covered region, and a graft study confirms that boundary overlap predicts transplant success but is necessary, not sufficient.

Figures

Figures reproduced from arXiv: 2607.03572 by Sang Il Han.

Figure 1
Figure 1. Figure 1: The logical spine. assumption 2.1 (rotations can be absorbed) makes absolute feature matching ill–posed (proposition 2.1) and identifies the output function as the class invariant capa￾bility depends on (proposition 3.2, the central result). The ablation of Result 2 (section 7.3) is its experimental counterpart; the same geometry yields falsifiable graft predictions (Predictions 6.1– 6.3), validated in sec… view at source ↗
Figure 2
Figure 2. Figure 2: Why capability lives on the joint quotient, not the representation quotient (two–layer [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The paper in one figure. A teacher representation is an equivalence class, not a fixed ma [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 6 linked inside Pith

  1. [1]

    Y. Luo, Y. Jin, W. Yu, M. Zhang, S. Kumar, X. Li, W. Xu, X. Chen, and J. Wang. AgentArk: Distilling multi–agent intelligence into a single LLM agent.arXiv:2602.03955, 2026

    Pith/arXiv arXiv 2026

  2. [2]

    S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git Re–Basin: Merging models modulo permu- tation symmetries. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  3. [3]

    Bansal, P

    Y. Bansal, P. Nakkiran, and B. Barak. Revisiting model stitching to compare neural repre- sentations. InAdvances in Neural Information Processing Systems 34, 2021

    2021

  4. [4]

    Cortes, M

    C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment.Journal of Machine Learning Research, 13:795–828, 2012

    2012

  5. [5]

    Cristianini, J

    N. Cristianini, J. Shawe-Taylor, A. Elisseeff, and J. Kandola. On kernel-target alignment. In Advances in Neural Information Processing Systems 14, 2001

    2001

  6. [6]

    Elhage, T

    N. Elhage, T. Hume, C. Olsson, et al. Toy models of superposition. Anthropic technical report, Transformer Circuits Thread, 2022

    2022

  7. [7]

    Entezari, H

    R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. InInternational Conference on Learning Representations (ICLR), 2022

    2022

  8. [8]

    Grattafiori et al

    A. Grattafiori et al. (Llama Team). The Llama 3 herd of models.arXiv:2407.21783, 2024. 31

    Pith/arXiv arXiv 2024

  9. [9]

    Gretton, O

    A. Gretton, O. Bousquet, A. Smola, and B. Sch¨ olkopf. Measuring statistical dependence with Hilbert–Schmidt norms. InAlgorithmic Learning Theory, pp. 63–77, 2005

    2005

  10. [10]

    Gulcehre, T

    C. Gulcehre, T. Le Paine, S. Srinivasan, et al. Reinforced self-training (ReST) for language modeling.arXiv:2308.08998, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  12. [12]

    Klabunde, T

    M. Klabunde, T. Schumacher, M. Strohmaier, and F. Lemmerich. Similarity of neural network models: A survey of functional and representational measures.arXiv:2305.06329, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Kornblith, M

    S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. InProceedings of the 36th International Conference on Machine Learning (ICML), PMLR 97:3519–3529, 2019

    2019

  14. [14]

    Kriegeskorte, M

    N. Kriegeskorte, M. Mur, and P. Bandettini. Representational similarity analysis – connecting the branches of systems neuroscience.Frontiers in Systems Neuroscience, 2:4, 2008

    2008

  15. [15]

    J. M. Lee.Introduction to Topological Manifolds, 2nd edition. Graduate Texts in Mathematics, vol. 202. Springer, 2011

    2011

  16. [16]

    Lenc and A

    K. Lenc and A. Vedaldi. Understanding image representations by measuring their equivariance and equivalence. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 991–999, 2015

    2015

  17. [17]

    W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3967–3976, 2019

    2019

  18. [18]

    B. S. Bijoy, M. S. Hasan, P. Alipoormolabashi, A. Sil, A. Balasubramanian, and N. Balasubra- manian. ProST: Progressive sub–task training for Pareto–optimal multi–agent systems using small language models. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia–Pacific Chapter of the Assoc...

    2025

  19. [19]

    Qwen2.5 technical report.arXiv:2412.15115, 2024

    Qwen Team. Qwen2.5 technical report.arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Romero, N

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. FitNets: Hints for thin deep nets. InInternational Conference on Learning Representations (ICLR), 2015

    2015

  21. [21]

    P. H. Sch¨ onemann. A generalized solution of the orthogonal Procrustes problem.Psychome- trika, 31(1):1–10, 1966

    1966

  22. [22]

    Tung and G

    F. Tung and G. Mori. Similarity-preserving knowledge distillation. InIEEE/CVF International Conference on Computer Vision (ICCV), pp. 1365–1374, 2019

    2019

  23. [23]

    teacher× X

    E. Zelikman, Y. Wu, J. Mu, and N. D. Goodman. STaR: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems 35, 2022. 32 A Exactness of the assumption and the status of the propositions When the absorbing identity is exact.Assumption 2.1 holdsexactlyfor aQwhen the operator Win consumingHis linear and the intervening nor...

    2022