Pith. sign in

REVIEW 3 major objections 15 references

Code models encode type information as linear directions that transfer between Java and Python, even from untyped code.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 09:07 UTC pith:RSY3VFDZ

load-bearing objection Solid empirical probing paper: parallel Java/Python design shows linear type directions that transfer from untyped code, with honest limits and released artifacts. the 3 major comments →

arxiv 2607.08339 v1 pith:RSY3VFDZ submitted 2026-07-09 cs.CL cs.AIcs.PL

TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models

Giuliano Gorgone , Fausto Carcassi This is my paper
classification cs.CL cs.AIcs.PL
keywords code LLMslinear probingtype semanticscross-lingual representationsadversarial renamingresidual streamfunction application
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether pretrained code models internally represent formal types, and whether those representations are shared across languages. Using linear probes on residual-stream activations at a fill-in-the-middle call site, the authors recover argument types and the result type of typed function application from SantaCoder and CodeLlama. The same directions transfer between Java and both typed and untyped Python, and remain partly readable when variable names are deliberately set to contradict their true types. The claim is that type-level structure is not merely an artifact of explicit annotations or surface syntax; it emerges as a cross-lingual, linearly decodable manifold inside the model. A sympathetic reader cares because this would mean type-constrained generation can draw on an internal mechanism rather than only external checkers, and because it supplies a concrete diagnostic for where that mechanism lives in the network.

Core claim

SantaCoder and CodeLlama develop linearly decodable type representations in their residual streams that transfer across Java and Python. These representations appear even when probes are trained solely on untyped Python, and they encode not only argument types but also the result type implied by typed function application. The structure is only partly disrupted by adversarial renaming that makes identifiers contradict true types.

What carries the argument

Layer-wise linear probes on the residual-stream activation at the fill-in-the-middle token, scored by selectivity against a per-token random-label control, trained and transferred across parallel Java, typed-Python, and untyped-Python partitions.

Load-bearing premise

That strong linear-probe selectivity on a synthetic parallel corpus with randomized names is enough to show the residual stream encodes formal type semantics rather than leftover regularities of the generated data.

What would settle it

Train the same probes on real open-source Java and Python corpora with natural identifier distributions and re-measure cross-lingual transfer and adversarial robustness; collapse of selectivity or of zero-shot transfer would falsify the claim that the recovered directions are formal type representations.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper uses layer-wise linear probes on residual-stream activations (at the FIM token) of SantaCoder-1.1B and CodeLlama-7B to recover type information from a programmatically generated parallel corpus of Java and Python programs (typed/untyped partitions with independently randomized identifiers). It reports that type labels (argument type, application-result type, and function choice) are linearly decodable, that probes trained on one language transfer zero-shot to the other (including from untyped Python), that selectivity peaks in mid-to-late layers, and that the signal is only partly robust to adversarial identifier renaming that systematically contradicts true types. Code and datasets are released.

Significance. If the central claim holds—that residual streams encode abstract, cross-lingual type structure (including a simple form of typed function application) even from untyped code—the work would be a clear advance for code-model interpretability. Prior probing of code models has largely targeted syntax/identifiers rather than formal type semantics; the parallel-corpus design, adversarial renaming, control-task selectivity, and public release of code/data are genuine strengths that make the empirical findings reusable and falsifiable. The results could also motivate hybrid type-constrained decoding that exploits internal representations rather than purely external checkers.

major comments (3)
  1. §2.1 (Dataset and Design) and Evaluation Metrics: the claim that high selectivity demonstrates encoding of formal type semantics (rather than residual statistical regularities of the synthetic distribution) is not fully secured. Even after independent identifier/literal randomization, every example is built from the same type-compatible skeletons with unique input types and uniformly distributed pairs; operators, literals and control-flow patterns that systematically co-vary with type (e.g., +1 with int, .length()/len with String/list) remain available. The per-token randomized-label control destroys only label–activation associations, not these structural co-occurrences. Consequently, non-trivial zero-shot transfer (pyUnt o Java) and Task-2 selectivity may still reflect synthetic distributional regularities. A stronger control (e.g., type-preserving but operator/literal-scrambled progra
  2. Tables 1–3 and §2.2 (Task 2): the interpretation of Application Result prediction as evidence that hidden states “linearly encode the result type implied by typed function application” overstates what the probe recovers. Because candidates always have unique input types and the call site is of the fixed form b:τb=f_⟨FIM⟩(a), success on Task 2 can be achieved by recovering τa and looking up the matching signature rather than performing a genuine compositional inference step. Disentangling these two routes (e.g., by introducing overloaded candidates or multi-argument functions) is required for the stronger semantic claim.
  3. Limitations (§4) and experimental scope: all positive results rest on a closed set of four base types plus homogeneous lists, two languages, and purely synthetic programs of bounded complexity. While the authors correctly flag this, the central claim of a “cross-lingual type manifold” is currently supported only inside this narrow distribution; without at least one experiment on real, multi-file code or a richer type system the generality of the finding remains untested and the paper’s title/abstract over-reach.

Circularity Check

0 steps flagged

No circularity: purely empirical linear-probe study with independent controls, adversarial partitions, and cross-lingual transfer; no definitional reductions or fitted-as-prediction steps.

full rationale

The paper's load-bearing claims (linearly decodable cross-lingual type representations, including result types of typed function application, that emerge even from untyped Python and are partly robust to adversarial renaming) rest entirely on measured selectivity of linear probes trained and evaluated on held-out folds of a programmatically generated parallel corpus. Selectivity is defined as task accuracy minus a per-token randomized-label control; peak layers, absolute drops under adversarial renaming, and zero-shot transfer (e.g., pyUnt-trained probes on Java) are reported as empirical outcomes, not derived by construction from fitted parameters. There are no equations equating a claimed prediction to an input by definition, no parameters fitted to a subset and then re-presented as predictions of a closely related quantity, no load-bearing self-citations of uniqueness theorems or ansätze, and no renaming of a known empirical pattern. The synthetic-data construction and residual co-occurrence regularities raise validity questions about what the probes recover, but those are correctness/generalization concerns, not circularity of the derivation chain. The work is self-contained against its own benchmarks.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard interpretability practice (linear probes recover encoded features), a domain mapping between Java and Python base and list types, and several design choices about where and how to probe. No numerical free parameters are fitted to produce the main selectivity numbers; the free parameters that exist are ordinary training hyperparameters. The only invented descriptive entity is the ‘cross-lingual type manifold’.

free parameters (1)
  • probe learning rate / weight decay / epochs
    AdamW lr=1e-4, weight decay 0.01, 20 epochs are fixed by hand; they affect absolute accuracy but are not claimed as scientific constants.
axioms (4)
  • domain assumption Linear probe selectivity above a randomized-label control indicates that the residual stream linearly encodes the probed property.
    Standard assumption of the probing literature (Alain & Bengio; Hewitt & Liang); invoked throughout the Probing Setup and Results sections.
  • domain assumption Base types and list containers of Java and Python can be placed in a direct operational correspondence (str↔String, list[T]↔List<T>, etc.).
    Stated in the Type Systems paragraph; required for all cross-lingual transfer claims.
  • ad hoc to paper The post-residual activation at the FIM token is a sufficient locus for recovering type information relevant to the masked call.
    Chosen in Probing Setup; no ablation of other positions is reported.
  • ad hoc to paper Independent randomization of identifiers and literals per partition eliminates lexical shortcuts that would otherwise allow probes to succeed without type structure.
    Dataset and Design paragraph; underpins the claim that success reflects type-level rather than surface cues.
invented entities (1)
  • cross-lingual type manifold no independent evidence
    purpose: Descriptive label for the aligned type-relevant directions recovered across Java and Python residual streams.
    Introduced in the Conclusions; no independent geometric characterization (e.g., SVCCA) is provided in the present experiments.

pith-pipeline@v1.1.0-grok45 · 15514 in / 2566 out tokens · 39966 ms · 2026-07-10T09:07:08.492872+00:00 · methodology

0 comments
read the original abstract

State-of-the-art code models achieve impressive performance, yet the extent to which they internally encode type information remains poorly understood. We probe the residual streams of pretrained code models for internal type representations using a parallel dataset of Java and Python code examples. Our results show that cross-lingual type representations emerge even from untyped code. Moreover, we test whether hidden states linearly encode the result type implied by typed function application by training probes on one language to infer argument and result types in the other. Finally, we find that this structure is partly robust to lexical perturbations and cross-language syntactic variations. To the best of our knowledge, prior work on interpretability of code models has not directly targeted formal type semantics or cross-lingual type representations. We release our code and datasets.

Figures

Figures reproduced from arXiv: 2607.08339 by Fausto Carcassi, Giuliano Gorgone.

Figure 1
Figure 1. Figure 1: Example dataset instances in Python (left) and Java (right) in SantaCoder’s [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Row-normalized confusion matrices for SantaCoder (1.1B) at layer 11 on Task 1. Probes are trained and evaluated on pyUnt under standard (left) and adversarial (right) settings. Per-layer Probing Performance. Selectivity peaks in mid-to-late layers (Santa￾Coder: L11–L16, CodeLlama: L14–L18; [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Row-normalized confusion matrices for CodeLlama (7B) at layer 16 on Task 1. Probes are trained and evaluated on pyUnt under standard (left) and adversarial (right) settings. late-layer selectivity is relatively stable in standard settings, it consistently drops across all models when facing adversarial renaming (Figs. 11 and 12). Disentangling Type-Level Semantics from Lexical-Level Semantics. Figures 2–3 … view at source ↗
Figure 4
Figure 4. Figure 4: Row-normalized confusion matrices for Task 2 (Application Result) with [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Row-normalized confusion matrices on Task 1. Probes are trained and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Row-normalized confusion matrices on Task 1. Probes are trained and [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Row-normalized confusion matrices for Task 2 (Application Result). Probes [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Row-normalized confusion matrices for Task 2 (Application Result). Probes [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Row-normalized confusion matrices for Task 2 (Application Result). [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Layer-wise selectivity and accuracy (Tasks 1 & 2) with probes trained [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 2 internal anchors

  1. [1]

    In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings

    Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net (2017),https://openreview.net/forum?id=HJ4-rAVtl

    work page 2017

  2. [2]

    Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., et al.: Santacoder: don’t reach for the stars! (2023),https://arxiv.org/abs/2301.03988

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Ammar, W., Mulcaire, G., Tsvetkov, Y., Lample, G., Dyer, C., Smith, N.A.: Massively multilingual word embeddings (2016),https://arxiv.org/abs/1602. 01925

    work page 2016

  4. [4]

    Computa- tional Linguistics48(1), 207–219 (Mar 2022).https://doi.org/10.1162/coli_a_ 00422,https://aclanthology.org/2022.cl-1.7/

    Belinkov, Y.: Probing classifiers: Promises, shortcomings, and advances. Computa- tional Linguistics48(1), 207–219 (Mar 2022).https://doi.org/10.1162/coli_a_ 00422,https://aclanthology.org/2022.cl-1.7/

    work page doi:10.1162/coli_a_ 2022

  5. [5]

    ACM Comput

    Cardelli, L., Wegner, P.: On understanding types, data abstraction, and polymor- phism. ACM Comput. Surv.17(4), 471–523 (Dec 1985).https://doi.org/10. 1145/6041.6042,https://doi.org/10.1145/6041.6042

    work page doi:10.1145/6041.6042 1985

  6. [6]

    Transformer Circuits Thread (2022), https:// transformer-circuits.pub/2022/toy_model/index.html

    Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, et al.: Toy models of superposition. Transformer Circuits Thread (2022), https:// transformer-circuits.pub/2022/toy_model/index.html

    work page 2022

  7. [7]

    In: Inui, K., Jiang, J., Ng, V., Wan, X

    Hewitt, J., Liang, P.: Designing and interpreting probes with control tasks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 2733–

    work page 2019

  8. [8]

    https://doi.org/10.18653/v1/D19-1275

    Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1275

    work page doi:10.18653/v1/d19-1275 2019

  9. [9]

    Kudugunta, S.R., Bapna, A., Caswell, I., Arivazhagan, N., Firat, O.: Investigating multilingual nmt representations at scale (2019),https://arxiv.org/abs/1909. 02197

    work page 2019

  10. [10]

    In: Belinkov, Y., Mueller, A., Kim, N., Mohebbi, H., Chen, H., Arad, D., Sarti, G

    Lucchetti, F., Guha, A.: Understanding how CodeLLMs (mis)predict types with activation steering. In: Belinkov, Y., Mueller, A., Kim, N., Mohebbi, H., Chen, H., Arad, D., Sarti, G. (eds.) Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. pp. 358–397. Association for Computational Linguistics, Suzhou, China (No...

    work page 2025

  11. [11]

    Mündler, N., He, J., Wang, H., Sen, K., Song, D., Vechev, M.: Type-constrained code generation with language models. Proc. ACM Program. Lang.9(PLDI) (Jun 2025).https://doi.org/10.1145/3729274

    work page doi:10.1145/3729274 2025

  12. [12]

    https://github.com/ TransformerLensOrg/TransformerLens(2022)

    Nanda, N., Bloom, J.: Transformerlens. https://github.com/ TransformerLensOrg/TransformerLens(2022)

    work page 2022

  13. [13]

    Rozière,B.,Gehring,J.,Gloeckle,F.,Sootla,S.,Gat,I.,Tan,X.E.,etal.:Codellama: Open foundation models for code (2024),https://arxiv.org/abs/2308.12950

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Fine-grained prediction of reading comprehension from eye movements,

    Troshin, S., Chirkova, N.: Probing pretrained models of source codes. In: Bast- ings, J., Belinkov, Y., Elazar, Y., Hupkes, D., Saphra, N., Wiegreffe, S. (eds.) Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. pp. 371–383 (2022).https://doi.org/10.18653/v1/ 2022.blackboxnlp-1.31

    work page doi:10.18653/v1/ 2022

  15. [15]

    Zhang, Z., Nanda, N.: Towards best practices of activation patching in language models: Metrics and methods. In: International Conference on Learning Representa- tions (ICLR) (2024), https://proceedings.iclr.cc/paper_files/paper/2024/ file/06a52a54c8ee03cd86771136bc91eb1f-Paper-Conference.pdf 12 G. Gorgone, F. Carcassi A Confusion Matrices: Task1 (Java,py...

    work page 2024