Differential syntactic and semantic encoding in LLMs

Alessandro Laio; Marco Baroni; Santiago Acevedo

REVIEW 4 major objections 3 minor 10 references

Differential syntactic and semantic encoding in LLMs

T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models store syntax and meaning as partly separate, linearly removable directions in their internal representations.

desk verdict A well-crafted, useful interpretability paper whose specificity claims are undercut by un-matched projection magnitudes in the random-ablated controls; worth a careful referee and revision, but not yet fully established. read the letter →

arxiv 2601.04765 v5 pith:TCE6OGLN submitted 2026-01-08 cs.CL cs.AIcs.LGphysics.comp-ph

Santiago Acevedo , Alessandro Laio , Marco Baroni This is my paper

classification cs.CLcs.AIcs.LGphysics.comp-ph

keywords syntacticrepresentationsemanticlinearencodingcentroidablationLLMinterpretabilityneighborhoodsimilaritypart-of-speechtemplatestranslation-paraphrasedata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the grammar of a sentence and its meaning live in different places inside a large language model. Using neural activations of sentences that share a part-of-speech template, the authors build a 'syntax centroid'; using translations of each sentence into six languages, they build a 'semantic centroid'. Subtracting a sentence's projection onto these centroids selectively weakens similarity to syntactically or semantically matched sentences, indicating that a good share of both kinds of information is linearly encoded. The two signals also separate by depth: syntax similarity is high from early layers onward, while semantic similarity peaks in middle layers. The authors conclude that syntax and semantics are partially decoupled, with syntax more robust to removal of the other component — a result that speaks directly to the old linguistic debate about the autonomy of syntax.

What carries the argument

The load-bearing objects are two centroid vectors. A syntax centroid Si is the average of the hidden representations of several sentences sharing the same part-of-speech template as Xi but with different words, so common structure survives and meaning averages out. A semantic centroid Ti is the average of the representations of Xi translated into six languages, so shared meaning survives and surface form changes. The operation that carries the argument is projection ablation: replacing Xi with Xi − (Xi·Si/|Si|²)Si, making the sentence vector orthogonal to the centroid direction. Similarity is measured by rank-based neighborhood overlap between pairs of matched sentences, which the authors ar

What would settle it

Take POS templates that are matched for word frequency, concreteness, and thematic content while differing only in structure, or construct templates from semantically empty function-word frames; if subtracting the resulting syntax centroid no longer ablates syntax similarity, the reported decoupling is an artifact of lexical contamination. Conversely, if translations into structurally parallel languages produce semantic centroids that, when subtracted, still ablate syntax similarity, the semantic centroid is carrying structural residue.

Watch

Extended reading notes

Core claim

LLM sentence vectors carry syntax and meaning along partly separable linear axes. A syntax centroid (average of same-POS sentences) and a semantic centroid (average of a sentence's translations into six languages) capture enough of each signal that subtracting a sentence's projection removes most corresponding similarity: syntax-twin similarity falls from >0.7, paraphrase similarity falls mainly in middle layers. The cross-effect is asymmetric: syntax similarity survives semantic removal, while semantic similarity is partly damaged by syntactic removal. Probes confirm it: syntax ablation collapses POS classification from 85% to 10%, semantic ablation drops paraphrase recall from 85% to 66%,

Load-bearing premise

The argument stands or falls on the purity of the centroids: the averaged same-POS sentences must carry syntax without leftover lexical or meaning regularities, and the averaged translations must carry meaning without leftover structural residue; the paper itself flags this possibility and checks only samples.

Editorial extensions

If this is right

If syntax and semantics are linearly encoded, simple vector arithmetic — subtracting a centroid direction — can remove or attenuate one kind of linguistic information from LLM internal states without retraining.
Syntax appears more autonomous than meaning: syntactic similarity survives semantic ablation, while semantic similarity is partially damaged by syntactic ablation, consistent with syntax acting as an independent scaffolding.
The layer profiles are diagnostic: constant syntax similarity across depths versus a semantic peak in middle layers implies that a single vector-based probe can localize where meaning is built.
Probes confirm the mechanism behaviorally: POS-template classification collapses under syntax ablation, paraphrase recall drops under semantic ablation, and syntax ablation even slightly improves paraphrase recall.
Because the results reproduce across a very large model and two smaller models, the decoupling is not an artifact of one architecture's scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the centroids are pure, the same subtraction could be turned into a control knob for generation: suppressing a syntactic centroid direction may let a model paraphrase without changing meaning, and suppressing a semantic centroid direction may let it reorder words while keeping grammar intact — a testable steering application the paper leaves open.
The paper's own limitation note says the centroids explain at most about 40% of the squared norm in central layers; an inference is that richer or non-linear centroids should recover more of that residual if the linear-encoding claim is complete, and if they do not, the residual is genuinely non-linear code.
The asymmetry aligns with the idea that syntax is a shared skeleton across languages; a natural test is whether syntax centroids built from one language ablate syntax similarity in another language more effectively than semantic centroids do — predicting a language-general syntax subspace.
Token averaging helps semantic similarity but hurts syntactic similarity; an editor's extension is that this is consistent with syntax living in high-frequency, position-sensitive components and meaning in low-frequency, position-invariant components — the frequency decomposition the authors mention as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before the next meeting: it claims a simple linear handle on syntax and semantics in DeepSeek-V3 and two smaller models. The genuinely new parts are the semantic centroid built from six translations, the cross-layer differential profiles, and the asymmetric decoupling (semantic ablation leaves syntax similarity intact, while syntax ablation weakens semantic similarity). It is clearly written and methodologically careful in several ways: centroids are built from held-out sentences, the similarity metric is rank-based and robust to anisotropy, random-matching controls give baseline zero, and the main qualitative pattern replicates across three models. The probe results are striking—syntax ablation collapses POS classification from 0.85 to 0.10, while semantic ablation barely moves it, and semantic recall actually improves slightly after syntax ablation.

The soft spot is a real confound in the random-ablated controls. The true centroid is an average of sentences that share the template or meaning with Xi, so its projection on Xi is large. A permuted centroid is effectively random, so the removed component is much smaller. The larger similarity drop under the true centroid could therefore reflect intervention strength—removing a larger-magnitude vector perturbs neighborhoods more—rather than specific syntax/semantic content. This affects Figures 1–4 and Table 3. The authors show awareness of spurious signals from direct subtraction (Appendix D) but never control for the magnitude of the removed projection. A matched control, e.g., a random direction scaled to the same removed norm, is needed before the specificity claim is solid. There is also partial circularity: the syntax centroid is the class mean of the same POS-template labels used to define similarity; the held-out construction mitigates but does not eliminate it. The semantic centroid is less circular because it uses translations rather than paraphrases, but the magnitude confound still applies. Minor issues: Table 3 is a single run with no error bars, and code/data are promised but not yet released.

I agree with the conditional verdict. The qualitative layer profiles (syntax stable, semantics peaked in middle layers) are probably robust, and the paper is honest about its limitations. But the central claim of specifically and linearly encoded, decoupled syntax/semantics is not yet established. This is a valuable empirical package that deserves a serious referee and likely acceptance after the magnitude-matched controls are added. I would bring it to reading group to discuss the confound, and I would cite it with a caveat as evidence for the layer profiles, not for the decoupling claim.

Recommendation: send to peer review, with emphasis on the projection-magnitude issue.

Referee Report

4 major / 3 minor

Summary. This paper proposes that syntax and semantics are partially linearly encoded in the residual stream of LLMs. The authors construct syntactic centroids by averaging hidden states of sentences sharing a POS template, and semantic centroids by averaging translations of a sentence into six languages. Using a rank-based neighborhood similarity (an adaptation of Information Imbalance), they show in DeepSeek-V3 (and qualitatively in Qwen2-7b and Gemma3-12b) that subtracting the projection of a sentence onto its syntactic or semantic centroid reduces similarity to syntactically or semantically matched sentences, with distinct layer profiles. Cross-ablations suggest an asymmetry: semantic ablation leaves syntax similarity nearly intact, while syntactic ablation reduces paraphrase similarity. A POS-template linear probe and a paraphrase-recall probe corroborate the ablation effects. The paper interprets the results as evidence for linear, partially separable encoding of syntax and semantics, with a semantic core in middle layers.

Significance. If the findings survive revision, they would be a useful contribution to LLM interpretability and to the debate on the autonomy of syntax. The method is simple and potentially widely applicable; the rank-based similarity metric with random-matching control and the replication across DeepSeek-V3, Qwen2-7b, and Gemma3-12b are strengths. The manuscript also provides code and data (subject to the availability inconsistency noted below) and includes a careful control in Appendix D showing that direct subtraction of centroids can create spurious signal. However, the central ablation claims currently rest on random controls that are not matched for intervention strength, and the cross-ablation design is not symmetric with respect to which sentence the centroid is derived from. These issues are fixable and do not necessarily invalidate the qualitative picture, but they need to be addressed before the conclusions about selectivity and decoupling can be accepted.

major comments (4)

[Eq. (1), Figs. 1–4, Table 3] The random-ablated controls do not match the norm of the removed projection. For a sentence X_i, the true syntactic centroid S_i is an average of sentences sharing X_i's POS template, so it has a large projection onto X_i; a randomly picked centroid typically has a small projection. The same holds for semantic centroids. A larger similarity drop after true ablation is therefore expected even if the centroid direction carried no syntactic/semantic content, because removing a larger component perturbs distances more. Since S_i is also a class-mean of the very property used to define the matched pairs, some drop is guaranteed by construction; the informative comparison is thus the random control, which must be matched in removed norm. Please add controls where the removed component is a random direction (or a permuted centroid) scaled to the same projection norm as the true ablation, and re
[§2.1, Table 2, Appendix A.1] As written, the syntactic centroid used for X_i is the average of all its syntax twins, including the twin s'_i with which X_i is compared (Table 2). If this definition is used in the ablations, the operation applied to X_i is not independent of the test pair: it explicitly removes a component along the comparison sentence. The alternative described in Appendix A.1 (centroids built from original sentences to ablate twins and vice versa) still has the property that each member of a pair is ablated with a centroid containing the other member. This can spuriously inflate or distort the similarity drop. Please state precisely which centroids are used for each member, and recalculate with leave-one-pair-out centroids to ensure the paired sentence is not part of the centroid construction.
[§3.3, Figs. 3 and 4] The cross-ablation experiments are not symmetric with respect to the derivation of centroids. In Fig. 3, the semantic centroid T_i is derived from translations of the original sentence X_i; the syntax twin s'_i is ablated using this same T_i, even though T_i is not a semantic centroid for s'_i. In Fig. 4, the paraphrase P_i is ablated using the syntax centroid S_i of X_i, rather than a syntax centroid built from sentences with P_i's own POS template. The observed asymmetry may therefore simply reflect that each centroid is aligned with X_i but not with its partner: the partner is only weakly perturbed, so syntax similarity is preserved (Fig. 3), while X_i is strongly moved, so semantic similarity drops (Fig. 4). To test whether syntax and semantics are truly separable, each item must be ablated with a centroid constructed from its own matched sentences (translations of the twin, or synta
[§2.1, footnote 1] The interpretation of the results as 'syntax' vs 'semantics' rests on the assumption that POS-matched twins share syntax but not lexical/semantic structure, and that translations capture meaning without systematic structural residue. The footnote acknowledges that identical POS sequences can correspond to different syntactic structures, and only manual sample checks are reported. A more systematic validation is needed, e.g., annotating a sample with parse trees or measuring per-template lexical/embedding similarity, to show that the syntactic centroids are not contaminated by lexical or template-specific style. Likewise, the semantic centroids should be checked for accidental structural regularities across the six languages. Without such controls, the decoupling results could reflect the properties of the proxies rather than of the underlying representations.

minor comments (3)

[Code and data availability / Appendix J.1] The main text states that code and data are publicly available at a GitHub URL, while Appendix J.1 says they will be released upon acceptance. Please reconcile this inconsistency.
[Table 3 and Appendix H.2] The probe results in Table 3 are single-run experiments without variance estimates; the table caption should state this clearly, since the main text uses these values to support quantitative claims such as the 5% increase in paraphrase recall.
[Fig. 5 caption] The vertical axis is cut at 0.6 while the fractions sum to 1; please state this explicitly in the caption or show the full range.

Circularity Check

3 steps flagged · score 5.0 of 10

Same-ablation and syntax-probe results are partially in-sample: the syntactic centroid includes the paired twin, and random controls are unmatched in projection magnitude; cross-ablation retains independent content.

fitted input called prediction [Sec. 2.1 (Table 2), Eq. (1), Sec. 3.1 / Fig. 1]
"for each original sentence Xi we gather all the syntax twins that share its POS template, that we call sj_i with j>=0 and we average them to construct the syntactic centroid vector, Si."

Table 2's j=0 member is exactly the paired syntax twin s0_i used in the Fig. 1 similarity comparison. Eq. (1) removes from Xi its projection onto Si before measuring Xi versus s0_i similarity. Because Si is an average containing s0_i, the ablation direction is in-sample for the measured pair: part of the similarity drop is the algebraic consequence of orthogonalizing Xi against a direction containing its own comparison sentence, not an independent demonstration that syntax is linearly encoded.
fitted input called prediction [Sec. 3.1 (Fig. 1), Sec. 3.2 (Fig. 2), with Eq. (1)]
"As a further control, we misaligned the syntax centroids, that is, we subtracted the projection corresponding to a different, randomly-picked POS template from each sentence in a pair. ... As a control, if we permute the semantic centroids so that each sentence is orthogonalized to an unrelated semantic centroid, the reduction in similarity between pairs is smaller."

The true centroid Si (or Ti) is by construction an average of sentences sharing Xi's POS template (or its semantic translations), so the removed vector in Eq. (1) has a much larger projection norm than a permuted centroid does. A random centroid is typically nearly orthogonal to Xi and removes a much smaller component. The rank-based similarity drop scales with how much of Xi is removed, so the larger drop under the true centroid is expected from projection magnitude alone even if the centroid carried no syntax- or semantics-specific content; the reported specificity contrast is therefore confounded with intervention strength.

1 more flagged steps

fitted input called prediction [Appendix H.1, Table 3]
"The training set corresponds to the set of syntax twins, s0_i, leaving the original sentences Xi and their different ablations as test sets, reported in Table 3."

The syntax centroids used to ablate the test original sentences are built by averaging the same syntax-twin items that constitute the POS classifier's training set. The syntax-ablation collapse in Table 3 (0.85 to 0.10) therefore removes directions estimated from the classifier's own training examples, so part of the probe effect is train/test leakage rather than an out-of-sample prediction about linear syntactic encoding.

full rationale

The derivation chain is not uniformly circular. The similarity metric is imported from Glielmo et al. (2022) via Acevedo et al. (2025) and is independently described; there is no load-bearing self-citation or imported uniqueness theorem. The semantic centroid is built from translations that are excluded from the paraphrase test pairs, the cross-ablation in Fig. 3 uses centroids not constructed from the tested syntax twins, and the main profiles are reproduced on Qwen2-7b and Gemma3-12b, so the central claim retains independent empirical content. However, the same-ablation figures (Figs. 1-2), the specificity controls, and the syntax-probe row of Table 3 are partially in-sample: the syntactic centroid includes the exact matched twin, and the random controls are not matched in projection magnitude. These features make a nontrivial part of the reported selectivity follow by construction rather than from an independent property of the representations. Score 5 reflects partial circularity, not full equivalence of the derivation to its inputs.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The central method introduces no fitted parameters; the main constructs are centroid directions defined as averages of labeled classes. The load-bearing assumptions are that POS-template equality and translation averaging yield pure proxies for syntax and semantics, and that rank-based similarity plus linear projection removal faithfully expose the underlying encoding.

free parameters (1)

probe L2 regularization C = 10^3
Used in the POS-template logistic regression probe (Appendix H.1); selected by sweeping across values and tracking best performance. This affects Table 3 but not the main similarity results.

assumptions (5)

domain assumption POS template equality is a proxy for syntactic structure
Section 2.1 and footnote 1 acknowledge that identical POS sequences can correspond to different syntactic structures; the paper relies on manual checks of sample templates.
domain assumption Cross-lingual translations preserve meaning, and averaging them isolates semantics from syntax
Section 2.1/Appendix A.3: the semantic centroid Ti is the average of translations. If translations carry structural residue or meaning drift, the centroid is impure.
domain assumption Rank-based neighborhood similarity captures representation-relevant information
Section 2.2 and Appendix B: the paper relies on rank-based similarity because CKA is weak in high dimensions; this metric's relevance to syntax/semantics is assumed.
domain assumption Projection removal is a valid way to ablate a linear feature
Eq. (1): the paper assumes the target information is encoded in a single linear direction per sentence, so subtracting a projection removes it without introducing distortions.
domain assumption Activation clipping and normalized-L2 distances do not distort the signal
Appendix B: activations are clipped to quintiles 0.05/0.95 and distances are computed on unit-normalized vectors, following prior work; this is a preprocessing choice that could affect the signal.

invented entities (1)

Syntactic and semantic centroid directions Si and Ti
purpose: Linear proxies for the syntactic or semantic content of a sentence, used for projection ablation.
The centroids are constructed as class means from the same data used for evaluation, so they have no falsifiable handle outside this paper; their validity as pure syntax/semantics directions is the central premise.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Differential syntactic and semantic encoding in LLMs." pith.science (2026). https://pith.science/paper/TCE6OGLN

@misc{pith2026260104765,
  author       = {Pith},
  title        = {Pith review of: Differential syntactic and semantic encoding in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCE6OGLN}},
  note         = {Machine review of arXiv:2601.04765}
}

read the original abstract

We study how syntactic and semantic information is encoded in inner layer representations of Large Language Models (LLMs), focusing on the very large DeepSeek-V3. We find that, by averaging hidden-representation vectors of sentences sharing syntactic structure or meaning, we obtain vectors that capture a significant proportion of the syntactic and semantic information contained in the representations. In particular, subtracting these syntactic and semantic ``centroids'' from sentence vectors strongly affects their similarity with syntactically and semantically matched sentences, respectively, suggesting that syntax and semantics are, at least partially, linearly encoded. We also find that the cross-layer encoding profiles of syntax and semantics are different, and that the two signals can to some extent be decoupled, suggesting differential encoding of these two types of linguistic information in LLM representations.

Figures

Figures reproduced from arXiv: 2601.04765 by the authors.

**Figure 1.** Syntax similarity and its ablation. Similarity between equal-syntax sentences (syntax twins), such as those presented in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Semantic similarity and its ablation. Similarity between English sentences and their (English) paraphrases. Panels a) and b) represent sentences by token concatenation and average, respectively. The shaded colored areas represent 1 standard deviation, calculated by subsampling five times half of the samples. randomly-picked POS template from each sentence in a pair. We observe that this operation produces a smaller … view at source ↗

**Figure 3.** Syntactic similarity with semantic ablation. The shaded colored areas represent 1 standard deviation, calculated by subsampling five times half of the samples. 3.3. Ablations across linguistic components Having measured the syntactic and semantic similarity of hidden representations, we next ask if these two quantities can be dissociated. We start by removing the semantic centroids Ti (see Sec. 3.2) from syntactical… view at source ↗

Figures from the paper (13 more)

**Figure 5.** Figure 5: Decomposition of sentence vectors. Average fraction of squared norm from sentence activations Xi contained in syntax centroids Si (blue) and semantic centroids T i (green), across the network. The gray sections represent the residual fraction of norm that is not captur…

**Figure 6.** Figure 6: Statistics of original sentences Xi after filtering. Please do not write a script: complete the task manually. The generated sentences should be in grammatical, fluent and meaningful English, and very importantly, they should be very different from the input sentences …

**Figure 7.** Figure 7: Random matching control. Syntactic and semantic similarities with and without ablation on misaligned data, i.e., performing a batch-shuffling on one of the spaces. This destroys similarities between representations. The shaded colored areas represent 1 standard deviati…

**Figure 8.** Figure 8: shows in blue the similarity between English paraphrases processed by DeepSeek-V3 (same as [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: shows the similarity between our original English sentences and their translations in each of the languages used to form the semantic centroids. The similarity profiles closely follow the one obtained for English paraphrases in [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: shows the semantic ablation of paraphrase similarity given semantic centroids composed by pooling an increasing number of languages, up to the 6 we have data for. We don’t find significant changes between 4 and 6 languages, suggesting that there is no need to include …

**Figure 11.** Figure 11: Syntax similarity and its ablation across 3 LLMs. Similarity between equal-syntax sentences (syntax twins), such as those presented in [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: Semantic similarity and its ablation for Qwen2-7b. Similarity between English sentences and their (English) paraphrases, with the same setup of [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]

**Figure 13.** Figure 13: Semantic similarity and its ablation for Gemma3-12b. Similarity between English sentences and their (English) paraphrases, with the same setup of [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]

**Figure 14.** Figure 14: Syntactic similarity with semantic ablation for Qwen2-7b and Gemma3-12b. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

**Figure 15.** Figure 15: Semantic similarity with syntax ablation for Qwen2-7b and Gemma3-12b. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]

**Figure 16.** Figure 16: Decomposition of sentence vectors for Qwen2-7b. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]

**Figure 17.** Figure 17: Decomposition of sentence vectors for Gemma3-12b. Same setup as [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 3 linked inside Pith

[2]

Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J., Laio, A., and Baroni, M

URL https://proceedings.mlr.press/ v139/caucheteux21a.html. Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J., Laio, A., and Baroni, M. Emergence of a high- dimensional abstraction phase in language transformers. InProceedings of ICLR, Singapore, 2025. Published online: https://openreview.net/group?id= ICLR.cc/2025/Conference. Chomsky, N.Aspects of ...

arXiv 2025
[8]

Simon, P

URL https://openreview.net/forum? id=x2780VcMOI. Simon, P. J. D., Chemla, E., King, J.-R., and Lakretz, Y . Probing syntax in large language models: Successes and remaining challenges. InSecond Conference on Lan- guage Modeling, 2025. URL https://openreview. net/forum?id=nrZysNmJ0n. Skean, O., Arefin, R., Zhao, D., Patel, N., Naghiyev, J., Le- Cun, Y ., a...

2025
[300]

findings-acl.300/

URL https://aclanthology.org/2021. findings-acl.300/. Lee, A., Weber, M., Vi´egas, F., and Wattenberg, M. Shared global and local geometry of language model embed- dings. InProceedings of COLM, Montreal, Canada,

2021
[315]

emnlp-main.315/

URL https://aclanthology.org/2024. emnlp-main.315/. Qwen Team. Qwen2 technical report, 2024. Shaib, C., Suriyakumar, V . M., Sagun, L., Wallace, B. C., and Ghassemi, M. Learning the wrong lessons: Syntactic- domain spurious correlations in language models, 2025. URLhttps://arxiv.org/abs/2509.21155. Simon, P. J. D., d’Ascoli, S., Chemla, E., Lakretz, Y ., ...

arXiv 2024
[2018]

Niu, J., Lu, W., and Penn, G

URL https://openreview.net/forum? id=HkuGJ3kCb. Niu, J., Lu, W., and Penn, G. Does BERT rediscover a classical NLP pipeline? InProceedings of COLING, pp. 3143–3153, Gyeongju, Korea, 2022. Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models. InForty-first International Conference on Ma- c...

2022
[2020]

New/Modified BSD

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 3acb2a202ae4bea8840224e6fce16fd0-Paper. pdf. 11 Differential syntactic and semantic encoding in LLMs Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical NLP pipeline. InProceedings of ACL, pp. 4593–4601, Florence, Italy, 2019. Wild, R., Wodaczek, F., Del Tatto, V ., Cheng, B....

2020
[2021]

org/CorpusID:235435759

URL https://api.semanticscholar. org/CorpusID:235435759. Bondarenko, Y ., Nagel, M., and Blankevoort, T. Quantizable transformers: Removing outliers by helping attention heads do nothing. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=sbusw6LD41. Brinkmann, J., Wendler, C., Bartelt, C., and...

2023
[2024]

Jawahar, G., Sagot, B., and Seddah, D

URL https://proceedings.mlr.press/ v235/huh24a.html. Jawahar, G., Sagot, B., and Seddah, D. What does BERT learn about the structure of language? InProceedings of ACL, pp. 3651–3657, Florence, Italy, 2019. Jørgensen, O., Cope, D., Schoots, N., and Shanahan, M. Im- proving activation steering in language models with mean- centring.CoRR, abs/2312.03813, 202...

arXiv doi:10.48550/arxiv.2312.03813 2019

Show all 10 references

[2025]

net/forum?id=aJDykpJAYF

Published online https://openreview. net/forum?id=aJDykpJAYF. Levy, R., Kim, Y ., and Fox, D. The science of language in the era of generative AI. https://mit-genai. pubpub.org/pub/ak3evnmm, may 6 2025. Li, M. and Subramani, N. Echoes of BERT: Do modern language models redisco...

2025 arXiv
[8856]

Survey Certification, Expert Certifi- cation

URL https://openreview.net/forum? id=Hiq7lUh4Yn. Survey Certification, Expert Certifi- cation. Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. InFirst Con- ference on Language Modeling, 2024. URL https: //openreview.net/forum?id=F7aA...

2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J., Laio, A., and Baroni, M

URL https://proceedings.mlr.press/ v139/caucheteux21a.html. Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J., Laio, A., and Baroni, M. Emergence of a high- dimensional abstraction phase in language transformers. InProceedings of ICLR, Singapore, 2025. Published online: https://openreview.net/group?id= ICLR.cc/2025/Conference. Chomsky, N.Aspects of ...

arXiv 2025

[2] [8]

Simon, P

URL https://openreview.net/forum? id=x2780VcMOI. Simon, P. J. D., Chemla, E., King, J.-R., and Lakretz, Y . Probing syntax in large language models: Successes and remaining challenges. InSecond Conference on Lan- guage Modeling, 2025. URL https://openreview. net/forum?id=nrZysNmJ0n. Skean, O., Arefin, R., Zhao, D., Patel, N., Naghiyev, J., Le- Cun, Y ., a...

2025

[3] [300]

findings-acl.300/

URL https://aclanthology.org/2021. findings-acl.300/. Lee, A., Weber, M., Vi´egas, F., and Wattenberg, M. Shared global and local geometry of language model embed- dings. InProceedings of COLM, Montreal, Canada,

2021

[4] [315]

emnlp-main.315/

URL https://aclanthology.org/2024. emnlp-main.315/. Qwen Team. Qwen2 technical report, 2024. Shaib, C., Suriyakumar, V . M., Sagun, L., Wallace, B. C., and Ghassemi, M. Learning the wrong lessons: Syntactic- domain spurious correlations in language models, 2025. URLhttps://arxiv.org/abs/2509.21155. Simon, P. J. D., d’Ascoli, S., Chemla, E., Lakretz, Y ., ...

arXiv 2024

[5] [2018]

Niu, J., Lu, W., and Penn, G

URL https://openreview.net/forum? id=HkuGJ3kCb. Niu, J., Lu, W., and Penn, G. Does BERT rediscover a classical NLP pipeline? InProceedings of COLING, pp. 3143–3153, Gyeongju, Korea, 2022. Park, K., Choe, Y . J., and Veitch, V . The linear represen- tation hypothesis and the geometry of large language models. InForty-first International Conference on Ma- c...

2022

[6] [2020]

New/Modified BSD

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 3acb2a202ae4bea8840224e6fce16fd0-Paper. pdf. 11 Differential syntactic and semantic encoding in LLMs Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical NLP pipeline. InProceedings of ACL, pp. 4593–4601, Florence, Italy, 2019. Wild, R., Wodaczek, F., Del Tatto, V ., Cheng, B....

2020

[7] [2021]

org/CorpusID:235435759

URL https://api.semanticscholar. org/CorpusID:235435759. Bondarenko, Y ., Nagel, M., and Blankevoort, T. Quantizable transformers: Removing outliers by helping attention heads do nothing. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=sbusw6LD41. Brinkmann, J., Wendler, C., Bartelt, C., and...

2023

[8] [2024]

Jawahar, G., Sagot, B., and Seddah, D

URL https://proceedings.mlr.press/ v235/huh24a.html. Jawahar, G., Sagot, B., and Seddah, D. What does BERT learn about the structure of language? InProceedings of ACL, pp. 3651–3657, Florence, Italy, 2019. Jørgensen, O., Cope, D., Schoots, N., and Shanahan, M. Im- proving activation steering in language models with mean- centring.CoRR, abs/2312.03813, 202...

arXiv doi:10.48550/arxiv.2312.03813 2019