Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BERT assembles dependency trees bottom-up: micro-syntactic structures appear in lower layers, and the root-verb macro-structure is integrated later.

desk verdict The bottom-up derivation claim is likely an artifact of the macro/micro edge-set definition; the paper needs to settle that before the main result can be trusted. read the letter →

arxiv 2506.21861 v1 pith:565LTSFI submitted 2025-06-27 cs.CL

Taiga Someya , Ryo Yoshida , Hitomi Yanaka , Yohei Oseki This is my paper · ORCID
classification cs.CL
keywords derivationalprobingstructuralexpectedlayermetricsyntacticstructurederivationBERTinterpretabilitysubject-verbagreementbottom-up
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Derivational Probing, which combines structural probing with the expected-layer metric to watch individual pieces of a dependency tree emerge across a language model's layers. Applied to BERT-base and BERT-large on filtered single-clause sentences from Wikitext-103, it finds that micro-syntactic structures—subject, object, and prepositional phrases—reach their expected layers before the macro-syntactic structure linking the root verb to its direct dependents. The authors interpret this as a bottom-up derivation: local dependencies are assembled first, then integrated into a global tree. A targeted subject-verb agreement test with intervening nouns suggests that integrating the macro-structure too early or too late hurts accuracy, implying an optimal integration window. If right, the paper adds a derivation timeline to the static picture of syntax in neural networks and offers a method for comparing derivation strategies across models and languages.

What carries the argument

The machinery is the fusion of two existing instruments. A structural probe learns a linear projection $B_\ell$ such that Euclidean distances between transformed embeddings approximate dependency-tree distances; per-layer Unlabeled Undirected Attachment Score (UUAS) on extracted subgraphs measures how recoverable each structure is. The expected-layer metric $E[\ell] = \sum_\ell \ell\,(S(\ell)-S(\ell-1)) / \sum_\ell (S(\ell)-S(\ell-1))$ turns the per-layer UUAS curve into a single layer index for each subgraph, using scalar-mixed embeddings (weighted averages of layers $0$ through $\ell$). Micro and macro structures are defined as edge sets in the parse tree: local relations such as nsubj, dobj, prep, and attr versus the root verb and its direct dependents. Comparing expected layers across these edge sets is what carries the bottom-up conclusion.

What would settle it

A concrete control: train the same per-layer structural probes on embeddings whose layer order has been shuffled, and on Jabberwocky sentences that strip semantic cues. If macro structures still show systematically higher expected layers than micro structures under both controls, the bottom-up reading survives; if the ranking disappears or flips, the result is an artifact of edge difficulty, probe training dynamics, or semantic confounds rather than derivation order.

Watch

Extended reading notes

Core claim

The central claim is that BERT constructs dependency trees bottom-up across layers. In all four sentence-structure sets studied, and for both BERT-base and BERT-large, the macro-syntactic structure (root verb with its direct dependents) has a consistently higher expected layer than the micro-syntactic structures (nsubj, dobj, prep, attr), meaning the local phrases become recoverable earlier. The agreement analysis adds that the timing of macro-structure formation is behaviorally consequential: BERT-base failures coincide with premature subject–verb linkage before the subject phrase's internal dependencies are stable, whereas BERT-large failures coincide with delayed linkage, and successful cases link subject and verb at an intermediate layer. The authors also report that GPT-2 small and medium show nearly parallel expected layers for micro and macro structures, which they take as evidence that bottom-up derivation is not a universal property.

Load-bearing premise

The load-bearing assumption is that the layer where a structure's parse-recovery score jumps is the layer where that structure is actually built, rather than the layer where its edges happen to become easy to recover; if edge difficulty, training dynamics, or meaning-based cues explain the timing, the bottom-up conclusion would not follow.

Editorial extensions

If this is right

  • For BERT-style encoders, syntax is not present all at once; interpretability methods that treat a single layer as 'the' syntactic representation will miss the derivation timeline.
  • The gap between macro and micro expected layers gives a quantitative signature for distinguishing bottom-up, top-down, and parallel derivations in any transformer.
  • Subject-verb agreement accuracy tracks the timing of macro-structure formation: forming it too early or too late both hurt, so layer-level interventions that shift macro integration may improve accuracy.
  • The method carries over to autoregressive models, where GPT-2 shows nearly parallel expected layers, indicating that derivation strategy is a measured model property rather than an assumption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The BERT versus GPT-2 contrast suggests that bidirectional context may let local phrases stabilize before global attachment, while left-to-right models have less freedom to defer global structure; the paper does not claim this.
  • A natural next step the paper does not run is to shift macro-structure integration by weighting or fine-tuning specific layers and check whether agreement accuracy follows the U-shaped curve implied by the failure analysis.
  • Because the pipeline is parser- and language-agnostic, it could be applied to multilingual models and, if a distance-based probe for nonterminal structures existed, to constituency grammars; the paper identifies the latter as an open limitation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Derivational Probing, which combines Hewitt and Manning's structural probe with Tenney et al.'s expected-layer metric to estimate the layer at which individual dependency subgraphs become linearly decodable in BERT. The authors define micro-syntactic structures (nsubj, dobj, prep, attr) and macro-syntactic structures (the root verb with its direct dependents), and report that macro structures have consistently higher expected layers across four sentence types in both BERT-base and BERT-large, which they interpret as bottom-up derivation. A follow-up analysis on masked-LM subject-verb agreement with 1,000 positive and 1,000 negative items suggests an 'optimal timing' for macro-structure construction. Appendix results for GPT-2 show a more parallel derivation pattern.

Significance. If the reported ordering were robust, the paper would offer a useful methodological refinement of structural probing and a concrete, falsifiable claim about layer-wise syntactic construction in transformer language models. The study has strengths: four structure sets, two BERT sizes, five seeds with reported error bars, a released code repository, and an extension to GPT-2 in the appendix. However, the central comparison is currently undermined by an apparent mismatch between the stated edge-set definitions and the computed expected layers; until that mismatch is resolved, the bottom-up conclusion cannot be evaluated. The secondary 'optimal timing' claim rests on very few failure cases without statistical support, and the interpretation of expected-layer differences as construction order is an operational assumption that should be justified or softened.

major comments (3)
  1. [§4, §5.1, Fig. 4] The manuscript must state exactly which edges enter the macro and micro UUAS computations. As written, 'macro-syntactic structures (the root verb with its direct dependents)' and micro relations 'nsubj, dobj, prep, attr' (§4) together with the grouping in §5.1 ('dependency relations emanating from the root verb') imply that the macro edge set is the union of the micro edge sets for each structure set in Table 1. In that case UUAS_macro(ℓ) is a weighted average of UUAS_micro(ℓ) at every layer, and under the monotone UUAS curves shown in Figure 3 the expected layer in Eq. (4) is a convex combination of the micro expected layers; it cannot exceed all of them. Figure 4 nevertheless shows macro with the highest expected layer in all four sets for both BERT sizes. This is an internal inconsistency, not merely an interpretation issue: either the code uses additional macro edges (e.g., det, amod, aux), or micro structures denote phrase-internal edges rather than the root-dependent relations named in the text. Please specify the edge sets precisely and, if macro is a strict superset of the named micro relations, report the composition. If macro is indeed the union, the main bottom-up claim is unsupported by Figure 4.
  2. [§7.2] The 'optimal timing' claim is based on 16 failed cases for BERT-base and 17 for BERT-large out of 1,000 items, with no significance tests on the success/failure expected-layer differences and with representative MDS plots from a single example per condition. The text describes BERT-base failures as premature macro integration and BERT-large failures as delayed integration (Figures 5–7), but the error bars in Figure 5 and the absence of any test make it impossible to distinguish a systematic timing effect from noise, and 'frequently' overstates a 1.6% failure rate. Please report per-condition expected-layer means and standard errors, a permutation test or bootstrap interval for the success–failure difference, and the number of cases on which the qualitative MDS characterization is based.
  3. [§4, Eq. (4)] The paper interprets S(ℓ) improvements as the layer at which a structure is 'constructed,' but the metric measures when a trained linear probe can recover the structure from scalar-mixed embeddings. These could diverge: an edge that is present earlier but linearly hard, or an artifact of per-layer scalar mixing weights, would shift E[ℓ] without changing construction order. The abstract's language ('emerge,' 'gradually integrated') is stronger than this operational definition licenses. Please either temper the causal framing or add a control (e.g., per-layer probes trained without scalar mixing, a non-linear probe baseline, or an analysis of probe convergence) to justify the construction-order reading.
minor comments (5)
  1. [Throughout] Typos and inconsistent notation: 'Marco' for 'Macro' in Figure 2, Table 1, and §5.1; 'd-dimentional' in §3.1; 'scaler-mixed' throughout; 'strucv-tures' in §4; 'Bertbase' in Figure 5. Please proofread and standardize the terminology.
  2. [Figures 6–7] The captions of Figures 6 and 7 do not explain what the MDS axes are, how the 'Layer0/Layer5/Layer12' labels are chosen, or how the red highlighting is derived; add a legend and a description of the plotting pipeline.
  3. [§4, §5.1] The term 'micro-syntactic structures' is used both for dependency relations (nsubj) and for phrases ('subject noun phrases') in §4; this duality should be reconciled because it directly affects how the edge sets are interpreted.
  4. [Eq. (4)] The expected-layer formula sums from ℓ=1 to L but the numerator includes S(0); please state explicitly how S(0) is defined (e.g., the embedding layer) and whether S(0) is included in the denominator as well.
  5. [Appendix A] The GPT-2 results are presented without the subject-verb agreement analysis; if included, they would strengthen the claim that GPT-2 exhibits parallel derivation rather than bottom-up derivation.

Circularity Check

1 steps flagged · score 6.0 of 10

The macro/micro comparison is definitionally degenerate: under the paper's own edge-set and UUAS definitions, the macro expected layer is a weighted average of the micro expected layers, so the headline bottom-up ordering is not an independent empirical finding.

  1. self definitional [§4 (Derivational Probing), §5.1 (Data), Eq. (4), Figure 4]
    "we introduce a distinction between macro-syntactic structures (the root verb with its direct dependents) and micro-syntactic structures (local components, such as subordinate phrases like nsubj) ... we group sentences based on dependency relations emanating from the root verb, thereby distinguishing between the overall (macro-syntactic; Marco) structure and subordinate (micro-syntactic) structures."

    Under these definitions, in each structure set the macro edge set is the union of the named micro relation edge sets. The paper defines S(ℓ) as UUAS over extracted edges and E[ℓ] by Eq. (4), a normalized weighted average of layer indices with weights ΔS(ℓ). Because UUAS is an edge-count proportion, S_macro(ℓ) = Σ_k (n_k·S_k(ℓ)) / Σ_k n_k over the constituent micro structures k. Substituting this into Eq. (4) makes E[macro] a convex combination of E[k]. Therefore macro cannot lie strictly above every micro relation. Figure 4 nevertheless reports macro as highest in all four structure sets for both BERT-base and BERT-large. Consequently, the bottom-up conclusion does not follow from the stated macro/micro definitions; it requires an unstated macro edge set or a different computation.

full rationale

The paper does not rely on self-citations, and the expected-layer values are obtained from independently trained structural probes rather than fitted to the target conclusion. However, the central macro-versus-micro comparison is definitionally degenerate. Macro is defined as 'the root verb with its direct dependents,' while the micro structures are exactly the dependency relations emanating from the root (nsubj, dobj, prep, attr). For each structure set, the macro edge set is therefore the union of the micro edge sets. Given the UUAS definition as an edge-count proportion and the linear form of Eq. (4), E[macro] is a weighted average of E for the constituent micro relations, so macro strictly exceeding all micro relations is impossible under the paper's own definitions. Figure 4's macro-highest result thus cannot be evidence for the stated bottom-up hypothesis; it indicates an unstated edge-set definition or an inconsistent computation. The subject-verb agreement analysis is post-hoc and interpretive rather than circular, but it inherits the same macro-definition problem. Overall, this is a partial definitional circularity in the central claim, not a self-citation-based circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central result rests on trained probe and mixing parameters, plus a set of interpretive assumptions about what expected layers mean. No fundamentally new entities are introduced; the micro/macro distinction is a re-labeling of dependency edge subsets.

free parameters (2)
  • Per-layer probe matrix B_l = learned, not reported
    For each layer l, a linear projection is trained to map scalar-mixed embeddings to syntactic distance space; UUAS and expected layers depend on it.
  • Scalar mixing weights a_l and scale gamma_l = learned, not reported
    Used in Eq. (3) to form scalar-mixed embeddings up to layer l; trained separately for each l.
assumptions (4)
  • domain assumption spaCy's EN_CORE_WEB_LG dependency parses of Wikitext-103 provide correct ground-truth syntactic trees.
    Section 5.1 uses these parses as reference for computing UUAS and defining micro/macro structures.
  • domain assumption After a linear transformation, squared Euclidean distances between word embeddings approximate dependency tree distances.
    Inherited from Hewitt and Manning (2019), used throughout as the probe objective in Eq. (2).
  • domain assumption The expected layer metric E[l] with S(l) = UUAS(l) identifies the layer at which a syntactic subgraph is constructed.
    Eq. (4) in Sec. 4; this is the core interpretive assumption behind the bottom-up/top-down/parallel classification.
  • ad hoc to paper The split of dependency edges into micro (nsubj, dobj, prep) and macro (root verb with direct dependents) captures the notion of derivation order.
    Defined in Sec. 4 and used to test the three derivation hypotheses; the mapping from this split to derivation strategy is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models." pith.science (2026). https://pith.science/paper/565LTSFI

@misc{pith2026250621861,
  author       = {Pith},
  title        = {Pith review of: Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/565LTSFI}},
  note         = {Machine review of arXiv:2506.21861}
}
read the original abstract

Recent work has demonstrated that neural language models encode syntactic structures in their internal representations, yet the derivations by which these structures are constructed across layers remain poorly understood. In this paper, we propose Derivational Probing to investigate how micro-syntactic structures (e.g., subject noun phrases) and macro-syntactic structures (e.g., the relationship between the root verbs and their direct dependents) are constructed as word embeddings propagate upward across layers. Our experiments on BERT reveal a clear bottom-up derivation: micro-syntactic structures emerge in lower layers and are gradually integrated into a coherent macro-syntactic structure in higher layers. Furthermore, a targeted evaluation on subject-verb number agreement shows that the timing of constructing macro-syntactic structures is critical for downstream performance, suggesting an optimal timing for integrating global syntactic information.

Figures

Figures reproduced from arXiv: 2506.21861 by the authors.

Figure 1
Figure 1. Derivational Probing investigates how syntactic structures are constructed across layers in neural language [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Macro-syntactic structure (Marco) and micro￾syntactic structures (nsubj and dobj). 5 Experimental Setup 5.1 Data We utilize the Wikitext-103 dataset (Merity et al., 2016) as our primary source of natural language, parsing each sentence with spaCy’s dependency parser (EN_CORE_WEB_LG) (Honnibal et al., 2020). To focus on the language model’s ability to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Global UUAS by each layer for each model. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Expected layer for each model across different structure sets. Error bars represent standard deviation [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Expected layers for syntactic structures in successful and failed subject-verb agreement cases. Error bars ertbaseの内部状態可視化 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Derivation process visualizations for BERT-base on subject-verb agreement for a successful case (“The [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: MDS visualizations of syntactic structure evolution in BERT-large for subject-verb agreement for a [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Expected layer for each GPT-2 model across different structure sets. Error bars represent standard [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Global UUAS by each layer for each GPT-2 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [2]

    BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. Tiwal...

    work page 2019

  2. [5]

    A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. Matthew Honnibal, Ines M...

    work page 2019

  3. [7]

    In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, Online

    Do syntactic probes probe syntax? experiments with jabberwocky probing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, Online. As- sociation for Computational Linguistics. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher

    work page 2021

  4. [9]

    Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computa- tional Linguistics. R. C. Prim

    work page 2018

  5. [10]

    In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy

    Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy. As- sociation for Computational Linguistics. Jennifer C. White, Tiago Pimentel, Naomi Saphra, and Ryan Cotterell

    work page 2019

  6. [11]

    In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies , pages 132–138, Online

    A non-linear structural probe. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies , pages 132–138, Online. Association for Computa- tional Linguistics. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, ...

    work page 2021

  7. [2016]

    Preprint, arXiv:1609.07843

    Pointer sentinel mixture mod- els. Preprint, arXiv:1609.07843. Joakim Nivre

    arXiv

  8. [2018]

    In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium

    Targeted syn- tactic evaluation of language models. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. Rowan Hall Maudslay and Ryan Cotterell

    work page 2018

Show all 13 references
  1. [2019]

    In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy

    What does BERT look at? an analysis of BERT‘s attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Com- putational Linguistics. Jacob Devlin, Ming-Wei Chang, Kenton...

    2019

  2. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online

    Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. Haiyan Zhao, Hanjie Chen, Fan Yang, N...

    2020

  3. [2021]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1626–1639, Online and Punta Cana, Dominican Re- public

    Conditional probing: measuring usable information beyond a baseline. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 1626–1639, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. John Hewitt...

    2021

  4. [2022]

    In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 2801–2813, Abu Dhabi, United Arab Emirates

    Probing for incremental parse states in autoregressive language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 2801–2813, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. Katsuhiko Hayashi, Taro Watanabe, ...

    2022

  5. [2024]

    ACM Trans

    Explainability for large language models: A survey. ACM Trans. Intell. Syst. Technol., 15(2). A The Experimental Results for GPT-2 Models Figures 8 and 9 show the experimental results with the same experimental setup as §5, but con- ducted with GPT-2 small3/medium4 (Radford et...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.