{"id":"70e9d7e2-9ffb-45c4-9c92-48e2a61bde41","arxiv_id":"2506.21861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new probing method shows BERT constructs micro-syntactic dependencies in lower layers and integrates them into macro-syntactic structures in higher layers.","lead":"This paper introduces Derivational Probing, a way to track how small syntactic pieces (like subject phrases) and the overall sentence structure (the verb with its dependents) appear across the layers of BERT. It finds that BERT builds small pieces first and combines them later, and that getting the timing of the big structure right affects subject-verb agreement accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported macro > micro expected-layer ordering is mathematically impossible if macro edges are the union of the named micro edges; the edge-set definition must be settled before the bottom-up claim can be assessed.","rationale":"The reader’s weakest assumption concerns whether expected-layer differences reflect construction order rather than edge difficulty or probe dynamics. I agree that this is a serious interpretational threat, but I found a more immediate, load-bearing problem: under the paper’s stated definitions, the macro and micro edge sets overlap to the point where the reported macro > micro ordering is arithmetically impossible. This is not a matter of philosophical interpretation; it is a concrete inconsistency between the formal definition of UUAS in §4, the expected-layer formula in Eq. (4), and the empirical pattern in Figure 4. If the macro edge set is actually the full parse tree, or includes relations beyond the named micro relations, then the abstract’s characterization of macro as “the relationship between the root verbs and their direct dependents” is inaccurate, and the comparison is confounded by edge-set composition and edge difficulty. If the macro edge set is indeed the union of the micro relations, Figure 4 cannot be correct as reported. Either way, the central bottom-up claim is presently not well supported by the stated methodology. The paper has useful strengths: the method is clearly specified, code is released, hyperparameters are reported, and the GPT-2 appendix provides a meaningful contrast. A focused reanalysis that precisely defines and enumerates the macro and micro edge sets, and ideally controls for dependency distance and edge difficulty, could resolve this concern. Until then, I would not accept the paper’s strong bottom-up conclusion; a conditional verdict requiring this clarification and reanalysis is appropriate.","tokens_in":9318,"tokens_out":9437,"duration_ms":109752,"concrete_test":"Inspect the released code for one structure set (e.g., Marco+nsubj+dobj) and enumerate the exact macro and micro edge sets per sentence. Recompute S_macro(ℓ) from the per-relation UUAS curves as a weighted average of S_nsubj(ℓ) and S_dobj(ℓ), then compute E[macro] via Eq. (4). If E[macro] still exceeds both component expected layers, enumerate which additional relations (det, amod, aux, etc.) are included in the macro set; if the sets are identical, verify that the reported Figure 4 ordering cannot arise from averaged UUAS curves. Report whether the bottom-up ordering survives when macro is restricted to exactly the root’s direct-dependent edges.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In §4, macro-syntactic structure is defined as “the root verb with its direct dependents,” while micro-syntactic structures are the relations nsubj, dobj, prep, and attr. In the four structure sets of Table 1, these appear to be the same edges: e.g., in “The concert caused a major stir,” the macro edges are root→concert (nsubj) and root→stir (dobj), and the micro edges are exactly those two relations. If macro is the union of the micro edge sets, then S_macro(ℓ) is a weighted average of S_nsubj(ℓ) and S_dobj(ℓ) by the UUAS definition in §4. Inserting this into Eq. (4), E[macro] is a weighted average of E[nsubj] and E[dobj], so it cannot be strictly larger than both. Yet Figure 4 shows macro with the highest expected layer in all four sets for both BERT-base and BERT-large. This is an internal inconsistency, not merely an interpretational concern. Either the macro edge set in the code includes additional relations (e.g., det, amod, aux) not described in the text, or macro UUAS is computed differently from the stated definition, or the paper’s “root + direct dependents” characterization is inaccurate. In any case, Figure 4 does not test the stated macro/micro distinction, and the central bottom-up conclusion rests on an unresolved definitional mismatch.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Derivational Probing, which combines Hewitt and Manning's structural probe with Tenney et al.'s expected-layer metric to estimate the layer at which individual dependency subgraphs become linearly decodable in BERT. The authors define micro-syntactic structures (nsubj, dobj, prep, attr) and macro-syntactic structures (the root verb with its direct dependents), and report that macro structures have consistently higher expected layers across four sentence types in both BERT-base and BERT-large, which they interpret as bottom-up derivation. A follow-up analysis on masked-LM subject-verb agreement with 1,000 positive and 1,000 negative items suggests an 'optimal timing' for macro-structure construction. Appendix results for GPT-2 show a more parallel derivation pattern.","tokens_in":9683,"tokens_out":9335,"duration_ms":97635,"significance":"If the reported ordering were robust, the paper would offer a useful methodological refinement of structural probing and a concrete, falsifiable claim about layer-wise syntactic construction in transformer language models. The study has strengths: four structure sets, two BERT sizes, five seeds with reported error bars, a released code repository, and an extension to GPT-2 in the appendix. However, the central comparison is currently undermined by an apparent mismatch between the stated edge-set definitions and the computed expected layers; until that mismatch is resolved, the bottom-up conclusion cannot be evaluated. The secondary 'optimal timing' claim rests on very few failure cases without statistical support, and the interpretation of expected-layer differences as construction order is an operational assumption that should be justified or softened.","major_comments":[{"comment":"The manuscript must state exactly which edges enter the macro and micro UUAS computations. As written, 'macro-syntactic structures (the root verb with its direct dependents)' and micro relations 'nsubj, dobj, prep, attr' (§4) together with the grouping in §5.1 ('dependency relations emanating from the root verb') imply that the macro edge set is the union of the micro edge sets for each structure set in Table 1. In that case UUAS_macro(ℓ) is a weighted average of UUAS_micro(ℓ) at every layer, and under the monotone UUAS curves shown in Figure 3 the expected layer in Eq. (4) is a convex combination of the micro expected layers; it cannot exceed all of them. Figure 4 nevertheless shows macro with the highest expected layer in all four sets for both BERT sizes. This is an internal inconsistency, not merely an interpretation issue: either the code uses additional macro edges (e.g., det, amod, aux), or micro structures denote phrase-internal edges rather than the root-dependent relations named in the text. Please specify the edge sets precisely and, if macro is a strict superset of the named micro relations, report the composition. If macro is indeed the union, the main bottom-up claim is unsupported by Figure 4.","section":"§4, §5.1, Fig. 4"},{"comment":"The 'optimal timing' claim is based on 16 failed cases for BERT-base and 17 for BERT-large out of 1,000 items, with no significance tests on the success/failure expected-layer differences and with representative MDS plots from a single example per condition. The text describes BERT-base failures as premature macro integration and BERT-large failures as delayed integration (Figures 5–7), but the error bars in Figure 5 and the absence of any test make it impossible to distinguish a systematic timing effect from noise, and 'frequently' overstates a 1.6% failure rate. Please report per-condition expected-layer means and standard errors, a permutation test or bootstrap interval for the success–failure difference, and the number of cases on which the qualitative MDS characterization is based.","section":"§7.2"},{"comment":"The paper interprets S(ℓ) improvements as the layer at which a structure is 'constructed,' but the metric measures when a trained linear probe can recover the structure from scalar-mixed embeddings. These could diverge: an edge that is present earlier but linearly hard, or an artifact of per-layer scalar mixing weights, would shift E[ℓ] without changing construction order. The abstract's language ('emerge,' 'gradually integrated') is stronger than this operational definition licenses. Please either temper the causal framing or add a control (e.g., per-layer probes trained without scalar mixing, a non-linear probe baseline, or an analysis of probe convergence) to justify the construction-order reading.","section":"§4, Eq. (4)"}],"minor_comments":[{"comment":"Typos and inconsistent notation: 'Marco' for 'Macro' in Figure 2, Table 1, and §5.1; 'd-dimentional' in §3.1; 'scaler-mixed' throughout; 'strucv-tures' in §4; 'Bertbase' in Figure 5. Please proofread and standardize the terminology.","section":"Throughout"},{"comment":"The captions of Figures 6 and 7 do not explain what the MDS axes are, how the 'Layer0/Layer5/Layer12' labels are chosen, or how the red highlighting is derived; add a legend and a description of the plotting pipeline.","section":"Figures 6–7"},{"comment":"The term 'micro-syntactic structures' is used both for dependency relations (nsubj) and for phrases ('subject noun phrases') in §4; this duality should be reconciled because it directly affects how the edge sets are interpreted.","section":"§4, §5.1"},{"comment":"The expected-layer formula sums from ℓ=1 to L but the numerator includes S(0); please state explicitly how S(0) is defined (e.g., the embedding layer) and whether S(0) is included in the denominator as well.","section":"Eq. (4)"},{"comment":"The GPT-2 results are presented without the subject-verb agreement analysis; if included, they would strengthen the claim that GPT-2 exhibits parallel derivation rather than bottom-up derivation.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is readable and the experimental design is mostly transparent, but the definitional mismatch between the macro and micro edge sets is serious; it could be a typo or a genuine flaw. If the authors' code shows that macro includes additional edges beyond the named micro relations, the paper may be salvageable with a major revision. The lack of statistical testing in §7.2 should be addressed regardless. Given the current wording, the central Figure 4 result cannot be assessed until the edge-set definitions are clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central claim—BERT builds micro-syntactic structures before macro ones—is almost certainly an artifact of how the macro edge set is defined. Under the definitions in §4, macro is the union of the micro edge sets (e.g., root→nsubj and root→dobj). That makes the macro UUAS a weighted average of the micro UUAS, so the macro expected layer cannot strictly exceed all micro expected layers. Yet Figure 4 shows exactly that in every condition for both BERT sizes. Either the macro edge set includes extra relations not described in the text, or the UUAS is computed on a different parse tree than the micro sets. As written, the result is internally inconsistent, and the bottom-up conclusion is unsupported.\n\nThere is real value here. The idea of tracking subgraph-level UUAS across layers with the expected layer metric is a natural but new extension of Hewitt & Manning and Tenney et al. The method is clearly described, the code is public, and the consistency across four structure sets and two BERT models is suggestive that something systematic is happening. The GPT-2 appendix showing parallel derivation is interesting if it survives scrutiny.\n\nBeyond the definitional problem, there are other soft spots. No significance tests are reported, so we don't know if the macro/micro differences are even statistically reliable. The subject-verb agreement analysis in §7.2 rests on only 16 and 17 error cases, far too few to support the 'optimal timing' claim. There is also no control for edge difficulty—some dependency relations may simply be easier to probe earlier, which would produce the same layer ordering without implying construction order. The limitations section mentions semantic confounds and model coverage but does not address the edge-set issue.\n\nThe paper is not worthless. The method is reproducible, and if the authors resolve the macro/micro definition—either by clarifying that macro includes additional edges or by proving the ordering can hold—the empirical phenomenon may still be real. But as it stands, the main plot line does not survive contact with the equations. I'd send it to a careful reviewer, but I wouldn't trust the headline result until the edge-set definition is settled.","headline":"The bottom-up derivation claim is likely an artifact of the macro/micro edge-set definition; the paper needs to settle that before the main result can be trusted.","tokens_in":10123,"tokens_out":6094,"would_cite":false,"duration_ms":60342,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BERT assembles dependency trees bottom-up: micro-syntactic structures appear in lower layers, and the root-verb macro-structure is integrated later.","keywords":["derivational probing","structural probing","expected layer metric","syntactic structure derivation","BERT interpretability","subject-verb agreement","bottom-up derivation"],"falsifier":"A concrete control: train the same per-layer structural probes on embeddings whose layer order has been shuffled, and on Jabberwocky sentences that strip semantic cues. If macro structures still show systematically higher expected layers than micro structures under both controls, the bottom-up reading survives; if the ranking disappears or flips, the result is an artifact of edge difficulty, probe training dynamics, or semantic confounds rather than derivation order.","tokens_in":9167,"feed_emoji":"🧩","tokens_out":7434,"duration_ms":76870,"temperature":0.7,"pith_summary":"The paper introduces Derivational Probing, which combines structural probing with the expected-layer metric to watch individual pieces of a dependency tree emerge across a language model's layers. Applied to BERT-base and BERT-large on filtered single-clause sentences from Wikitext-103, it finds that micro-syntactic structures—subject, object, and prepositional phrases—reach their expected layers before the macro-syntactic structure linking the root verb to its direct dependents. The authors interpret this as a bottom-up derivation: local dependencies are assembled first, then integrated into a global tree. A targeted subject-verb agreement test with intervening nouns suggests that integrating the macro-structure too early or too late hurts accuracy, implying an optimal integration window. If right, the paper adds a derivation timeline to the static picture of syntax in neural networks and offers a method for comparing derivation strategies across models and languages.","feed_headline":"BERT builds syntax bottom-up: small phrases first, full tree later","feed_subtitle":"Layer-by-layer probing shows subject and object phrases emerge before the root verb structure; timing predicts agreement errors.","key_machinery":"The machinery is the fusion of two existing instruments. A structural probe learns a linear projection $B_\\ell$ such that Euclidean distances between transformed embeddings approximate dependency-tree distances; per-layer Unlabeled Undirected Attachment Score (UUAS) on extracted subgraphs measures how recoverable each structure is. The expected-layer metric $E[\\ell] = \\sum_\\ell \\ell\\,(S(\\ell)-S(\\ell-1)) / \\sum_\\ell (S(\\ell)-S(\\ell-1))$ turns the per-layer UUAS curve into a single layer index for each subgraph, using scalar-mixed embeddings (weighted averages of layers $0$ through $\\ell$). Micro and macro structures are defined as edge sets in the parse tree: local relations such as nsubj, dobj, prep, and attr versus the root verb and its direct dependents. Comparing expected layers across these edge sets is what carries the bottom-up conclusion.","core_discovery":"The central claim is that BERT constructs dependency trees bottom-up across layers. In all four sentence-structure sets studied, and for both BERT-base and BERT-large, the macro-syntactic structure (root verb with its direct dependents) has a consistently higher expected layer than the micro-syntactic structures (nsubj, dobj, prep, attr), meaning the local phrases become recoverable earlier. The agreement analysis adds that the timing of macro-structure formation is behaviorally consequential: BERT-base failures coincide with premature subject–verb linkage before the subject phrase's internal dependencies are stable, whereas BERT-large failures coincide with delayed linkage, and successful cases link subject and verb at an intermediate layer. The authors also report that GPT-2 small and medium show nearly parallel expected layers for micro and macro structures, which they take as evidence that bottom-up derivation is not a universal property.","pith_inferences":["The BERT versus GPT-2 contrast suggests that bidirectional context may let local phrases stabilize before global attachment, while left-to-right models have less freedom to defer global structure; the paper does not claim this.","A natural next step the paper does not run is to shift macro-structure integration by weighting or fine-tuning specific layers and check whether agreement accuracy follows the U-shaped curve implied by the failure analysis.","Because the pipeline is parser- and language-agnostic, it could be applied to multilingual models and, if a distance-based probe for nonterminal structures existed, to constituency grammars; the paper identifies the latter as an open limitation."],"forward_implications":["For BERT-style encoders, syntax is not present all at once; interpretability methods that treat a single layer as 'the' syntactic representation will miss the derivation timeline.","The gap between macro and micro expected layers gives a quantitative signature for distinguishing bottom-up, top-down, and parallel derivations in any transformer.","Subject-verb agreement accuracy tracks the timing of macro-structure formation: forming it too early or too late both hurt, so layer-level interventions that shift macro integration may improve accuracy.","The method carries over to autoregressive models, where GPT-2 shows nearly parallel expected layers, indicating that derivation strategy is a measured model property rather than an assumption."],"supporting_citations":[{"why":"Supplies the structural probe: a linear projection whose distances approximate dependency-tree distances, the foundation of the per-layer UUAS scoring used here.","marker":"Hewitt and Manning (2019)"},{"why":"Supplies the expected-layer metric and the scalar-mixed embedding construction that turns per-layer UUAS curves into a single layer index.","marker":"Tenney et al. (2019)"},{"why":"Provides the arc-standard transition-based parsing order whose bottom-up construction order motivates the bottom-up derivation hypothesis.","marker":"Nivre (2004)"},{"why":"Provides the head-driven, top-down parsing order that defines the competing top-down derivation hypothesis.","marker":"Hayashi et al. (2012)"},{"why":"Supplies the targeted syntactic evaluation templates (subject-verb agreement with intervening attractor nouns) used in the detailed analysis.","marker":"Marvin and Linzen (2018)"},{"why":"Supplies the masked-token pseudo-whole-sentence probability scoring used to judge whether grammatical sentences are preferred.","marker":"Salazar et al. (2020)"},{"why":"Supplies the BERT-base and BERT-large models that are the objects of the empirical study.","marker":"Devlin et al. (2019)"}],"fun_headline_variants":["BERT builds syntax from small phrases to full tree across layers","Micro syntax first, macro later: BERT's bottom-up derivation","Probing BERT layers: local structures emerge before global ones","Syntax timing in BERT predicts subject-verb agreement errors","BERT constructs dependency trees bottom-up, layer by layer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the layer where a structure's parse-recovery score jumps is the layer where that structure is actually built, rather than the layer where its edges happen to become easy to recover; if edge difficulty, training dynamics, or meaning-based cues explain the timing, the bottom-up conclusion would not follow.","fun_headline_variants_meta":{"raw":{"variants":["BERT builds syntax from small phrases to full tree across layers","Micro syntax first, macro later: BERT's bottom-up derivation","Probing BERT layers: local structures emerge before global ones","Syntax timing in BERT predicts subject-verb agreement errors","BERT constructs dependency trees bottom-up, layer by layer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4375,"prompt_tokens":864,"completion_tokens":3511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":3428}},"tokens_in":480,"tokens_out":3511,"duration_ms":28765,"temperature":1.0,"reasoning_tokens":3428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:16:24.783325+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete control: train the same per-layer structural probes on embeddings whose layer order has been shuffled, and on Jabberwocky sentences that strip semantic cues. If macro structures still show systematically higher expected layers than micro structures under both controls, the bottom-up reading survives; if the ranking disappears or flips, the result is an artifact of edge difficulty, probe training dynamics, or semantic confounds rather than derivation order.","supporting_citations":[],"review_version":1}