{"id":"26d58958-b1c7-46c1-9258-c70c124b011f","arxiv_id":"2412.14670","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In BERT, middle layers around layers 3 and 4 of BERT-base produce the strongest clustering of verb-particle constructions, with phrasal verbs like 'give up' separated more sharply than prepositional verbs like 'agree on'.","lead":"A team analyzed BERT's internal representations of verb-particle phrases such as 'give up' and 'agree on', using 995 sentences from the British National Corpus. They found that the model's middle layers separate these constructions most clearly, and that different verb types behave differently.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No null baseline or surface-context control underlies the GDV analysis; observed clustering may just mirror the lexical identity of the following particle, leaving the middle-layer syntactic claim unsupported.","rationale":"The reader's weakest assumption identified the lack of statistical baselines and the interpretive leap from embedding separation to syntactic structure. We agree with that, but sharpen it: the most specific artifact is the immediate lexical identity of the particle, which is present in every input. Because the verb embedding attends to the next token, the reported clustering may be an automatic consequence of adjoining a different particle word, not a representation of the verb-particle construction as a unified syntactic object. The proposed control directly removes that cue. If the control shows no clustering, the layer-wise patterns in Section 3 lose their claimed linguistic significance; if it shows clustering, the constructional interpretation gains support. We therefore recommend keeping the conditional verdict: acceptance hinges on this control.","tokens_in":17413,"tokens_out":7106,"duration_ms":61121,"concrete_test":"Recompute GDV at the reported peak layers after substituting the particle token with a frequency-matched non-particle word (e.g., 'give banana'), leaving all other tokens fixed; if the control GDV drops toward zero or is not significantly more negative than a permutation baseline, the clustering is a surface-lexical artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on interpreting GDV separability of verb-token embeddings as evidence that BERT 'captures syntactic structures' (Section 3). However, in every input sentence the target verb is immediately followed by the particle (e.g., 'give' before 'up', 'out', 'in'). Because BERT's contextualized embeddings are computed with bidirectional attention, the verb's hidden state at any layer is strongly influenced by the adjacent particle token. The reported GDV profiles (e.g., 'give' reaching -0.305 at layer 3 vs 'agree' -0.256 at layer 4) could therefore be trivially driven by the orthographic identity of the next word rather than by any construction-level syntactic or semantic unit. The paper defines GDV in Section 2.4 but never presents a null distribution, permutation test, or control condition that removes or randomizes the local particle cue. Without such a baseline, one cannot distinguish true constructional separation from surface lexical co-occurrence; the phrase 'significant variability' is descriptive, not statistical. This is load-bearing because the main novelty—layer-specific and category-specific constructional representation—collapses if the observed clustering merely reflects the lexical content of the immediately following token.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes how BERT and CxG-BERT represent English verb-particle constructions (VPCs) by extracting target-verb embeddings from each layer of the two models, projecting them with multidimensional scaling (MDS), and computing the Generalized Discrimination Value (GDV) to quantify cluster separability. The dataset consists of 995 naturally occurring sentences from the British National Corpus, covering 11 VPC types across three verb categories ('agree', 'come', 'give'). The results show that GDV-based clustering becomes strongest in middle layers (layers 3–4 in BERT-base, layers 6–7 in CxG-BERT), with the 'give' and 'come' categories exhibiting stronger within-category separation than 'agree'. The authors interpret this as evidence that BERT's middle layers capture syntactic structures and that the model treats phrasal verbs as lexical units, in line with construction grammar.","tokens_in":17584,"tokens_out":7617,"duration_ms":59531,"significance":"If the central pattern were established with proper controls, the paper would provide a useful layer-resolved characterization of how a transformer represents multi-word constructions, speaking directly to BERTology and to construction-grammar interpretations of neural models. The study is among the first to target verb-particle constructions specifically, and it uses a parameter-free geometric measure (GDV) on embeddings from externally trained, off-the-shelf models, which is a methodological strength. However, the central claim currently rests on a single descriptive statistic whose interpretation is confounded by the immediate lexical context, and the paper provides no uncertainty quantification or null baseline. These issues are fixable but are load-bearing, so the paper is not yet ready for acceptance.","major_comments":[{"comment":"Section 2.4 defines the GDV and Section 3 reports GDV values for the target-verb embeddings, but in every input sentence the target verb is immediately followed by its particle (e.g., 'give up', 'give out', 'give in'; 'agree on', 'agree with'). Since BERT's contextualized embeddings are computed with bidirectional attention, the verb's hidden state at any layer is directly conditioned on the adjacent particle token. The within-category separation of 'give' constructions, for example, could therefore be trivially driven by the orthographic identity of the following particle ('up' vs 'out' vs 'in') rather than by any construction-level syntactic or semantic representation. No null distribution, permutation test, or control condition that removes or randomizes the local particle cue is provided. Without such a baseline, the central claim that BERT's middle layers 'capture syntactic structures' of VPCs is not established.","section":"§2.4, §3"},{"comment":"Section 3 presents GDV values as point estimates (e.g., 'give' reaching -0.305 at layer 3 vs -0.288 at layer 4; 'come' -0.236 at layer 3 vs -0.191 at layer 4), but the dataset has only 93–100 sentences per construction (Section 2.1) and no error bars, confidence intervals, or significance tests are reported. The abstract's phrase 'significant variability' is descriptive only. The authors should provide bootstrap confidence intervals over sentences and a permutation null distribution (e.g., randomly permuting the particle labels) to determine whether the layer- and category-dependent differences exceed chance.","section":"§3"},{"comment":"The interpretation in Section 4 that strong GDV clustering reflects 'lexical unity of phrasal verbs' or 'syntactic structures' is an over-reach relative to the measure. GDV quantifies Euclidean separability of the target-verb embeddings by class label, but separability by the following particle is compatible with the model encoding simple lexical co-occurrence statistics rather than constructional knowledge. A direct syntactic probe (e.g., a classifier trained on the embeddings to predict argument structure, or a controlled test set that manipulates the particle while holding context constant) is needed to support the constructional interpretation. At minimum, the claims should be scaled back to what the measure shows: that the verb embeddings are separable by construction type.","section":"§4"},{"comment":"Figure 3 and the text in Section 3 report that the GDV computed over all input data is already -0.358 at layer 1 in the original BERT, which is more negative than the strongest within-category values. This suggests that global separation into the three verb groups is present from the earliest layer and may dominate the metric, yet the paper does not analyze how the overall clustering relates to the category-specific GDV curves. For example, the weaker within-category clustering of 'agree' may simply reflect that the verb 'agree' is always the same token with less distinctive following prepositions. The authors should report and discuss the relationship between the overall and within-category GDV values, for instance by conditioning on verb identity.","section":"§3"}],"minor_comments":[{"comment":"Figures 1–3 are referenced but not embedded in the manuscript text, so the MDS projections cannot be inspected; the captions alone are insufficient for verification.","section":"Figures 1–3"},{"comment":"The manuscript contains numerous typos and grammatical errors (e.g., 'showing week clustering', 'the string clustering recorded', 'its meaning is emerged from its form'); a thorough proofreading pass is needed.","section":"Throughout"},{"comment":"CxG-BERT is described only as having 'extra-training using sentences instantiating constructions'; the training details (fine-tuning vs continued pre-training, data size, hyperparameters) should be specified.","section":"§2.2"},{"comment":"The paper does not state whether the analysis code or dataset are publicly available; providing them would strengthen the empirical claims.","section":"General"},{"comment":"Table numbering is inconsistent (Table 2 appears before Table 1, and the data-cleaning table is numbered Table 12); renumber the tables sequentially.","section":"Tables"},{"comment":"The reference to Mickus et al. appears as 2019 in the text but 2020 in the bibliography; please ensure all in-text citations match the reference list.","section":"References"},{"comment":"Section 2.2 notes that the Wikipedia portion of BERT's and CxG-BERT's training data 'might be not standard from a pure linguist view'; the paper should discuss the implications of this limitation for the linguistic generalizability of the findings.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript leans heavily on the authors' own GDV/MDS methodology from previous work; while this is not a circularity problem in itself, the absence of any independent baseline or comparative method makes the empirical contribution hard to evaluate. The paper would also be strengthened by a clearer statement of how it advances beyond the authors' prior applications of GDV to word classes. I am willing to review a revised version that addresses the control and statistical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. The paper tests whether BERT's hidden layers separate verb-particle constructions. The new bit is applying the authors' GDV separability measure layer-by-layer to three verb categories (agree, come, give) in both vanilla BERT and CxG-BERT, using BNC sentences. The finding that middle layers cluster most strongly is consistent with prior work (Jawahar, Hewitt-Manning), so the novelty is the construction-specific angle, not the middle-layer claim.\n\nWhat's good: The dataset is described, the method is parameter-free and reproducible in principle, and the authors are honest when results don't support their theoretical expectation (e.g., the 'agree' category). The MDS figures are legible.\n\nThe soft spot is load-bearing: there is no control for the particle token itself. In every sentence, the target verb is immediately followed by the particle (give up, come back, agree on). With bidirectional attention, the verb's embedding is strongly influenced by that adjacent token. The reported GDV differences between verb categories could simply reflect the orthographic/lexical identity of the next word, not any construction-level syntactic encoding. The paper defines GDV but gives no null distribution, permutation test, or baseline that randomizes or removes the particle. Without that, the phrase 'capturing syntactic structures' goes beyond what cluster separability can show. This isn't a minor quibble; it's the main interpretive step.\n\nThere are also smaller issues: no error bars, no released code/data, and the abstract's 'significant variability' is descriptive, not statistical. The self-citation of GDV is fine, but the method needs a proper validation in this context.\n\nWho is this for? People working on BERT interpretability or construction grammar might get a useful starting hypothesis. But as it stands, I'd treat the result as a prompt for a controlled study, not a demonstrated fact.\n\nRecommendation: Send it to peer review, but with the clear expectation that the authors add a surface-cue control and a null baseline. With that revision, the paper could be a solid small contribution.","headline":"A modest layer-wise BERT probe of verb-particle constructions whose main claim is undercut by the absence of a surface-cue control.","tokens_in":18145,"tokens_out":1670,"would_cite":false,"duration_ms":14040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Middle BERT layers best cluster verb-particle constructions, with category-specific peaks","keywords":["verb-particle constructions","BERT","construction grammar","generalized discrimination value","multi-dimensional scaling","hidden states","layer-wise analysis","phrasal verbs"],"falsifier":"Replace the particle token in each test sentence with a randomly chosen word from the same corpus and recompute the GDV on the verb token's hidden states; if strong negative GDV persists, the separation does not depend on the construction. Alternatively, permute the construction labels among the embedded verbs and measure how often the observed GDV exceeds what is obtained under random label assignment; if it is not an extreme outlier, the clustering is not statistically meaningful.","tokens_in":17177,"feed_emoji":"🧠","tokens_out":11779,"duration_ms":74511,"temperature":0.7,"pith_summary":"The paper asks whether a transformer language model such as BERT internally splits verb-particle constructions according to their grammatical type, and where in the network that split happens. It reports that the hidden states of the verb token form distinct clusters that are strongest in the middle layers (3–4 in BERT-base, 6–7 in construction-trained CxG-BERT), and that the separation strength varies by verb category. If correct, this means BERT does not encode linguistic constructions uniformly across its layers, and that construction grammar's distinction between lexical phrasal verbs and compositional verb-preposition combinations has a real correlate in the geometry of embeddings. The finding also suggests a practical guide for where to probe or intervene in such models.","feed_headline":"Middle BERT layers best cluster verb-particle constructions","feed_subtitle":"Clustering of verb-token embeddings peaks at layers 3–4 in BERT and 6–7 in CxG-BERT, varying by verb category.","key_machinery":"The load-bearing tool is the generalized discrimination value (GDV), a scalar that measures how separated pre-defined classes are in a high-dimensional space. For each layer, the authors take the 768-dimensional hidden state of the verb token, z-score each dimension, and compute the mean intra-class distance minus the mean inter-class distance, normalized by dimensionality; negative values indicate clustering, and -1 is already very strong separation. They pair this with multi-dimensional scaling (MDS) to project the same embeddings into two dimensions for visual inspection. The GDV is what turns the raw embeddings into a quantitative claim about layer-specific capture of structure, and its sign convention (negative means separated) is what lets the paper read the middle-layer trough as a peak of representational strength.","core_discovery":"On the paper's own terms, the central discovery is that BERT's internal representations of verb-particle constructions are organized by construction type, and that this organization peaks in the middle layers. Using the generalized discrimination value (GDV) on the hidden-state vectors of the verb token, the authors find that all three verb groups (agree, come, give) show increasingly negative GDV from early to middle layers, with give reaching the strongest separation (-0.305 at layer 3 in BERT-base) and the other two groups peaking at slightly weaker values around -0.26 and -0.24. The construction-trained CxG-BERT shows the same trend with a later peak (layers 6–7). The authors interpret the strong clustering of give as evidence that idiomatic phrasal verbs are represented as unified lexical units; the weak clustering of come aligns with its compositional, non-idiomatic character; and the unexpected clustering of agree suggests that even prepositional verbs receive construction-specific representations. Thus the paper claims that the model captures constructional knowledge in a layer-dependent, category-dependent way.","pith_inferences":["A natural extension would be to test whether the verb-token clustering survives when the particle token is masked or replaced; if it does, the separation reflects the verb's wider sentential context rather than the specific construction, which would alter the construction-grammar interpretation.","Computing GDV on verb-plus-particle token pairs, rather than the verb alone, would directly test whether the model treats the combination as a single lexical unit and could reveal whether the isolated verb clusters are driven by the particle's surface presence.","Permutation-based significance testing on the GDV values would provide a null distribution for no constructional structure, allowing direct statistical comparison of the layer peaks and a common scale for future model comparisons."],"forward_implications":["If the layer dependence is correct, the middle layers (3–4 for BERT-base, 6–7 for CxG-BERT) are the most promising places to extract construction-level features for downstream tasks.","The category-specific differences in GDV imply that conclusions about a model's syntactic knowledge should be stated per construction type, not globally.","The weakening of clustering in late layers, with more outliers, suggests that final-layer representations are the least reliable for reading constructional information.","The shift of the peak toward layers 6–7 in CxG-BERT indicates that additional training on construction examples can relocate where constructional knowledge is encoded."],"supporting_citations":[{"why":"This reference provides the BERT model whose layered hidden states are analyzed throughout the study.","marker":"Devlin et al. (2019)"},{"why":"This reference supplies CxG-BERT, the construction-trained transformer used as the second model.","marker":"Madabushi et al. (2020)"},{"why":"This reference introduces the generalized discrimination value, the quantitative measure of cluster separation on which all layer comparisons rest.","marker":"Schilling et al. (2021b)"},{"why":"This reference showed that BERT's middle layers encode syntactic structure, the prior result this paper's layer profile extends.","marker":"Jawahar et al. (2019)"},{"why":"This reference provides the valency-based distinction between prepositional verbs and phrasal verbs that defines the three construction categories.","marker":"Herbst and Schüller (2008)"},{"why":"This reference offers neurolinguistic evidence that phrasal verbs behave as single lexical units, which motivates the expected clustering pattern.","marker":"Cappelle et al. (2010)"},{"why":"This reference found syntactic tree-structure information in BERT's middle layers, supporting the interpretation of middle-layer clustering as syntactic.","marker":"Hewitt and Manning (2019)"}],"fun_headline_variants":["BERT's middle layers sharpen verb-particle clustering","Idiomatic phrasal verbs cluster best in BERT's middle","Verb-particle clarity peaks mid-network in BERT","Where does BERT store 'give up'? Middle layers","Not all verbs alike: BERT's layered verb-particle maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument treats a negative GDV for verb-token embeddings as direct evidence that the model captures syntactic or constructional structure, yet no null distribution or permutation test is supplied; the observed clustering could in principle be produced by surface cues such as the immediately following particle, without the model having any abstract knowledge of constructions.","fun_headline_variants_meta":{"raw":{"variants":["BERT's middle layers sharpen verb-particle clustering","Idiomatic phrasal verbs cluster best in BERT's middle","Verb-particle clarity peaks mid-network in BERT","Where does BERT store 'give up'? Middle layers","Not all verbs alike: BERT's layered verb-particle maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1388,"prompt_tokens":974,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":333}},"tokens_in":590,"tokens_out":414,"duration_ms":8998,"temperature":1.0,"reasoning_tokens":333,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:01:02.694111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the particle token in each test sentence with a randomly chosen word from the same corpus and recompute the GDV on the verb token's hidden states; if strong negative GDV persists, the separation does not depend on the construction. Alternatively, permute the construction labels among the embedded verbs and measure how often the observed GDV exceeds what is obtained under random label assignment; if it is not an extreme outlier, the clustering is not statistically meaningful.","supporting_citations":[],"review_version":1}