{"id":"23bbe91a-1cec-4b1f-af59-0c30cf417f7a","arxiv_id":"2412.05571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A trained linear probe reveals that LLM activations encode syntactic relation types and directions as directions (angles) and relation existence as distances in a low-dimensional subspace.","lead":"This paper introduces a Polar Probe that reads syntactic relations from both distance and direction in LLM activations. It shows that dependency types and directions are linearly decodable as angles, and that this polar geometry appears in several language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing label-permutation control: because the Polar Probe is supervised with UD labels and the angular structure is imposed by Eq. 4, high test scores do not yet show that the polar geometry is intrinsic to LLM activations rather than a generic linear fit to arbitrary labels.","rationale":"I read the paper as making a representational claim: LLM activations contain a low-dimensional subspace in which syntactic trees are encoded as a polar coordinate system, with distance coding existence and angle coding type/direction. The evidence for linear decodability is solid: held-out generalization is reported, the random-LLM baseline is near chance, and the layerwise peak at layer 16 is consistent with prior structural-probe work. These are real and should be credited. The load-bearing gap is that the polar geometry is explicitly optimized by the supervised loss, so observing it in the probe's output does not by itself show that the geometry is spontaneously present in the LLM. The missing experiment is a label-permutation or shuffle control on the same LLM activations. This is a sharper version of the reader's annotation concern: it is not only that the chosen UD annotation might be wrong, but that no test distinguishes syntax-specific geometry from a geometry that the probe can impose on any sufficiently rich representation. The paper's raw-activation baseline already achieves 80% AUC for type labels, which suggests rich generic information is available; a permutation control is needed to show that the specific UD label structure, rather than generic lexical/POS/positional structure, is what the probe exploits. I do not think this requires rejection. The empirical phenomenon is real and the methodology is standard; the fix is an additional control and a softened claim. The reader's CONDITIONAL verdict already captures the need for revision, and this control should be one of the explicit conditions. Therefore the verdict should remain UNCHANGED, but with the permutation control added to the requested revisions.","tokens_in":11985,"tokens_out":6987,"duration_ms":74091,"concrete_test":"Run a permutation control on Llama-2-7b layer 16 under the paper's exact protocol: keep the same sentences, edge set, number of classes, and training/validation/test split, but randomly permute the dependency-type labels (and, separately, randomly flip head/dependent direction labels) in the training set, then evaluate on the true test labels using the same metrics. Repeat over 10 random permutations. If permuted-label performance remains comparable to true-label performance (within about 5 points in LAS or AUC), the polar geometry is not specific to syntactic structure; if it drops to near chance while true-label performance is high, the UD labels are privileged and the spontaneous-geometry claim is supported. As a secondary check, apply the same permutation to the structural objective's tree-distance labels to test the distance component.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim, that LLMs 'spontaneously learn a geometry of neural activations that explicitly represents the main symbolic structures of linguistic theory', requires showing that the UD-labeled syntactic structure is privileged by the activations. But the Polar Probe is trained exactly on those labels through the angular objective (Eq. 4) and the combined loss (Eq. 11). The probe therefore imposes same-type collinearity and cross-type orthogonality; measuring that structure in the probe's output space is partly a check of the training objective. The controls reported ('No Probe' and 'Polar Probe on a Random LLM') do not settle whether the true UD labels are special. A Random LLM may fail because its activations carry less information, while a real LLM could still permit the probe to fit arbitrary label assignments using generic lexical, part-of-speech, or positional regularities. The raw-activation baseline's 80% AUC for type labels (Section 3, 'Comparison with baselines') shows that such generic regularities are present. Without a control in which the same LLM activations are trained with randomly permuted relation labels (or permuted head/direction assignments), the observed success could reflect probe capacity plus generic information, rather than a spontaneous syntax-specific polar geometry. This concern is load-bearing because the 'spontaneous' and 'explicitly represents' components of the central claim go beyond mere linear decodability, which would be established by any supervised probe fit. It also subsumes the reader's annotation concern: if the annotation scheme were wrong or arbitrary, the same missing control would mask the problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the Polar Probe, a linear transformation trained on LLM contextual embeddings with a joint objective that combines the Structural Probe distance loss (Eq. 3) and an angular contrastive loss (Eq. 4). The resulting probe represents each dependency edge as a vector B_P(h_i - h_j); the squared norm of this vector is claimed to encode the existence of a syntactic relation (Eq. 8), the absolute cosine to type-specific prototypes is claimed to encode the relation type (Eq. 9), and the sign of that cosine is claimed to encode head direction (Eq. 10). The authors evaluate the probe on the English Web Treebank across several LLMs (Llama-2, Mistral, BERT-large, GPT-2, Pythia), report that it outperforms the Structural Probe on labeled and directed attachment, analyze layer- and dimensionality-dependent performance, and present a small controlled nested-sentence dataset. They conclude that LLMs spontaneously learn a polar-coordinate geometry that explicitly represents labeled directed dependency trees.","tokens_in":12303,"tokens_out":7506,"duration_ms":75320,"significance":"If the geometry is genuinely intrinsic to LLM activations, the paper would provide a simple linear readout for labeled directed dependency trees, contributing to the long-standing debate about symbolic versus connectionist representations. The manuscript has several concrete strengths: evaluation on a held-out test set, a range of model families and sizes, the Random-LLM and No-Probe controls, and a dimensionality analysis that goes beyond the original Structural Probe. However, the load-bearing interpretation that the polar geometry is 'spontaneous' and 'explicitly represents' UD syntax is not yet fully supported: the probe is trained with UD labels, and the angular objective imposes the collinearity and orthogonality structure that is later measured. The quantitative comparison with the Structural Probe is not like-for-like, and the headline numerical claims are reported without confidence intervals or significance tests. With additional controls and clarifications, the central results could be made solid, but as presented the evidence is incomplete.","major_comments":[{"comment":"The 'spontaneous' and 'explicitly represents' claims require a label-permutation control. The angular objective (Eq. 4) explicitly pushes same-type edges toward collinearity and different-type edges toward orthogonality, so measuring that structure in the probe output partly checks the training objective. The Random-LLM and No-Probe baselines do not establish that the true UD labels are privileged: a real LLM's activations contain abundant lexical, part-of-speech, and positional regularities (as the No-Probe 80% AUC shows), and a supervised linear probe could plausibly fit permuted labels to a similar degree. I request a control in which the same LLM activations are trained with randomly permuted relation labels (or permuted head/direction assignments), preserving label frequencies and edge distributions; without it, the evidence supports linear decodability of UD labels, not a spontaneous syntax-specific polar geometry.","section":"§2.4, Eq. (11); §3 'Comparison with baselines'; §4 'Limitations'"},{"comment":"The comparison with the Structural Probe on label accuracy is apples-to-oranges. The Structural Probe is trained only to minimize the distance objective (Eq. 3) and is not designed to encode relation type or direction, so lower label accuracy is expected and does not demonstrate that the Polar Probe's geometry is superior. To support the claimed 'nearly two folds' improvement, the authors should compare against a supervised linear readout trained on top of the Structural Probe's output embeddings, or against a linear classifier applied to raw activations; otherwise the baseline is not a fair control for label and direction accuracy.","section":"§3 'Comparison with baselines'; Fig. 2.B; Fig. 3"},{"comment":"The paper reports no confidence intervals, error bars, or significance tests for the headline AUC, balanced accuracy, or LAS scores. Given that the central quantitative claims include 'substantially outperforms' and 'becomes increasingly precise in the latest frontier models,' the authors should report variance across test splits, random seeds, or bootstrap resamples, and where appropriate perform significance tests for the differences between probes.","section":"§3, Fig. 2.B, Fig. 3, Fig. 6"},{"comment":"The definition of the edge embedding s_i,j = h_i - h_j does not specify whether the ordered pair (w_i, w_j) is always (head, dependent), always (dependent, head), or in sentence order. This matters because Eq. (4) pushes all same-type edges toward positive collinearity and Eq. (7)/(10) reads head direction from the sign of the cosine relative to the prototype. If the ordering is not consistently head-to-dependent, the angular objective conflicts with direction encoding. The paper should clarify the ordering, and include a validation that the predicted direction correlates with UD gold heads independently of sentence position (e.g., by checking that swapping the order of the pair flips the predicted head).","section":"§2.1, Eq. (1); §2.3, Eqs. (4)-(7); §2.4, Eqs. (9)-(10)"}],"minor_comments":[{"comment":"The loss notation is inconsistent: Eq. (4) defines L_A with the Angular Probe matrix B_A, but Eq. (11) uses L_A within the Polar Probe objective for B_P; the authors should state that B_A is replaced by B_P during joint training or introduce a separate notation.","section":"§2.4, Eq. (11)"},{"comment":"The caption contains a typo: 'finds a a linear transform' should read 'finds a linear transform.'","section":"Fig. 1 caption"},{"comment":"The sentence 'The same relative results across probes are conserved for the Balanced Accuracy score' does not report the actual Balanced Accuracy numbers; please provide them in the text or table.","section":"§3 'Comparison with baselines'"},{"comment":"The text says 'as shown in Fig: 5' but the reference should be to Figure 5 without the colon; also the dataset description would benefit from a table of the 100 sentences and their level counts for reproducibility.","section":"§2.5 'Controlled dataset'"},{"comment":"The y-axis label 'Structural and Label Score' is vague, the 'SOTA' point is not explained in the caption or text, and there is no legend for the dashed versus solid markers; please clarify.","section":"Fig. 6"},{"comment":"The phase 'nearly two folds' is awkward; I suggest 'nearly two-fold' or 'roughly twice as accurate.'","section":"§3 'Comparison with baselines'"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an interesting and timely question, and the probe formulation is simple and promising. My recommendation of major revision is driven by the missing label-permutation control, the unfair comparison with the Structural Probe, and the lack of statistical inference; these are fixable within the scope of the paper. I also encourage the authors to temper the abstract's 'spontaneously learn' wording unless the permutation control supports it, since the current evidence shows supervised linear decodability rather than an emergent unsupervised geometry."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the polar decomposition itself: distance codes existence, angle codes type and direction. That is a genuinely new framing, and the paper shows it works—the Polar Probe recovers labeled, directed dependencies substantially better than the Structural Probe, generalizes across layers and models, and peaks at the same layer 16 that Hewitt and Manning found. The evaluations are held out, the controlled benchmark is a nice touch, and the paper is clear about the method. I believe the result that syntactic type/direction is linearly decodable from activation differences, and that the polar geometry is a fair way to describe this code.\n\nThe load-bearing weakness is the missing label-permutation control. The probe is trained on UD labels with an angular objective that explicitly collinearizes same-type edges and orthogonalizes different types. Measuring that structure in the probe's output is partly checking the training objective. The 'No Probe' baseline at 80% AUC for type shows generic lexical or part-of-speech regularities are already in the raw activations. Without a control where the same activations are trained with randomly permuted relation labels, the headline claim that LLMs 'spontaneously learn' this geometry remains unproven. The paper does acknowledge the supervised nature in its limitations, but the abstract oversells it.\n\nTwo smaller issues: the label-accuracy comparison with the Structural Probe is apples-to-oranges, since the Structural Probe was never trained to predict labels; and there are no confidence intervals or significance tests, which makes the reported margins hard to judge. The absence of released code and data also weakens reproducibility, though that is fixable.\n\nOverall, this is a solid probing paper with a clear methodological contribution. The central claim of linear decodability holds up; the stronger claim of spontaneous geometry needs one additional control experiment. A serious referee could extract that with a reasonable revision request.\n\nRecommendation: send it to review. I would not desk-reject this. If I were the editor, I would ask for the permutation control and a tightened abstract before acceptance.","headline":"A clean, useful extension of the Structural Probe showing that dependency type and direction are linearly decodable from angle, but the 'spontaneous' claim needs a label-permutation control.","tokens_in":12868,"tokens_out":1189,"would_cite":true,"duration_ms":13281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In the activations of large language models, dependency syntax is laid out as a polar coordinate system: distance marks whether a relation exists, while relative angle marks its type and direction.","keywords":["polar coordinate system","syntactic trees","dependency grammar","probing","large language models","Universal Dependencies","linear readout","neural representations"],"falsifier":"A decisive control would train the Polar Probe on the same model activations and corpus but with the dependency labels, or their head directions, randomly permuted during training; if the probe still identifies the true labels on a held-out test set, the claimed geometry would not be specifically about syntax, whereas chance-level performance would confirm that it is.","tokens_in":11809,"feed_emoji":"🧭","tokens_out":10566,"duration_ms":92280,"temperature":0.7,"pith_summary":"The paper sets out to show that large language models do not merely place syntactically related words near one another: they organize dependency trees as a polar coordinate system in a low-dimensional subspace of their activations. In that subspace, the distance between two word embeddings marks whether a dependency relation exists, while the relative angle between their difference vector and a prototype vector marks the relation's type, and the sign of that angle marks which word is the head. The authors introduce a single linear transformation, the Polar Probe, trained to satisfy both distance and angular constraints, and they report that it reads the type and direction of syntactic relations roughly twice as accurately as the earlier distance-only probe. If the claim is right, the main symbolic structure of syntax is not an external overlay on neural vectors but a readable geometry the models spontaneously acquire.","feed_headline":"LLMs encode syntax as distance and angle between word vectors","feed_subtitle":"A single linear readout now recovers relation type and head direction, not just whether words are linked.","key_machinery":"The central object is the probed edge vector $B_P s_{ij}$, the image under a linear map of the difference between two contextualized word embeddings; this is what the paper calls the polar coordinate system. Its squared norm encodes whether two words are syntactically linked, and its direction encodes the relation's type and head. The map $B_P$ is the machinery: a single linear transformation trained on the sum of a distance objective and a contrastive angular objective, so that edges of the same dependency type are pulled into the same direction and edges of different types are pushed toward orthogonality. Readout then reduces to comparing each probed edge with type prototype vectors by cosine similarity, making the whole tree linearly decodable.","core_discovery":"The central claim is that a single linear readout of contextualized word embeddings can recover the full labeled and directed dependency tree, not just the unlabeled graph. Writing the edge embedding between words $i$ and $j$ as $s_{ij}=h_i-h_j$, the Polar Probe learns one linear map $B_P$ such that the predicted existence of a relation is $\\hat d=\\|B_P s_{ij}\\|^2$ (Eq. 8), the predicted type is $\\hat t=\\arg\\max_c |\\cos(B_P s_{ij}, V_c)|$ where $V_c$ is the average probed vector for type $c$ (Eq. 9), and the head is $h_i$ when that cosine is nonnegative and $h_j$ otherwise (Eq. 10). The probe is trained jointly on a tree-distance loss and a contrastive angle loss, and the result is that same-type edges become collinear while different-type edges become orthogonal in the probed space. On the English Web Treebank, the probe reaches about 95% AUC for dependency-type classification versus 74% for the structural probe subspace, preserves the structural score for whether a dependency exists, and peaks at the same intermediate layer across three language models. The same angular code is recovered when a main clause is embedded in nested relative clauses, which the paper takes as evidence that the geometry is systematic across tree levels.","pith_inferences":["The paper does not test causality; a natural extension would be to rotate the angular coordinate of a probed edge and check whether the model's syntactic behavior shifts accordingly, which would show the code is used rather than merely correlational.","The same distance-plus-angle scheme could be adapted to other structured domains the paper does not probe, such as semantic role relations, knowledge-graph edges, or object-feature binding in vision.","Because the probe is supervised by one annotation scheme, an unsupervised variant would be needed to decide whether the polar geometry would be discovered from the activations alone or is partly imposed by the chosen labels.","The explicit direction sign suggests a cross-linguistic test: if the same relation types point the same way in head-initial and head-final languages under the same annotation framework, the code would be a candidate universal syntactic primitive."],"forward_implications":["Dependency type and head direction are linearly readable from activation differences, completing the distance-only picture and removing the need for a separate labeled probe.","The full labeled tree fits in about 128 dimensions, far smaller than the layer width, suggesting that syntax occupies a compact geometric subspace rather than being spread across all units.","The code appears at the same intermediate layer across several families of models and sharpens with model scale and recency, so it is a reproducible property of trained representations rather than a quirk of one architecture.","The same relation type is coded by the same direction whether it appears in a short sentence or inside a long-nested relative clause, supporting a coordinate system that is invariant to surface position.","Because a probe trained on a randomly initialized model performs at chance, the polar geometry is inherited from what the model learned, not manufactured by the probe's objective."],"supporting_citations":[{"why":"Introduces the Structural Probe that maps dependency-tree distances to squared Euclidean distances; this is the baseline and the distance-only code the Polar Probe extends.","marker":"Hewitt and Manning, 2019"},{"why":"Supplies the English Web Treebank corpus whose manually annotated sentences are the training and test data.","marker":"Silveira et al., 2014"},{"why":"Defines the Universal Dependencies framework that provides the labeled, directed edges treated as ground truth.","marker":"Nivre et al., 2017"},{"why":"Provides Llama-2-7B, one of the language models whose intermediate activations are probed.","marker":"Touvron et al., 2023"},{"why":"Provides Mistral-7B, another probed model used in the layer-wise and scaling analyses.","marker":"Jiang et al., 2023"},{"why":"Provides BERT-large, the masked language model that shows the same layer-peaking pattern.","marker":"Devlin et al., 2019"},{"why":"Supplies a more complex labeled-tree probe whose reported performance the Polar Probe matches with a simpler linear readout.","marker":"Müller-Eberstein et al., 2022"},{"why":"Supplies the controlled long-nested sentence set used to test whether the polar code stays consistent across tree levels.","marker":"Lakretz et al., 2021b"}],"fun_headline_variants":["Angle between LLM word vectors codes syntax type and direction","LLMs plot syntax in a polar coordinate system of activations","Polar probe: linear readout of distances and angles yields syntax","Syntax geometry: LLMs encode relations as distance and angle","From distances to angles: full dependency trees from LLM probes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manual grammar annotations used as ground truth are the correct specification of syntactic relations, their types, and their head directions, because the polar coordinate system is only measured against those labels.","fun_headline_variants_meta":{"raw":{"variants":["Angle between LLM word vectors codes syntax type and direction","LLMs plot syntax in a polar coordinate system of activations","Polar probe: linear readout of distances and angles yields syntax","Syntax geometry: LLMs encode relations as distance and angle","From distances to angles: full dependency trees from LLM probes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1560,"prompt_tokens":1054,"completion_tokens":506,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":421}},"tokens_in":670,"tokens_out":506,"duration_ms":5667,"temperature":1.0,"reasoning_tokens":421,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:34:56.905261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive control would train the Polar Probe on the same model activations and corpus but with the dependency labels, or their head directions, randomly permuted during training; if the probe still identifies the true labels on a held-out test set, the claimed geometry would not be specifically about syntax, whereas chance-level performance would confirm that it is.","supporting_citations":[{"cited_title":"and Manning, C","cited_arxiv_id":null,"evidence_quote":"Introduces the Structural Probe that maps dependency-tree distances to squared Euclidean distances; this is the baseline and the distance-only code the Polar Probe extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the English Web Treebank corpus whose manually annotated sentences are the training and test data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Universal Dependencies framework that provides the labeled, directed edges treated as ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Llama-2-7B, one of the language models whose intermediate activations are probed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT-large, the masked language model that shows the same layer-peaking pattern."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a more complex labeled-tree probe whose reported performance the Polar Probe matches with a simpler linear readout."}],"review_version":1}