{"id":"99e0353f-9d20-4523-977c-128e8b34eade","arxiv_id":"2506.02996","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Spatial relation words in LLaMA and Qwen models form antipodal, roughly orthogonal directions in a low-dimensional subspace, and steering along these directions changes the model's output.","lead":"This paper reports that LLaMA and Qwen models encode spatial words like 'above', 'below', 'left', and 'right' as linear directions in their internal activations, and that nudging these directions shifts the model's answers. The authors interpret this as evidence of an internal spatial world model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The composition result is an artifact of projecting onto a PCA subspace fit to the same atomic directions; a held-out or null-subspace test is needed before accepting the linear world model claim.","rationale":"Reader's REJECT is justified. The weakest assumption is the independence of the PCA subspace from the composition comparison. The paper's own numbers make the artifact visible: Table 1 shows \"in front\"/\"behind\" at layer 24 with original cosine 0.1130 and PCA cosine 0.9950; Table 2 shows mean composition cosine 0.3952 in the original space and 0.9931 in PCA. Because the subspace is fit to the very atomic vectors whose sums are being tested, the projection is expected to place sums in the subspace and to remove the orthogonal residual that accounts for most of the original-space discrepancy. A genuine low-dimensional spatial code could also behave this way, which is why the proposed check matters: out-of-sample subspaces and null-subspace baselines would distinguish a real structure from a projection artifact. The paper does get credit for near-perfect linear probe accuracy in the full activation space and for consistent inverse-pair patterns across Llama-3.2-3B, Llama-3.2-1B, and Qwen3-1B; those results survive this concern. But the central claim requires composition, and the only quantitative support for composition is the PCA comparison. The causal steering experiment lacks the controls needed to independently establish functional use of this subspace. I therefore keep the reader's REJECT.","tokens_in":16907,"tokens_out":5243,"duration_ms":65616,"concrete_test":"Re-run the composition analysis (Section 5.2, Tables 2 and 3) with the PCA subspace computed from a held-out subset of atomic directions: e.g., fit the 2D subspace using only above/below/left/right and test composed relations involving \"in front of\"/\"behind\", then reverse the split; also run the same metric on random subspaces of the same dimension and on subspaces fit to non-spatial probe directions. If the mean PCA cosine drops toward the original-space value (~0.40) or matches the random-subspace baseline, the composition result is a projection artifact.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing evidence for Property 2 (composition) is Tables 2 and 3: in a PCA subspace the vector for \"above and right\" aligns with \"above\" + \"right\" (mean cosine 0.9931 in 2D), versus 0.3952 in the original 3072-dimensional activation space. This subspace is not independent of the quantities being compared: it is fit to the six atomic probe directions (Section 5.1, Appendix F), so sums of atomic vectors lie in that span, and the projection discards exactly the orthogonal components that make the original-space comparison fail. Table 1 makes the artifact visible: the \"in front\"/\"behind\" pair goes from cosine 0.1130 (83.5 degrees) in the original space to 0.9950 (5.8 degrees) in PCA at layer 24; a projection chosen from those same vectors inflates antipodal alignment even for pairs that are nearly orthogonal in the model. Consequently, the 0.993 cosine is not evidence of an internal Euclidean geometry; it is largely a property of the projection. Because composition is one of the two properties in the paper's Definition/Property 2 and the abstract's \"linear spatial world models\" claim depends on it, this is a load-bearing validity threat. The causal steering results do not repair it: steering vectors are also built from the same PCA directions, and no control using random directions, unrelated relations, or null steering is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a formal definition of a linear spatial world model as a subspace of the residual stream that is approximately isomorphic to R^3, with atomic spatial relations encoded as basis vectors, inverse relations as antipodal vectors, and composed relations as vector sums. The authors train linear and nonlinear probes on a synthetic dataset of object-relation sentences, apply PCA to the probe directions, and report that the resulting low-dimensional subspace exhibits antipodal inverse relations and compositional structure. They also perform activation steering in this subspace and report a 74.3% success rate, concluding that LLMs encode linear spatial world models that are causally used during generation.","tokens_in":17204,"tokens_out":8824,"duration_ms":105039,"significance":"If the central claim were established, this would be a meaningful contribution to interpretability, connecting the linear representation hypothesis with spatial reasoning in LLMs and extending prior work on world models in chess and Othello. The paper has several strengths: it reports original-space metrics alongside PCA metrics, tests multiple model families (LLaMA-3.2-3B, LLaMA-3.2-1B, Qwen3-1.7B), provides a formal definition, and attempts causal interventions. However, the main evidence for the world-model claim is compromised by a circular analysis: the PCA subspace used for the antipodal and composition tests is fitted to the same atomic relation vectors that are subsequently evaluated. The causal steering experiments lack null controls. As a result, the paper does not currently provide valid empirical support for its abstract claim that LLMs encode linear spatial world models.","major_comments":[{"comment":"The composition analysis is circular. The PCA subspace into which vectors are projected is computed from the six atomic probe directions, as described in Appendix F. The sum of atomic vectors therefore lies in the span of the fitted directions, and the projection is chosen to maximize the variance of those exact directions. The reported PCA cosines (mean 0.9931, angle 6.02°) are measured between this sum and the projected composed vector, while the original-space comparison gives mean cosine 0.3952 and angle 66.72°. The inflation from 66.72° to 6.02° is a direct consequence of discarding the orthogonal components, not evidence of internal Euclidean structure. A proper test would use a subspace fitted on held-out relations or on an independent set of directions, or would compare against a null distribution of random subspaces. Without such a control, Property 2 and the composition claim are unsupported.","section":"Section 5.2, Tables 2 and 3"},{"comment":"The antipodal-structure claim suffers from the same circularity. In original activation space, only left/right shows a strong negative cosine (-0.964 at layer 24), while above/below has cosine 0.4465 and in front/behind has cosine 0.1130, which is nearly orthogonal. After projection into a PCA subspace fitted to these same six relation vectors, all pairs become near-antipodal (e.g., in front/behind goes from 0.1130 to 0.9950 at layer 24). This transformation is an expected property of projecting a small set of vectors onto their own principal subspace, not a discovery about the model's internal geometry. The paper needs an independent method for identifying the spatial subspace, such as cross-validated PCA on held-out relations or a subspace derived from object-position activations, before the basis property can be accepted.","section":"Section 5.1, Table 1"},{"comment":"The causal steering experiment lacks the necessary baselines. Steering vectors are constructed from the same PCA directions used elsewhere in the paper, and there is no control condition using random directions of matched norm or directions from unrelated semantic relations. Without such controls, the observed 74.3% success rate does not demonstrate that the spatial subspace is causally used; any direction with a consistent effect on the output could produce steering. In addition, the 'behind' condition succeeds in only 5% of trials, which is far below the 16.7% chance level for six alternatives. The paper does not discuss this result, but it suggests that the steering direction for 'behind' has an opposite or non-specific effect, further weakening the claim that all six basis directions are functionally active.","section":"Section 5.4, Figure 4"},{"comment":"The near-perfect linear probe accuracy may reflect direct decoding of the surface relation words rather than an abstract spatial world model. The synthetic prompts contain the explicit relation terms ('above', 'below', 'left of', etc.) in the input, and the probes are trained on the final token's residual stream. The model could be linearly separating the input words themselves. The paper does not include a control with paraphrased relation descriptions, non-spatial relations, or object positions that are not verbally stated. Without such controls, the probing results are consistent with lexical memory rather than with a grounded spatial representation.","section":"Section 5.1 and Appendix E"}],"minor_comments":[{"comment":"The word 'In ront' appears in the steering success table; this should be 'In front'.","section":"Figure 4 caption"},{"comment":"The definition of a spatial world model in Section 3.1 is W = <R^3, O, S>, but Section 5.3 writes W = <R^3, O, S, T>, introducing an undefined T. Please align the notation.","section":"Section 5.3 Motivation"},{"comment":"The sentence 'Results are summarized in Appendix 4' is unclear; there is no 'Appendix 4'. Presumably the authors mean Table 4 in Appendix E, but this should be corrected.","section":"Section 5.1 Results"},{"comment":"There are several typos, including 'strucutred' instead of 'structured' and 'structre' instead of 'structure'.","section":"Section 5.1 and 5.3"},{"comment":"The text says the authors tested 'six canonical spatial relations and their inverses', but Figure 4 lists only six conditions. Please clarify whether the reported rates are for the forward relations only or pooled over both directions, and why the table does not include inverse conditions separately.","section":"Section 5.4 and Figure 4"},{"comment":"The Qwen3-1.7B model is referenced to Qwen2 and Qwen2.5 technical reports (refs [53, 54]); please provide the correct citation for the Qwen3 model actually used.","section":"Appendix C"}],"recommendation":"reject","confidential_remarks":"The core empirical claim of the paper rests on a circular analysis: the PCA subspace is fitted to the same atomic vectors that are then used to evaluate antipodality and composition. The original-space numbers are far weaker, and the causal steering lacks the most basic null controls. I see no way to interpret the current results as evidence for a linear spatial world model. The paper would need a substantially redesigned evaluation, including cross-validated subspace discovery and proper baselines, before the claim can be assessed. I recommend rejection, though a future resubmission with such controls could be viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central claim—that LLMs encode a linear spatial world model with compositional vector addition—is not supported by the evidence as presented. The composition test (Property 2) is evaluated in a PCA subspace fit to the six atomic probe directions that define the sums. In that subspace composed vectors align beautifully (mean cosine 0.99), but in the original 3072-dimensional activation space the same comparison gives mean cosine 0.40 and angle 66.7 degrees. The projection is doing the work. Table 1 makes this visible: at layer 24, 'in front'/'behind' go from cosine 0.11 in original space to 0.995 in PCA. A projection chosen from those same vectors inflates antipodal alignment even for relations that are nearly orthogonal in the model. So the abstract's 'linear spatial world models' claim rests on a projection artifact.\n\nThat said, there is real value here. The formal definition of a spatial world model (basis, antipodal inverses, composition) is clean and useful. The probing results are solid: linear probes hit 100% accuracy at multiple layers, matching nonlinear probes, which is genuine evidence that spatial relation information is linearly decodable. The antipodal structure in PCA—even if inflated—does appear consistently across models (Llama-3.2-3B, 1B, Qwen3), which is worth noting. The steering results (74.3% overall) are suggestive, but they too use the same PCA directions, and there are no controls with random directions or null steering.\n\nThe paper also omits a key experimental detail: how the composed relation sentences (e.g., 'above and right') were generated. That matters for reproducibility. The limitations section is honest about scope but does not flag the circularity in the composition analysis.\n\nMy take: the paper is a genuine attempt with a good formal skeleton, but the load-bearing composition result is methodologically unsound as reported. The fix is straightforward—evaluate composition in a held-out subspace or in the original space with proper baselines. The weaker claim (linear decodability of spatial relations, some antipodal structure) likely survives; the strong claim (Euclidean isomorphism with vector arithmetic) does not. I would send this to a serious referee, because the question is important and the flaws are fixable. A desk reject would miss the salvageable core.\n\nRecommendation: engage with it as a borderline paper that needs a major revision: report original-space composition metrics, add a null-subspace control, describe composed sentence generation, and add steering baselines. Then the community can judge whether a linear spatial world model actually emerges.","headline":"The composition claim is not supported by the evidence as presented because the PCA subspace is fit to the same atomic directions being summed; the paper deserves a rigorous referee but the headline result needs a held-out spatial test.","tokens_in":17700,"tokens_out":1710,"would_cite":false,"duration_ms":22638,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs keep a vector map of space inside their activations.","keywords":["world models","spatial reasoning","linear representation","probing","activation steering","interpretability","compositionality","residual stream"],"falsifier":"Fit the subspace using only a subset of atomic directions, say above, below, left, and right, and then measure whether the composed vectors for \"above and behind\" or \"left and in front\" align with vector sums in that fixed subspace; if their cosine falls back to roughly the original-space value of 0.39 rather than the reported 0.99, the composition claim does not survive.","tokens_in":16707,"feed_emoji":"🧭","tokens_out":5450,"duration_ms":61165,"temperature":0.7,"pith_summary":"The paper's claim is that a large language model builds a linear spatial world model as a by-product of next-token prediction: inside the residual stream there is a low-dimensional subspace, approximately a copy of $\\mathbb{R}^3$, in which \"above\", \"left\", and \"in front\" act as orthogonal basis directions, their opposites are negative directions, and phrases such as \"above and to the left\" are encoded as vector sums. If this is right, spatial language is not surface mimicry: the model has an internally consistent geometry that can be read and manipulated. The authors test this on a synthetic dataset of object-position sentences, using linear probes, PCA projections, and activation steering, and report that linear probes match nonlinear ones, that composed relations align with vector sums in the PCA subspace, and that steering these directions changes generated answers in 74.3% of trials. A sympathetic reading is that the paper supplies the strongest current evidence for a compositional spatial code that is not merely present but causally used.","feed_headline":"LLMs store space as a simple 3-D vector geometry","feed_subtitle":"Probe directions act like basis vectors; steering them flips spatial answers in most trials.","key_machinery":"The central object is the PCA subspace recovered from six atomic probe directions. Linear probes are trained on residual-stream activations to classify spatial relations; each probe contributes a direction vector $\\mathbf{w}_i$. Principal component analysis projects these six directions onto a low-dimensional plane or 3D volume, and that projected subspace is treated as the model's internal $\\mathbb{R}^3$. It does the work of turning the claim \"linear spatial world model\" into testable geometry: inverse pairs are scored by cosine, independent pairs by orthogonality, composed relations by angle to the vector sum, and steering vectors are reconstructed by lifting the PCA directions back into the residual stream.","core_discovery":"On the paper's own terms, the discovery is that LLMs encode linear spatial world models. The authors define a spatial world model as a tuple $\\mathcal{W}=\\langle \\mathbb{R}^3, O, S\\rangle$ and propose that in the model's contextual embeddings there exists a basis $\\{\\vec{r}_{\\text{left}}, \\vec{r}_{\\text{above}}, \\vec{r}_{\\text{in front}}\\}$ satisfying three geometric properties: inverse relations are antipodal, independent atomic relations are orthogonal, and composed relations equal vector sums of their parts. The evidence is a probing and intervention pipeline on a 3-billion-parameter instruction-tuned transformer. Linear probes recover all six spatial relations essentially perfectly, PCA projections of the six probe directions show the predicted antipodal and orthogonal configuration, object embeddings cluster at consistent locations in the same subspace, and adding the projected directions back into the residual stream steers outputs toward the intended relation, with near-perfect success for \"above\", \"below\", and \"left\". The authors also report that composition holds in the PCA-reduced space with a mean cosine of 0.9931 and a mean angular deviation of 6.02 degrees in 2D, while the same comparison in the original activation space is far weaker, with a mean cosine of 0.3952.","pith_inferences":["A fair test would hold out composed directions from the PCA fit; because the reported subspace is fitted to the six atomic vectors being compared, the near-perfect composition alignment in PCA space is likely inflated relative to the model's true internal geometry.","If the spatial code is as algebraic as the paper suggests, a natural next test is whether motion and temporal dynamics occupy the same subspace as linear transition operators, since the paper explicitly leaves transition functions out.","A stronger statement would follow if continuous coordinates, not just six categorical relations, were linearly decodable; the current dataset only asks for discrete directions.","The steering result hints at an alignment application: injecting spatial directions could correct or constrain object-relation behavior without fine-tuning, though the paper does not test downstream task performance."],"forward_implications":["Spatial relations are linearly decodable from the residual stream at every tested layer (8, 16, 24), so a downstream system can read object positions with a simple affine map rather than a deep decoder.","Because composed expressions are vector sums in the PCA subspace, unseen combinations such as \"behind and to the right\" should be predictable by arithmetic on atomic directions.","Since steering the identified directions changes the model's next-token behavior, the subspace gives a practical handle for controlling spatial outputs in instruction-tuned models.","Objects occupy consistent positions in the same subspace, so the geometry supports tracking multiple object configurations in a shared frame, not just pairwise relation labels.","The atomic antipodal structure also appears in two smaller model families, suggesting the linear spatial organization is not unique to one scale or architecture."],"supporting_citations":[{"why":"Supplies the paradigmatic finding that a sequence model trained on a synthetic board game learns a linear world model, which the paper extends to LLM spatial relations.","marker":"[6]"},{"why":"Supplies the structural-probe methodology for reading linguistic structure out of word representations.","marker":"[24]"},{"why":"Supplies the linear representation hypothesis that justifies treating linear probes as evidence of linear internal structure.","marker":"[37]"},{"why":"Supplies the activation-addition method used to steer the model along the identified spatial directions.","marker":"[31]"},{"why":"Describes the model family used for the main experiments.","marker":"[45]"},{"why":"Supports the broader claim that language models learn state-tracking world models, here for chess.","marker":"[33]"},{"why":"Provides the warning that probing accuracy does not establish causal use, motivating the steering experiments.","marker":"[26]"},{"why":"Supports the claim that next-token prediction can yield a linear emergent world representation in a small transformer.","marker":"[13]"}],"fun_headline_variants":["LLMs keep a linear 3D map of the world","LLMs' spatial models are linear and steerable","Probing reveals LLMs' linear spatial world models","LLMs encode space as linear vector geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the PCA subspace fitted to the six atomic probe directions faithfully reveals the model's internal spatial geometry; if the projection merely rearranges those six chosen vectors, the claimed Euclidean basis and vector-sum composition could be artifacts of the fitting procedure.","fun_headline_variants_meta":{"raw":{"variants":["LLMs keep a linear 3D map of the world","LLMs' spatial models are linear and steerable","Probing reveals LLMs' linear spatial world models","LLMs encode space as linear vector geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00117,"raw_usage":{"total_tokens":4819,"prompt_tokens":901,"completion_tokens":3918,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":3855}},"tokens_in":517,"tokens_out":3918,"duration_ms":28130,"temperature":1.0,"reasoning_tokens":3855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:11:09.543335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the subspace using only a subset of atomic directions, say above, below, left, and right, and then measure whether the composed vectors for \"above and behind\" or \"left and in front\" align with vector sums in that fixed subspace; if their cosine falls back to roughly the original-space value of 0.39 rather than the reported 0.99, the composition claim does not survive.","supporting_citations":[{"cited_title":"Chess as a testbed for language model state tracking","cited_arxiv_id":null,"evidence_quote":"Supports the broader claim that language models learn state-tracking world models, here for chess."},{"cited_title":"Actually, othello-gpt has a linear emergent world representation.Neel Nanda’s Blog, March 2023","cited_arxiv_id":null,"evidence_quote":"Supports the claim that next-token prediction can yield a linear emergent world representation in a small transformer."}],"review_version":1}