{"id":"7f264f7d-5245-48a0-beca-652379dec9c8","arxiv_id":"1909.00504","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Word relationships in embedding space can be represented as orthogonal or linear transformations, not just translation vectors, with comparable or better analogy-solving accuracy.","lead":"This paper shows that word relationships in embedding space, usually captured by vector arithmetic, can also be captured by orthogonal rotations and general linear maps with comparable accuracy. The finding suggests that downstream models such as attention heads might learn relationships through rotations, and it questions the use of analogy tasks as a measure of embedding quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation is circular: R and A are fit to approximate the translation vector b, and b is computed from the same word pairs used for scoring, so the reported accuracies do not establish that rotations/linear maps are generalizable relationship representations.","rationale":"I read the paper's central claim as an existence/generalization claim: besides translation vectors, simple linear maps (including rotations/reflections) can represent word relationships, and a linear map can even beat translation on analogies. For that claim to hold, the evaluation must show that the matrices learned from some relationship evidence work on relationship instances not used to derive them. The paper instead derives b from all test pairs and then trains R/A to approximate b. The high accuracies are therefore a check on how well orthogonal Procrustes and least squares can approximate a fixed translation on random words, not a measure of how well the learned transformations represent the semantic relation. The reader flagged exactly this issue as the weakest assumption, and I agree. A held-out split would settle it: if R and A trained from half the pairs still map held-out source words to their targets, then the alternative representation is real and the paper's practical implication survives. If not, the conclusion reduces to a statement about approximating a translation by a rotation, which is less interesting and not about word relationships. I would keep the reader's CONDITIONAL verdict: the math is sound and the idea is worth testing, but the current evidence is insufficient. No ad hominem or style issues; the concern is purely about experimental control.","tokens_in":7327,"tokens_out":6758,"duration_ms":318117,"concrete_test":"Partition each analogy category's ordered pairs into two disjoint halves, T_train and T_test. Compute b_train from T_train only; construct X from random word vectors, Y = X + b_train, and fit R and A via Eqs. (2)–(3). Score top-1 accuracy and average cosine similarity on T_test only (excluding source and target words from the candidate list), and compare with a translation baseline using b_train. Repeat over 10 random splits and report mean ± std. If held-out accuracies remain close to Table 1 and exceed the translation baseline, the concern is resolved; if they drop materially, the current results are inflated by the shared b.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is that the same translation vector flows through both the construction of the transformations and the evaluation. In §4.1, b is defined by Eq. (1) as the mean of y−x over all word pairs in an analogy category; those same pairs are then the test set ('evaluated by how accurately it maps source words to target words in this set of word pairs'). The matrices R and A are fit to map random source vectors X to Y = X + b (Eqs. 2–3), i.e., they are explicitly trained to approximate the operation x ↦ x + b. Consequently, high Table 1 accuracy mostly demonstrates that R and A mimic b on random words, and that b happens to fit the pairs from which it was computed. The comparison with vector arithmetic is not an independent benchmark: the translation baseline uses the same b on the same pairs. This setup cannot support the implication that downstream models can learn word relationships as orthogonal or linear transformations; no relationship-defining information is held out from the quantity used to build the scorer. The central claim therefore rests on a single nontrivial fact—that a rotation can approximate a translation on a large random sample—and the measured accuracies are not evidence of generalization to unseen exemplars of the relationship.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether word relationships in embedding spaces can be represented by orthogonal or general linear transformations, rather than only by translation vectors. For each analogy category, the author computes the mean translation vector b by averaging pairwise differences over all ordered pairs in that category (Eq. 1), constructs source and target matrices X and Y = X + b from n = 2000 random vocabulary words, solves the orthogonal Procrustes problem (Eq. 2) and the ordinary least squares problem (Eq. 3) to obtain R and A, and then evaluates how accurately each transformation maps source words to target words in the same set of pairs used to define b. The reported results give average accuracies of 0.761 for orthogonal, 0.782 for translative, and 0.798 for linear transformations on GloVe, with similar trends on FastText. The paper argues that these are parsimonious representations of word relationships and discusses implications for embedding evaluation, Transformer attention heads, and debiasing.","tokens_in":7650,"tokens_out":4031,"duration_ms":40015,"significance":"If the empirical claim were established with an independent evaluation, the paper would provide a useful existence proof: closed-form orthogonal and linear maps can approximate translation-based analogical structure, which is relevant to understanding how downstream models might encode semantic and syntactic relationships. The paper is clearly written, uses standard closed-form solvers, and Figure 1's saturation in n and consistency across GloVe and FastText are informative. However, the current evaluation is not independent: the same translation vector both defines the transformations and is derived from the test pairs. The core novelty therefore remains unverified, and the paper's main quantitative claims require re-analysis with held-out pairs and sampling variability. The authors' explicit statement in Section 3 that they are 'ultimately learning to approximate a translation' is honest, but it also highlights that Table 1 primarily measures how well R and A mimic b on random points and how well b self-consistently fits the pairs from which it was computed.","major_comments":[{"comment":"The evaluation is circular. Equation (1) computes b by averaging y - x over all word pairs in a category, and Section 4.1 states that each transformation is evaluated on 'this set of word pairs' — that is, the same pairs used to define b. Equations (2) and (3) then fit R and A to approximate the map x -> x + b on a random sample of words, so both matrices are explicitly trained to mimic b. Scoring on the pairs used to define b cannot distinguish whether the orthogonal or linear transformation represents the relationship itself or merely reproduces b's fit to its own training data. The central claim in Section 4.2 that linear transformations are more accurate than translations, and the abstract's statement that these transformations 'can be as good a representation of word relationships as translation vectors,' are therefore not supported by Table 1 as reported. Please re-run the evaluation with held-out pairs: compute b on one subset of pairs, fit R and A, and evaluate on a disjoint subset of pairs from the same relationship. Report both in-category and cross-category generalization, and clarify whether the relationship is defined by the training subset or by an external resource.","section":"Section 4.1, Eq. (1), Table 1"},{"comment":"No measure of sampling variability is reported. The three average accuracies (0.761, 0.782, 0.798) differ by at most 0.037, but the matrices R and A depend on a random sample of n = 2000 words, and the paper gives no error bars, no repeated draws, and no significance test. A single random draw could easily shift these numbers by a nontrivial amount. Please report means and standard deviations over multiple random seeds (or bootstrap over word pairs), and test whether the linear-vs-translation and orthogonal-vs-translation differences are statistically reliable. This is load-bearing for the paper's main comparative claim.","section":"Section 4.2, Table 1"},{"comment":"The average cosine-similarity results are also computed on the same pairs used to define b, so they do not mitigate the circularity concern. At most, they show that the transformed source vectors are close to the target vectors on the training pairs; they do not establish that the transformations generalize to unseen exemplars of the relationship. Please report cosine similarity on held-out pairs as well, and consider whether this metric is meaningfully different from measuring how well R x approximates x + b when b was fitted to the same pairs.","section":"Section 4.2, cosine-similarity column"}],"minor_comments":[{"comment":"Please specify the exact accuracy criterion: is a pair counted correct only if the transformed source vector's nearest vocabulary word equals the target, and is the candidate set restricted to exclude the source word and possibly other words in the same category? This matters for comparability with standard analogy evaluations such as 3CosAdd.","section":"Section 4.1"},{"comment":"The 'Avg' row appears to be the unweighted mean over categories; since categories differ substantially in the number of pairs, please report the number of pairs per category and either weighted accuracy or both weighted and unweighted averages.","section":"Table 1"},{"comment":"The paper states that accuracy plateaus for n >= 250 and similarity for n >= 500, yet the main results use n = 2000. Please explain the choice of n = 2000 and confirm that the qualitative ranking of the three methods is stable for n between 250 and 2000.","section":"Section 4.2, Figure 1"},{"comment":"The claim that a single attention head has sufficient capacity to represent a word relationship is an extrapolation from a global linear map on static embeddings; attention heads apply learned linear maps in a context-dependent way, so the implications for Transformer architecture should be stated more cautiously.","section":"Section 4.3"},{"comment":"There are several typos and formatting issues: 'unsupservised' should be 'unsupervised', 'V oita' should be 'Voita', and the mathematical notation (e.g., bold vectors) is inconsistent in places.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical contribution is currently undercut by the circular evaluation: the same translation vector is used to construct the transformations and to score them on the same word pairs. I believe this is fixable within the manuscript's scope by adding held-out evaluation and error bars, but the conclusions may change substantially depending on what that evaluation shows. The paper is otherwise well written and the underlying question is worthwhile."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the Rotate King paper. The idea is clean: instead of a translation vector, a word relationship can be a rotation or a general linear map. Fitting those maps with closed-form Procrustes or least squares from a random sample of words is neat, and the connection to attention heads is a nice speculative payoff. The paper is honest about not recommending the method for analogy solving. What's actually new is showing that a single rotation can approximate the gender or country-capital relationship almost as well as translation on the tested pairs. That's interesting because it gives a parsimonious alternative to translation and suggests models might learn relationships in more than one geometric form. The soft spot is not subtle. The translation vector b is computed by averaging differences over all word pairs in a category. That same b is then used to construct R and A, and the same word pairs are the test set. So the accuracy numbers measure how well R and A mimic the operation x -> x+b on random words, and how well b happens to fit the pairs it was computed from. It is not an independent test of whether rotations generalize to held-out exemplars of the relationship. The paper essentially acknowledges this when it says it is learning to approximate a translation and that it is not solving analogies in the traditional sense, but it still presents Table 1 as evidence that linear transforms are more accurate than translations. With no held-out pairs and no error bars, those differences could well be noise. The central conceptual claim survives, but the comparative accuracy claims do not. That said, the flaw is fixable. Hold out some pairs to compute b, fit R and A on random words plus that b, and evaluate on the remaining pairs. Also report variance across random samples. The paper is short and the math is correct, so a serious referee could request that and likely get a solid contribution. The citation pattern looks fine; related work is covered. I would bring this to reading group as a cautionary tale about circular evaluation, and I would cite the conceptual framing despite the empirical weakness. It deserves peer review, but with a request for a proper held-out evaluation.","headline":"A novel idea about orthogonal relationship representations, but the evaluation is circular, so the accuracy claims are not established.","tokens_in":696,"tokens_out":1266,"would_cite":true,"duration_ms":34862,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that word relationships in embedding space can be represented as orthogonal or linear transformations with accuracy comparable to translation vectors, suggesting models need not encode relationships as translations.","keywords":["word embeddings","word analogies","orthogonal transformations","linear transformations","translation vectors","orthogonal Procrustes","embedding geometry","downstream NLP models"],"falsifier":"Split each analogy category in half: compute the mean direction and fit the orthogonal and linear maps on the first half, then score on the second half; if accuracy falls to chance, the headline result depends on the transformations seeing the test pairs' average.","tokens_in":7162,"feed_emoji":"🔄","tokens_out":8002,"duration_ms":68383,"temperature":0.7,"pith_summary":"Word relationships in embedding spaces are usually treated as translations: king minus man plus woman lands near queen. This paper argues that the same relationships can be carried by orthogonal transformations (rotations plus reflections) and by general linear maps, and that these alternatives are not just theoretical. Fitting an orthogonal matrix to approximate the mean translation vector for a relationship solves analogy-style mappings with 0.761 average accuracy, versus 0.782 for translations; a general linear map does better still, at 0.798. If the claim holds, analogy tests do not prove that embeddings store relationships as translations, and downstream models could learn relationships with purely linear components such as attention heads.","feed_headline":"Rotations match translations on word analogies","feed_subtitle":"A rotation or linear map learned from embedding pairs is nearly as accurate as classic king - man + woman arithmetic.","key_machinery":"The central object is a relationship-specific map $f$ that sends a source vector $x$ to a target $y$: translation $x \\mapsto x + b$, orthogonal $x \\mapsto R x$ with $R^\\top R = I$, or linear $x \\mapsto A x$. To build $R$, the paper solves the orthogonal Procrustes problem, minimizing $\\| \\Omega X - Y \\|_F$ subject to $\\Omega^\\top \\Omega = I$, where $X$ stacks $n$ randomly sampled word vectors and $Y$ shifts them by the mean translation vector $b$; $A$ is the ordinary least squares solution $Y X^\\top (X X^\\top)^{-1}$. Scoring the same transformations on analogy pairs then measures whether a simple linear map can carry the relationship.","core_discovery":"The paper's central claim is that a word relationship can be represented as an orthogonal or linear transformation of the embedding space with accuracy comparable to, and sometimes better than, the standard translation vector. Concretely, the author takes the mean difference vector b for a relationship, samples random word vectors to form a source matrix X and a target matrix Y = X + b, and solves the orthogonal Procrustes problem for R and the ordinary least squares problem for A. On analogy categories from Mikolov et al. (2013a), the orthogonally transformed source lands on the correct target 76.1% of the time, the translation 78.2%, and the linear map 79.8%; cosine similarity to the true target is nearly equal, suggesting the result is not an artifact of a sparse candidate set. The conclusion is that parsimonious linear representations of relationships exist between simple translations and complex non-linear maps.","pith_inferences":["A held-out evaluation that computes the mean translation vector only from training pairs would test whether orthogonal maps match translations because they genuinely represent the relationship or because the fitting step uses the same pairs that are later scored.","The same Procrustes fitting could be applied to contextualized embeddings to ask whether attention heads and feed-forward layers actually encode relationships as rotations; the paper raises this possibility but does not test it.","Fitting a random orthogonal matrix as a baseline would show how much of the near-parity is due to the flexibility of linear maps rather than to the specific structure of the embeddings."],"forward_implications":["Analogy-test accuracy alone no longer distinguishes translation-based from rotation-based representations, so arithmetic analogy performance should be re-examined as a measure of embedding quality.","A single linear layer or attention head has enough capacity to carry a semantic or syntactic relationship, so attention heads found to track syntax may also track relationships such as gender.","Debiasing by subtracting a bias subspace from word vectors may miss bias encoded as a rotation or linear map, allowing gender associations to persist after projection-based debiasing.","Because orthogonal maps preserve inner products, an orthogonally transformed embedding space still allows recovery of the implicitly factorized word-context matrix, which a translated space does not."],"supporting_citations":[{"why":"Defines the observation that word relationships are linear substructures and supplies the analogy categories used for evaluation.","marker":"Mikolov et al., 2013a"},{"why":"Proves that exact arithmetic analogies require strict corpus conditions, motivating a search for alternative linear representations.","marker":"Ethayarajh et al., 2019a"},{"why":"Provides the closed-form orthogonal Procrustes solution used to fit the rotation matrix R.","marker":"Schönemann, 1966"},{"why":"Supplies the pre-trained GloVe vectors used in the main experiments.","marker":"Pennington et al., 2014"},{"why":"Formalizes the 3CosAdd arithmetic baseline that translations are compared against.","marker":"Levy and Goldberg, 2014a"},{"why":"Shows that Transformers rely on linear maps in attention, the downstream architecture the paper argues can represent relationships as transformations.","marker":"Vaswani et al., 2017"}],"fun_headline_variants":["Word relationships: rotations rival translations","Orthogonal transforms match translation for analogies","Rotating word vectors solves analogies too","Word analogies via rotation: as good as translation","Linear maps outperform translation for analogies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that letting the rotation or linear map learn the average direction of the very word pairs that will be used to score it is a fair measure of how well it represents the relationship; if that learning step leaks the answers, the reported accuracies are overstated.","fun_headline_variants_meta":{"raw":{"variants":["Word relationships: rotations rival translations","Orthogonal transforms match translation for analogies","Rotating word vectors solves analogies too","Word analogies via rotation: as good as translation","Linear maps outperform translation for analogies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1823,"prompt_tokens":990,"completion_tokens":833,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":768}},"tokens_in":606,"tokens_out":833,"duration_ms":6705,"temperature":1.0,"reasoning_tokens":768,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:45:47.668599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split each analogy category in half: compute the mean direction and fit the orthogonal and linear maps on the first half, then score on the second half; if accuracy falls to chance, the headline result depends on the transformations seeing the test pairs' average.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained GloVe vectors used in the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that Transformers rely on linear maps in attention, the downstream architecture the paper argues can represent relationships as transformations."}],"review_version":1}