{"id":"66963a2f-7d08-4e3b-a2df-69bccda9696c","arxiv_id":"1908.09341","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper defines a set-to-set cosine similarity via projections onto linear spans and applies it to paraphrase detection, but the experiments are too weak to support the claimed advantages.","lead":"This paper proposes a similarity score for two sets of vectors, defined by projecting each vector onto the linear span of the other set and averaging the resulting cosines. It applies the score to Russian paraphrase classification, reporting an accuracy of 0.81, but the evaluation is weak and the claimed sensitivity to word order is not supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SIM formula in §2.1 is not normalized: identical groups of size >1 score above 1, so the empirical thresholds in Table 2 are uninterpretable.","rationale":"I came to the paper expecting the main risk to be the singular-matrix caveat identified by the reader; that is a real limitation for long or rank-deficient sentences. However, careful reading of §2.1 reveals a more immediate defect: the step from vector-to-group cosine to group-to-group similarity is not normalized. The derivation of cos(b,A) = |p|/|b| is mathematically sound and yields values in [0,1], but the symmetrized formula sums over all vectors in both groups and divides only by 2. Unless each sum is first divided by its group size, the result scales with the number of words and violates the paper's own axioms. This is load-bearing because every empirical number in Table 2 depends on the SIM scale, and thresholds in Eq. (14) are fitted to that scale. The singular-matrix issue is acknowledged and deferred, but the normalization problem is not acknowledged at all and affects even the cleanest invertible cases. I therefore recommend keeping the REJECT verdict, and I would place this normalization flaw ahead of the singular-matrix limitation as the decisive correctness risk. I agree with the reader only partially: the singular-matrix concern is valid and related, but it is not the most basic flaw in the construction.","tokens_in":6139,"tokens_out":8398,"duration_ms":92486,"concrete_test":"Take A = B = I_3, i.e. three mutually orthogonal unit vectors. Compute Eq. (13) for each vector: each projection is the vector itself, so each cos = 1. Apply the SIM formula exactly as printed in §2.1 and obtain SIM = (3+3)/2 = 3, contradicting the stated requirement that a full match equals 1. Then recompute with per-group means, ((1/3)∑cos(b_i,A) + (1/3)∑cos(a_j,B))/2, yielding 1. If the authors' implementation gives 1 for identical sentences, their code uses a different formula than the one in the paper; if it gives 3, the threshold-fitting procedure in Table 2 operates on an unnormalized scale. The same check can be repeated on real paraphrase pairs with A = B to verify whether SIM(A,A) = 1 as required.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central formula for group-to-group similarity is not actually a normalized similarity even in the non-singular case where Eq. (13) is valid. The paper states that the proximity coefficient should take values from 0 to 1 and equal 1 for a full match, but the printed symmetrization SIM(A,B) = (∑_i cos(b_i,A) + ∑_j cos(a_j,B))/2 divides only by 2, not by the group sizes. Take A = B = I_3, the 3×3 identity matrix with row vectors e1, e2, e3. Then A·A^T is invertible, every cos(e_i,A) = 1, and the formula gives SIM = (3 + 3)/2 = 3. Identical sets therefore do not receive similarity 1 and the value can exceed 1. To be a genuine average, the formula must divide the first sum by |B| and the second sum by |A|, and then average the two means. The paper does not state that such normalization is silently applied. This matters directly for the empirical claims: the thresholds a and b in Eq. (14) and all accuracies in Table 2 are fitted to this SIM scale. If the implementation used the formula as printed, the scores are not confined to [0,1] and the fitted thresholds are meaningless; if the implementation used a normalized version, then the evaluated method differs from the formula presented as the paper's contribution. Either way, the headline empirical result is unsupported. This defect is more basic than the acknowledged singular-matrix caveat: it already fails on well-conditioned, equal-size, invertible inputs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a proximity measure between two sets of vectors. For a vector b and a matrix A, it defines cos(b, A) as the Euclidean length of the orthogonal projection of b onto the row space of A divided by |b|, as given in Eq. (13). For two groups A and B, it then proposes a symmetrized average of vector-to-group cosines. This measure is applied to Russian paraphrase classification, where thresholds a and b are fit on a 70/30 split and results are compared with the authors' semantic-cores method and with a public competition leaderboard. The paper reports accuracy 0.81 and F1 0.5169 for Alekseev's method.","tokens_in":6359,"tokens_out":6718,"duration_ms":68674,"significance":"The core projection-cosine formula is geometrically natural, and the algebraic derivation of Eq. (13) is correct under the stated invertibility condition. The method is also essentially parameter-free apart from two classification thresholds, and the authors are explicit that singular matrices are deferred to future work. If the measure were correctly defined and carefully evaluated, a simple vector-group similarity of this kind could be useful for text-comparison tasks. However, the symmetrization formula as printed is not normalized, and the experimental evaluation has serious flaws, so the paper's main claims are not currently supported.","major_comments":[{"comment":"The symmetrization formula immediately after Eq. (13) is not a normalized similarity. As printed, SIM(A,B) = (Σ_i cos(b_i,A) + Σ_j cos(a_j,B)) / 2 divides only by 2, not by the group sizes |B| and |A|. For A = B = I_3 (the 3×3 identity matrix), every cosine term is 1 and the formula gives SIM = 3, not 1. This violates the paper's own requirement in Section 1 that the proximity coefficient lie in [0,1] and equal 1 for a full match. Consequently, the thresholds a and b in Eq. (14) and all entries in Table 2 are calibrated to an unnormalized scale; if the implementation used a properly normalized version, then the evaluated method differs from the formula presented. Either way, the headline empirical results are unsupported and the definition must be corrected.","section":"Section 2.1, definition of SIM(A,B)"},{"comment":"Eq. (13) requires the square matrix A·A^T to be invertible, which holds only when the rows of A are linearly independent. Real sentence groups frequently have linearly dependent word vectors, or more words than embedding dimensions, making A·A^T singular. The conclusion explicitly defers this case to future work, but Section 3.2 does not report how singular matrices were handled in the experiments, what fraction of inputs were singular, or whether a pseudoinverse or regularization was used. Without this information, the method as defined is inapplicable to many real inputs and the experiments cannot be reproduced.","section":"Section 2.1 and Conclusion (singular matrices)"},{"comment":"The comparison with public competition results is not valid. The cited competition used 7,000 training pairs and 1,000 test pairs, while the reported 0.81 accuracy is obtained on a 30% holdout (roughly 573 records) of 1,913 records, with thresholds optimized on the training part. No confidence intervals, error bars, or significance tests are provided, and the two-class and three-class results in Table 2 vary widely across adjacent rows. The statement that Alekseev's method scored 0.81 accuracy versus the competition's 0.59 is therefore not an apples-to-apples comparison and does not support the paper's central claim.","section":"Section 3.2 and Table 2 (evaluation validity)"},{"comment":"The experimental section does not specify the pre-trained word embeddings, the exact composition of the semantic and grammatical feature vectors, or the precise construction of the 1,913-record dataset. Since the behavior of the proposed measure depends entirely on the vector representations used, the results cannot be replicated or meaningfully compared with other methods without these details.","section":"Section 3.2 (reproducibility)"}],"minor_comments":[{"comment":"The abstract claims the measure 'does not lose any semantic characteristics and takes account of the words' traits,' but no formal definition or empirical evidence is given for this statement; it should be either substantiated or removed.","section":"Abstract and Section 1"},{"comment":"The matrix notation with F, C, and M is confusing and is not used later in the paper; consider removing it or clearly connecting it to the word-vector group formulation.","section":"Section 1, Eq. (1)"},{"comment":"The sentence 'This will no longer require nm/2 calculations, but only m' is unclear, especially since the symmetrized version is state to require n+m calculations; please clarify the intended comparison.","section":"Section 2.1"},{"comment":"The column heading 'Bound Evaluation' is ambiguous, and it is not explained how the lower and upper thresholds a and b are optimized or whether the reported accuracies are on the training or validation set.","section":"Table 2"},{"comment":"Reference [5] is a blog post; for a survey of text similarity methods, the authors should cite a peer-reviewed primary source, e.g., [3] and [4], and use those consistently.","section":"References"}],"recommendation":"reject","confidential_remarks":"The normalization error in the central similarity formula and the invalid comparison with the competition leaderboard are, in my view, load-bearing defects that cannot be remedied by a local edit; the experiments would need to be redone and the definition corrected, which goes beyond a normal revision. The paper also lacks critical experimental details and is not currently reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core projection-cosine derivation is standard and correct, but the paper's symmetrized formula divides by 2 instead of by group sizes, so SIM(A,A)=3 for two copies of a 3-vector identity group. The stress-test is right. That single issue guts the empirical section, because the thresholds a and b in Eq. (14) and all Table 2 numbers are fit to that particular scale. If the code silently normalized by group size, then the evaluated method is not the formula in the paper; if it didn't, the thresholds are meaningless. Either way the headline result is unsupported.\n\nWhat's actually here: the vector-to-group cosine in Eq. (13) is just the standard cosine between a vector and its orthogonal projection onto a subspace, written as sqrt(b A^T(AA^T)^{-1} A b^T)/|b|. The derivation is correct for invertible AA^T, and the paper deserves credit for presenting it cleanly and for noting the O(n+m) complexity versus O(nm) pairwise cosines. The comparison against their own semantic-cores method on the same 70/30 split is at least honest, though the held-out set is small (about 570 pairs) and there are no error bars.\n\nThe soft spots beyond normalization: (1) The abstract and intro claim the method is sensitive to word order, but projection onto a span is permutation-invariant, so it cannot distinguish word order any better than a bag-of-words method. That undercuts the stated motivation. (2) The external comparison against 'competition results' uses different training and test data, so the 0.81 vs 0.59 accuracy claim is not a valid comparison. (3) The singular-matrix caveat is acknowledged, but the normalization bug is more basic—it fails on well-conditioned equal-size inputs. (4) The paper cites its own prior work for the baseline, which is fine, but the 'Alekseev's method' naming without citing the underlying linear algebra is a bit cheeky.\n\nWho this is for: someone looking for a simple set-to-set similarity for sentence embeddings might glance at it, but in the current form the math doesn't hold up where it counts. The idea is salvageable—normalize properly, rerun, and make the word-order claim honest—but the preprint as written is not a reliable research contribution.\n\nRecommendation: reject in current form. If the authors fix the normalization and re-report results, it could be worth a referee; as it stands, a desk reject with a clear explanation of the normalization error would be appropriate.","headline":"The paper's headline measure is unnormalized as printed—identical groups score 3, not 1—so the empirical thresholds and accuracies built on it don't support the claims.","tokens_in":6974,"tokens_out":3320,"would_cite":false,"duration_ms":33559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training-free projection cosine, averaged symmetrically over two word-vector groups, measures sentence similarity and scores 0.81 accuracy on Russian paraphrase classification.","keywords":["Neural Networks","Machine Learning","NLP","Paraphrasing","Text Similarity","Vector Representation","sentence similarity","orthogonal projection"],"falsifier":"Take a sentence pair from the paraphrase task in which one sentence has more words than the embedding dimension or contains repeated words, so the rows of $A$ are linearly dependent; the formula in the paper is then undefined as written. Tracking how often this happens in the reported test set, and what accuracy a pseudoinverse or regularized variant achieves, would directly show whether the 0.81 result generalizes to ordinary text.","tokens_in":5845,"feed_emoji":"🧮","tokens_out":10611,"duration_ms":95934,"temperature":0.7,"pith_summary":"The paper tries to establish that two groups of vectors can be compared by the angle each vector makes with the linear span of the other group, rather than by averaging each group into a single vector or by computing all pairwise cosines. The proposed text-proximity measure symmetrizes those angles and, on the Russian paraphrase task, is reported to reach 0.81 accuracy and 0.5169 F-measure, better in accuracy than the trained semantic-cores baseline it compares against and the published competition results it cites. If the claim is right, a parameter-free geometric score can serve as a general set-to-set similarity for any object that has vector representations, at a cost of only $n+m$ projection computations instead of $n\\times m$ pairwise comparisons. The paper also claims the measure preserves semantic characteristics of individual words and can be combined with syntactic or morphological vector components.","feed_headline":"Projection cosine without training scores 81% on Russian paraphrases","feed_subtitle":"A parameter-free geometric distance between word-vector groups rivals trained paraphrase classifiers.","key_machinery":"The load-bearing object is the projection cosine: $\\cos(b,A)=\\sqrt{b\\cdot A^T(A\\cdot A^T)^{-1}A\\cdot b^T}/|b|$. It measures the cosine of the angle between a vector $b$ and its orthogonal projection onto the linear span, which the paper calls a plane, formed by the rows of $A$. The group-to-group similarity is the symmetrized average $\\mathrm{SIM}(A,B)=\\left(\\sum_i \\cos(b_i,A)+\\sum_j \\cos(a_j,B)\\right)/2$, computed in $n+m$ projection evaluations rather than $n\\times m$ pairwise vector cosines. This formula only works when $A\\cdot A^T$ is invertible, meaning the group's vectors must be linearly independent.","core_discovery":"The paper's central proposal is to define the cosine of the angle between one vector $b$ and a group of vectors $A$ as the ratio of the length of the orthogonal projection of $b$ onto the subspace spanned by $A$ to the length of $b$: $\\cos(b,A)=\\sqrt{b\\cdot A^T(A\\cdot A^T)^{-1}A\\cdot b^T}/|b|$. The proximity of two groups $A$ and $B$ is then the symmetrized average of $\\cos(b_i,A)$ over all $b_i$ in $B$ and $\\cos(a_j,B)$ over all $a_j$ in $A$. The authors argue that this measure preserves the contribution of each word's vector rather than collapsing a sentence into one aggregate, that it is invariant under adding or removing vectors linearly dependent on the group's basis, and that it can be combined with extra vector components such as syntactic or morphological features. They report that on a Russian paraphrase corpus the method achieves 0.81 accuracy and 0.5169 F-measure for two-class classification, outperforming the semantic-cores method on the same data.","pith_inferences":["Editorial extension: if singular groups are handled by a pseudoinverse, the measure likely becomes a canonical angle between subspaces; comparing it with the sum of squared principal angles between the two spans would place it in standard subspace geometry.","Editorial extension: the reported comparison with competition results uses different protocols, so a direct head-to-head run on the same 1000-pair test set would be a straightforward way to confirm the claimed advantage over trained systems.","Editorial extension: the method measures content coverage rather than word order, so combining it with a small order-sensitive feature, such as positional weighting, is a natural testable variant."],"forward_implications":["Set-to-set similarity becomes a parameter-free geometric computation: no trained classifier is needed, only embeddings and orthogonal projections.","The score can be merged with any kind of vector feature, including syntactic and morphological annotations, without changing the formula.","Because the cosine is invariant under adding vectors linearly dependent on the group's basis, redundant words do not tilt the score toward mere repetition.","The cost structure makes the method practical for large collections, since the matrix $A^T(A A^T)^{-1}A$ for a fixed group can be reused across many query vectors."],"supporting_citations":[{"why":"Supplies the survey of text similarity approaches against which the paper positions its proposed method.","marker":"[3]"},{"why":"Describes the semantic cores method used as the paper's main trained comparison baseline.","marker":"[1]"},{"why":"Provides the earlier proximity measure that the semantic cores model uses in the reported numerical experiment.","marker":"[2]"}],"fun_headline_variants":["No-training cosine projection hits 81% on Russian paraphrases","Geometric sentence similarity: 81% on Russian without training","Group-vector cosine beats aggregation, scores 81% in Russian","Projection cosine for word groups: 81% on Russian paraphrase test","Parameter-free cosine measure: 81% accuracy on Russian pairs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measure is only defined when the word vectors in each group are linearly independent, because the formula requires inverting $A A^T$; real sentences often violate this, and the paper leaves that singular case to future work.","fun_headline_variants_meta":{"raw":{"variants":["No-training cosine projection hits 81% on Russian paraphrases","Geometric sentence similarity: 81% on Russian without training","Group-vector cosine beats aggregation, scores 81% in Russian","Projection cosine for word groups: 81% on Russian paraphrase test","Parameter-free cosine measure: 81% accuracy on Russian pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2395,"prompt_tokens":965,"completion_tokens":1430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1341}},"tokens_in":581,"tokens_out":1430,"duration_ms":10668,"temperature":1.0,"reasoning_tokens":1341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:15:13.993827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sentence pair from the paraphrase task in which one sentence has more words than the embedding dimension or contains repeated words, so the rows of $A$ are linearly dependent; the formula in the paper is then undefined as written. Tracking how often this happens in the reported test set, and what accuracy a pseudoinverse or regularized variant achieves, would directly show whether the 0.81 result generalizes to ordinary text.","supporting_citations":[{"cited_title":"The Training of Neuromodels for Machine Comprehension of Text. Brain2Text Algorithm","cited_arxiv_id":"1804.00551","evidence_quote":"Describes the semantic cores method used as the paper's main trained comparison baseline."},{"cited_title":"Neural Network-based Object Classification by Known and Unknown Features (Based on Text Queries)","cited_arxiv_id":"1906.00800","evidence_quote":"Provides the earlier proximity measure that the semantic cores model uses in the reported numerical experiment."}],"review_version":1}