{"id":"5e602c77-713e-4ef1-8479-fae26431d2f0","arxiv_id":"2608.01283","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Per-token Riemannian metrics make attention scores non-factorizable, but the paper does not prove they prevent rank collapse and the claimed computational overhead is understated.","lead":"This paper proposes giving each Transformer token its own learned Riemannian metric, so attention can measure distances in a token-specific geometry. It proves a structural property of these scores and sketches an architecture, but does not prove the hoped-for anti-collapse benefit and contains an error in its cost analysis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.3's O(r/d) overhead claim omits the cost of projecting every key under every token's metric; actual attention overhead is O(r), not O(r/d).","rationale":"The paper is transparent about what is proven versus conjectured: the non-Gram theorem is explicitly a structural observation, not a rank-preservation proof, and the anti-collapse question is labeled open. The single stated mathematical claim that must be true for the central practical contribution to hold is Proposition 4.3. That claim fails: the metric-weighted pairwise interaction (Section 4.2, Equation (16)) requires U_i^T k_j, which depends on both i and j, so it cannot be amortized into per-token precomputation. The correct asymptotic cost is O(BL^2 d r), making the overhead ratio O(r) rather than the advertised O(r/d). This directly undermines the abstract's 'negligible overhead at billion-parameter scale' claim. The microbenchmark or corrected asymptotic count described above would settle the point. I agree with the reader's identification of this as the weakest assumption, and the reader's REJECT verdict follows from the false feasibility proposition. I do not see a separate objection that would change the verdict; the paper's own limitations sections already disclose the other major gaps.","tokens_in":22420,"tokens_out":11907,"duration_ms":112451,"concrete_test":"Re-derive the Section 4.3 complexity table including the cost of computing U_i^T k_j for every pair. Concretely, instrument a small implementation of Equation (15) for L=1024, d=512, r=8: compute all U_i^T k_j by the paper's only available route (for each i, multiply U_i^T against all keys, O(L d r) per i) and count multiply-adds relative to standard QK^T. If the metric-correction cost scales as L^2 d r (ratio ~r) rather than L^2 r (ratio ~r/d), Proposition 4.3 is refuted as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim (Abstract and Proposition 4.3) is invalid as stated. Equation (15)-(16) require, for every pair (i,j), the vector U_i^T k_j (and symmetrically U_j^T q_i). The paper's per-pair cost O(r) for (U_i^T q_i)^T(U_i^T k_j) assumes U_i^T k_j is already available; computing it costs O(d r) per pair, or O(L d r) per i for U_i^T K, hence O(L^2 d r) total. This term is not covered by the per-token precomputation of U_i^T q_i listed in Section 4.2. The corrected complexity of Riemannian score computation is O(BL^2 d r), not O(BL^2 d + BL^2 r); the overhead ratio is O(r), not O(r/d). For r=8, d=4096, the promised 'negligible' overhead is actually about an 8x increase in attention-score FLOPs, before metric generation and inversion. Since the practical-feasibility contribution is the only quantitative bridge to 'billion-parameter scale', this error is load-bearing. The non-Gram theorem and the anti-collapse conjecture are explicitly qualified in the paper, but Proposition 4.3 is stated as a mathematical result and its complexity table is wrong.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the Euclidean inner-product attention of Transformers with per-token low-rank Riemannian metrics g_t = I + U_t U_t^T. Its claimed contributions are: (1) a proof that heterogeneous per-token metrics make the attention score matrix non-Gram in a universal factorization sense (Theorem 3.2); (2) a perturbation analysis identifying a critical metric-strength regime (Proposition 3.5, Conjecture 3.6); (3) a computational-feasibility analysis claiming negligible overhead O(r/d) for Riemannian attention (Propositions 4.1--4.3); (4) a complete architecture specification, the Fiber Bundle Transformer, with MetricNet, TorsionNet, geodesic attention, curvature and torsion proxies, and a metric-preconditioned FFN; and (5) several predictions and open problems. The paper is explicitly theoretical and reports no experiments.","tokens_in":22816,"tokens_out":10730,"duration_ms":96085,"significance":"If the stated results were correct, the paper would provide a clean structural observation about the algebraic difference between Euclidean and Riemannian attention, and the Fiber Bundle Transformer would be a concrete architectural proposal targeting rank collapse. The paper deserves credit for carefully separating proven results from conjectures, for explicitly identifying the central open problem (whether heterogeneous metrics prevent row-stochastic rank collapse), and for stating the limitations of its own perturbation analysis. However, the computational-feasibility claim is the only quantitative bridge to practical scale, and that claim is invalid as written. As a result, the paper's central practical conclusion is unsupported, and the main structural theorem is stated too broadly.","major_comments":[{"comment":"The complexity table and the 'negligible overhead' claim are incorrect. For every pair (i,j), the score requires the vectors U_i^T k_j and U_j^T q_i, because the expansions in Equation (16) contain the terms ||U_i^T k_j||^2 and (U_i^T q_i)^T (U_i^T k_j), and symmetrically for U_j^T q_i. Computing U_i^T k_j for all j costs O(L d r) per token i, and U_j^T q_i for all i costs O(L d r) per token j, giving a total of O(B L^2 d r), not O(B L^2 r), for the pairwise interaction. The per-token precomputation of U_i^T q_i listed in Section 4.2 does not supply these per-pair projections. The corrected attention overhead ratio relative to standard attention is O(r), not O(r/d). In addition, MetricNet as specified in Definition 5.2 uses Linear(d,d*m) followed by Linear(d*m,d*r); with the stated choice m=1 this requires O(d^2 r) operations per token, not the O(B L d r) listed in the table. These errors invalidate Proposition 4.3 and the abstract's claim of feasibility at billion-parameter scale with negligible overhead.","section":"Section 4.3, Proposition 4.3 and Equations (15)--(16)"},{"comment":"The theorem statement overclaims relative to its own proof. The premise that the family {U_t U_t^T} spans a subspace of Sym(d) of dimension at least 2 does not imply that no O(d)-dimensional universal factorization exists: if the span has dimension m = O(d), Equation (7) gives only d' ≥ m, which is compatible with d' = O(d). The Ω(d^2) lower bound in the proof requires the family to span all of Sym(d), or at least a subspace of dimension Ω(d^2), not merely a non-constant family. Furthermore, Equation (7) ignores the Euclidean part of the score: the term -||q_i - k_j||^2 alone already requires the factor maps to encode the d-dimensional dot product, so the minimal universal factorization dimension is at least d even for homogeneous metrics. The limitation stated in Section 8.3, that for fixed inputs with L ≤ d a trivial d' = O(d) factorization exists, also contradicts the unqualified abstract statement. The theorem should be restated in the universal, fixed-functional-form sense with the full-span condition made explicit.","section":"Section 3.2, Theorem 3.2 and Equation (7)"}],"minor_comments":[{"comment":"The symbol \\bar{U}^2 is used both as the running maximum max_t ||U_t||_F^2 in Proposition 3.5 and as the critical parameter \\bar{U}_{crit}^2 in Conjecture 3.6; this overloading is confusing and should be clarified.","section":"Section 3.3, notation"},{"comment":"The decomposition of s_Riem - s_Euc is helpful, but the label 'constant in row i' for the -||q_i||^2/tau term is imprecise: the term is per-row constant and therefore cancels in the row-wise softmax, whereas the key-norm penalty -||k_j||^2/tau genuinely varies across columns.","section":"Section 5.8, Equation (28)"},{"comment":"There are numerous typographical artifacts such as 'donot', 'O(BLdr 2)', and missing spaces in the PDF text; a careful proofreading pass is needed before resubmission.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The feasibility error in Proposition 4.3 is load-bearing because it is the paper's only quantitative argument for practical scalability; correcting it changes the paper's central pitch from 'negligible overhead' to a substantial overhead that the current architecture does not support. The non-Gram theorem also needs a substantive restatement before its claims match its proof. These are not merely presentation issues, and the revision would require replacing, rather than repairing, the main claims as currently written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. The non-Gram theorem (Theorem 3.2) is real and new: with heterogeneous per-token metrics, the Riemannian score matrix cannot be factored as QK^T with O(d)-dimensional factors. And the authors are refreshingly explicit that this structural observation does not solve the rank-collapse problem; they state the anti-collapse conjecture as open and give a perturbation analysis that only bites in the sub-critical regime. Good faith.\n\nThe trouble is Proposition 4.3. The claim that the geodesic metric correction adds only O(BL^2 r) pairwise cost is wrong. Equation (16) needs U_i^T k_j and U_j^T q_i for every pair, and those projections are not covered by the per-token precomputation. Computing them costs O(d r) per pair, so the total is O(BL^2 d r), and the overhead ratio is O(r), not O(r/d). For r=8 and d=4096 this is an 8x increase in attention-score FLOPs before metric generation and inversion. Since the 'negligible overhead' claim is the bridge to billion-parameter feasibility, this is load-bearing, not a typo.\n\nThe rest is a mixed bag. The Woodbury inversion result (Prop 4.1) is correct but standard. The architecture spec is detailed and the authors flag the approximations they're making, which I respect. The scaling discussion in Conjecture 6.4 is honest about not knowing r_min. But there's no empirical validation, and the main conjecture remains unproven, so the practical payoff is entirely contingent.\n\nCitation pattern looks fine; the Nash embedding discussion is a bit casual but serviceable. The paper's own limitations section is a model of candor—it tells you exactly where the argument stops.\n\nWho is this for? Someone working on geometric attention or rank-collapse theory, who wants a clear statement of the non-Gram obstruction and a concrete architecture to think about. It deserves a serious referee, because the structural claim and the design are worth scrutiny, but I'd expect major revision on the complexity analysis—the current form doesn't support the headline feasibility conclusion.\n\nI'd bring it to reading group mainly to watch someone find the complexity error, but I wouldn't cite it this year.","headline":"A new non-Gram result the authors correctly refuse to oversell, undermined by a load-bearing complexity error that flips the promised overhead from O(r/d) to O(r).","tokens_in":23248,"tokens_out":2762,"would_cite":false,"duration_ms":23361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Riemannian attention scores under heterogeneous per-token metrics cannot be factored as $QK^\\top$ with $O(d)$-dimensional factors, and the paper proves this while showing the geometric operations stay computationally cheap.","keywords":["Riemannian attention","rank collapse","non-Gram matrix","per-token metric","geodesic distance","Transformer architecture","low-rank metric","Woodbury identity"],"falsifier":"Train a pure self-attention stack without residuals or feed-forward layers, with heterogeneous per-token metrics $g_t = I + U_tU_t^\\top$ at metric strength above the conjectured threshold $\\Theta(\\tau/L)$ and sequence length $L = \\Omega(d^2)$, and record effective rank across depth: doubly-exponential decay to rank 1 would refute the anti-collapse conjecture, while rank stabilizing above 1 would support it. A complementary check is to time the metric correction and test whether its cost scales as $O(r/d)$ of the standard attention cost or as $O(L r/d)$.","tokens_in":22198,"feed_emoji":"📐","tokens_out":9672,"duration_ms":73539,"temperature":0.7,"pith_summary":"This paper argues that the Euclidean inner product at the heart of Transformer attention is a fixable structural flaw: by replacing the single global metric with a learned, low-rank Riemannian metric carried by each token position, the attention score matrix loses its Gram structure and can no longer be written as $QK^\\top$ with $O(d)$-dimensional factors. The paper proves this non-Gram property, shows that the needed geometric operations—geodesic distances and metric inversion—cost only $O(d\\cdot r)$ to $O(d\\cdot r^2)$ per token when the metric is a low-rank perturbation of the identity, and presents a complete architecture, the Fiber Bundle Transformer, built around per-token metrics. The authors are explicit that the non-Gram result is structural, not a proof that rank collapse is cured: the post-softmax attention matrix remains row-stochastic, and whether heterogeneous metrics preserve rank in the large-perturbation regime is left as the central open problem. If the conjectured anti-collapse effect holds, deep Transformers could retain higher effective rank without residual connections or extra width.","feed_headline":"Per-token Riemannian metrics break QK^T attention factorization","feed_subtitle":"New proof forces O(d^2)-dimensional score factorization, a possible escape from rank collapse.","key_machinery":"The load-bearing construction is the low-rank per-token metric $g_t = I_d + U_tU_t^\\top$ with $U_t \\in \\mathbb{R}^{d\\times r}$, $r \\ll d$, which keeps every geometric operation tractable while still letting the geometry bend differently at every position. The pairwise metric $g_{ij} = (g_i + g_j)/2$ turns the squared geodesic distance into the Euclidean distance plus two metric corrections, and it is exactly the quadratic sub-terms—each token's vector measured under the other token's metric—that block an $O(d)$-dimensional factorization. On the computational side, the Woodbury identity converts metric inversion into an $O(d r^2)$ operation and the geodesic decomposition turns pairwise score computation into per-token precomputation plus $O(r)$ per-pair terms.","core_discovery":"The central discovery is Theorem 3.2: with heterogeneous per-token metrics $g_t = I + U_t U_t^\\top$, the Riemannian attention scores $s_{ij}^{\\mathrm{Riem}} = -(q_i-k_j)^\\top g_{ij}(q_i-k_j)/\\tau$ cannot be factorized as $\\phi_i^\\top \\psi_j$ with factors of dimension $O(d)$; any universal factorization needs dimension at least $\\min\\{\\dim\\mathrm{span}\\,\\{U_tU_t^\\top\\}_t,\\ d(d+1)/2\\}$, which is $\\Omega(d^2)$ when the metric factors span the symmetric matrices. The proof locates the obstruction in the quadratic sub-terms $\\|U_i^\\top k_j\\|^2$ and $\\|U_j^\\top q_i\\|^2$, which are quadratic in one token's vector under the other token's metric, and it shows the bound is tight because vectorization separates the score at dimension $d^2$. The paper pairs this structural result with a feasibility analysis: the low-rank form makes geodesic distance computable by a per-token decomposition in $O(d\\cdot r)$ and metric inversion via the Woodbury identity in $O(d\\cdot r^2)$, avoiding the $O(d^3)$ cost of a general matrix. It then specifies the Fiber Bundle Transformer, in which attention is geodesic distance, feed-forward steps are metric-preconditioned, and curvature and torsion enter through explicit proxies.","pith_inferences":["Beyond the paper: even if the anti-collapse conjecture fails, the non-Gram structure changes the inductive bias of attention, so the architecture could still generalize differently at the same parameter count; effective-rank measurements alone may miss that effect.","Beyond the paper: the claimed $O(r/d)$ overhead is computed from per-token precomputation, but the per-pair terms $U_i^\\top k_j$ and $U_j^\\top q_i$ sum to $O(L^2 d r)$; a direct cost accounting is needed before the 'negligible overhead' claim guides scaling decisions.","Beyond the paper: the critical metric-strength threshold $\\bar U^2_{\\mathrm{crit}} = \\Theta(\\tau/L)$ suggests a small-scale experiment—train identical models with metric strengths just below and above the threshold and compare the depth profile of effective rank.","Beyond the paper: because the metric factor $U_t$ is a low-rank parameter on the positive-definite cone, the design connects naturally to parameter-efficient fine-tuning methods, where the same factor could serve as both geometry and adapter."],"forward_implications":["Standard attention becomes a special case: when all metric factors are identical, the Riemannian scores reduce to a Gram matrix under a global modified inner product, and the known doubly-exponential rank-collapse analysis applies unchanged.","The distance-based score parameterization introduces a key-norm penalty and a temperature-scaling mismatch relative to dot-product attention, so any empirical comparison must separate these artifacts from the genuine metric effect.","The metric correction can be fused into existing IO-aware attention kernels as an additive bias, so the architecture need not materialize an $L \\times L$ score matrix.","Train/inference blending mismatch is generically suboptimal: evaluating a model at a blending coefficient different from the one used in training yields a confounded result unless the model is retrained at the inference coefficient.","Metric collapse to the identity is predicted to be the dominant failure mode, so anti-flatness, diversity, and smoothness regularizers are needed to keep the per-token geometry active."],"supporting_citations":[{"why":"Establishes the doubly-exponential rank-collapse result for pure self-attention stacks that motivates the geometric diagnosis.","marker":"[3]"},{"why":"The Nash embedding theorem invoked to argue that widening a Euclidean hidden space cannot reproduce the expressivity of explicit metrics.","marker":"[1]"},{"why":"Supplies the dimensional-blowup and differentiability objections that block the Nash-embedding escape route for learned representations.","marker":"[2]"},{"why":"Defines the standard attention score $QK^\\top/\\sqrt{d}$ that the non-Gram theorem contrasts against.","marker":"[5]"},{"why":"The IO-aware attention kernel into which the paper claims the metric correction can be fused as an additive bias.","marker":"[13]"}],"fun_headline_variants":["Riemannian attention needs d^2 factors, not QK^T","Heterogeneous metrics force attention scores beyond QK^T","Fiber Bundle Transformer: attention via geodesics, no QK^T","Geodesic attention breaks QK^T factorization","Per-token metrics demand O(d^2) attention factors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The promise that Riemannian attention adds negligible overhead rests on a cost accounting in which the per-pair metric corrections are cheap; a direct count of the pairwise terms scales with sequence length squared, hidden dimension, and metric rank, so the overhead premise is not established.","fun_headline_variants_meta":{"raw":{"variants":["Riemannian attention needs d^2 factors, not QK^T","Heterogeneous metrics force attention scores beyond QK^T","Fiber Bundle Transformer: attention via geodesics, no QK^T","Geodesic attention breaks QK^T factorization","Per-token metrics demand O(d^2) attention factors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001349,"raw_usage":{"total_tokens":5572,"prompt_tokens":1134,"completion_tokens":4438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":4348}},"tokens_in":750,"tokens_out":4438,"duration_ms":29143,"temperature":1.0,"reasoning_tokens":4348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:10:10.522106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a pure self-attention stack without residuals or feed-forward layers, with heterogeneous per-token metrics $g_t = I + U_tU_t^\\top$ at metric strength above the conjectured threshold $\\Theta(\\tau/L)$ and sequence length $L = \\Omega(d^2)$, and record effective rank across depth: doubly-exponential decay to rank 1 would refute the anti-collapse conjecture, while rank stabilizing above 1 would support it. A complementary check is to time the metric correction and test whether its cost scales as $O(r/d)$ of the standard attention cost or as $O(L r/d)$.","supporting_citations":[{"cited_title":"Dong, J.-B","cited_arxiv_id":null,"evidence_quote":"Establishes the doubly-exponential rank-collapse result for pure self-attention stacks that motivates the geometric diagnosis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Nash embedding theorem invoked to argue that widening a Euclidean hidden space cannot reproduce the expressivity of explicit metrics."},{"cited_title":"Vaswani et al","cited_arxiv_id":null,"evidence_quote":"Defines the standard attention score $QK^\\top/\\sqrt{d}$ that the non-Gram theorem contrasts against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The IO-aware attention kernel into which the paper claims the metric correction can be fused as an additive bias."}],"review_version":2}