{"id":"957cc919-89fb-4c72-b1b9-ad82880a28c6","arxiv_id":"2606.01294","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"CCQ adds a curvature-based query contraction to linear attention backbones, improving perplexity, retrieval, and long-context performance on GLA and Gated DeltaNet at low extra cost.","lead":"The paper introduces Curvature-Conditioned Query (CCQ), a read-time contraction of the query in linear attention derived from a second-order Taylor expansion of softmax geometry using running key covariance. A smart generalist might read it to see whether a cheap geometric fix can close the gap between fast linear attention and standard attention on long-context and retrieval tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Taylor expansion at isotropic point justifies covariance curvature only locally; no argument shows this yields effective contraction for typical queries far from the expansion point.","rationale":"The reader's weakest_assumption already isolates the same step. Because the original review examined only the abstract, the reported perplexity and retrieval gains cannot yet be read as confirmation that the local quadratic supplies a useful operator. The concrete test above directly probes whether the geometric motivation survives outside the expansion neighborhood; a negative result would require the paper to supply either a different justification or stronger empirical isolation of the CCQ component.","tokens_in":1749,"tokens_out":464,"duration_ms":28148,"concrete_test":"Construct a toy retrieval task with 128 keys drawn from a two-component Gaussian mixture (means separated by 4σ). Sample queries from the same mixture. Compute (a) standard linear attention weights and (b) weights after applying the covariance-derived contraction operator. Measure the increase in weight on the single target key; repeat for 500 draws. If the median improvement is <5 % or changes sign when query norm is scaled by 3×, the local-curvature justification does not transfer to the operating regime.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central construction begins from the second-order Taylor expansion of the log-partition log Z(q) = log ∑_j exp(q · k_j) evaluated at an isotropic-attention point q_iso (taken to be the point of uniform attention, typically q = 0 after centering). The Hessian at that point equals the key covariance Σ under the uniform measure. The paper then defines a linear operator from Σ that contracts the incoming query before the linear read. This step is load-bearing because the Taylor remainder grows with ||q - q_iso||; attention queries are produced by learned projections whose norms and alignments are independent of q_iso. If the operating regime lies outside the neighborhood where the local quadratic remains faithful, the covariance-derived contraction has no guaranteed relation to the true geometry of softmax and may not preferentially suppress high-density directions. The abstract asserts composability and gains on GLA/Gated DeltaNet, yet supplies no global bound, no comparison of query norms to the expansion radius, and no ablation isolating the contraction from other implementation choices.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Curvature-Conditioned Query (CCQ), a read-time modification for linear attention. A second-order Taylor expansion of the softmax log-partition at the isotropic point yields a local quadratic whose Hessian equals the key covariance; this covariance is maintained recurrently and used to define a linear contraction operator applied to the incoming query before it reads the fast-weight state. CCQ is presented as composable with any linear-attention backbone and is evaluated on GLA and Gated DeltaNet, reporting gains in perplexity, zero-shot accuracy, S-NIAH retrieval (in- and out-of-context), length extrapolation (4K to 20K), and LongBench at modest extra cost.","tokens_in":1969,"tokens_out":485,"duration_ms":16337,"significance":"If the local Taylor-derived contraction remains effective for learned queries, the approach supplies a lightweight, geometry-motivated fix for the additive dilution problem in linear attention without altering the write path. The composability claim and the reuse of an already-maintained covariance statistic are attractive; successful validation would be a modest but concrete advance for efficient long-context models.","major_comments":[{"comment":"§3.2 (derivation of the CCQ operator): the second-order Taylor expansion of log Z(q) is performed at the isotropic point q_iso; the resulting Hessian is identified with the key covariance Σ. No bound on the remainder term or estimate of typical ||q − q_iso|| for the learned query projections is supplied. Because the contraction operator is applied directly to arbitrary queries, the local nature of the approximation is load-bearing for the central claim that the operator preferentially suppresses high-density directions.","section":"§3.2"}],"minor_comments":[{"comment":"The experimental section should report standard deviations across seeds and explicit ablations that isolate the contraction operator from other implementation choices (e.g., gating or chunking).","section":"§4"},{"comment":"Notation for the isotropic point and the precise definition of the contraction matrix (e.g., whether it is Σ^{-1/2} or a regularized variant) should be stated once in a single equation block for clarity.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comment on the local nature of the Taylor approximation in §3.2. The observation correctly identifies that the validity of the curvature-based contraction rests on the quality of the second-order expansion for the queries encountered in practice. We address this directly below and will strengthen the manuscript accordingly.","responses":[{"response":"We acknowledge that the manuscript does not supply an analytic bound on the Lagrange remainder or a quantitative estimate of ||q − q_iso||. Deriving a uniform bound that holds for arbitrary learned query projections is non-trivial because the remainder depends on the third- and higher-order derivatives of log Z, which are themselves data-dependent. In the revised version we will add an empirical section that reports the distribution of ||q − q_iso|| (in the appropriate norm) measured on held-out sequences from the training distribution for both GLA and Gated DeltaNet backbones. We will also include a short discussion of how the observed distances relate to the scale at which the quadratic term dominates, thereby providing concrete support for the claim that the contraction preferentially damps high-density directions. This addition does not alter the core derivation but directly addresses the load-bearing assumption.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (derivation of the CCQ operator): the second-order Taylor expansion of log Z(q) is performed at the isotropic point q_iso; the resulting Hessian is identified with the key covariance Σ. No bound on the remainder term or estimate of typical ||q − q_iso|| for the learned query projections is supplied. Because the contraction operator is applied directly to arbitrary queries, the local nature of the approximation is load-bearing for the central claim that the operator preferentially suppresses high-density directions."}],"tokens_in":1393,"tokens_out":380,"duration_ms":15521,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one or two things your colleague should know are that the paper introduces Curvature-Conditioned Query (CCQ) as a cheap, composable addition to linear attention backbones like GLA and Gated DeltaNet, and that it derives a contraction operator from the Hessian of the log-partition at the isotropic point, which they maintain as the running key covariance.\n\nWhat is new is the specific step of using that covariance to contract the query before the read, rather than altering the write side or the feature map. The paper does well in keeping the change modular and low-cost while targeting the dilution problem in linear attention on retrieval and long-context tasks.\n\nThe soft spot is the one flagged in the stress-test note. The second-order Taylor expansion is local around the uniform-attention point, and nothing in the abstract shows that typical learned queries stay close enough for the quadratic model to remain useful. Query norms are independent of the expansion point, the remainder term can grow, and there is no bound or norm statistic to confirm the contraction actually aligns with softmax geometry in practice. The reported gains on S-NIAH, length extrapolation to 20K, and LongBench are listed at a high level, but without ablations isolating CCQ or verification that the covariance term is doing the work, attribution stays unclear.\n\nThis is aimed at people already working on linear attention variants for efficient long-context modeling. A reader tuning those methods would get a concrete idea to try, even if the justification needs more support.\n\nI would send it for peer review. The construction is specific enough that referees can test whether the local approximation delivers measurable gains once the full experiments and controls are available.","headline":"CCQ is a modular read-time query contraction from key covariance via local Taylor expansion of the softmax partition, but the local nature of the approx is a real soft spot that needs checking against actual query distributions.","tokens_in":2418,"tokens_out":427,"would_cite":false,"duration_ms":29125,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Curvature from key covariance contracts the query to improve linear attention on long contexts.","keywords":["linear attention","curvature-conditioned query","key covariance","long-context retrieval","softmax log-partition","recurrent state","query contraction","in-context learning"],"falsifier":"If attaching CCQ to GLA or Gated DeltaNet produces no gain or a loss in S-NIAH retrieval accuracy on sequences longer than the training context, the utility of the curvature-derived contraction would be refuted.","tokens_in":2663,"feed_emoji":"📐","tokens_out":673,"duration_ms":29856,"temperature":0.7,"pith_summary":"Linear attention maintains a recurrent state for efficiency but reads every past key additively, diluting useful targets amid the bulk of stored vectors. The paper derives a read-time contraction from a second-order Taylor expansion of the softmax log-partition around the isotropic-attention point. This expansion produces a local quadratic model whose curvature equals the running key covariance, a quantity trackable with the same recurrent mechanism as the linear-attention state. The resulting linear operator contracts the query along high-density directions of memory before it reads the state. The mechanism, called Curvature-Conditioned Query, modifies only the read step and composes with existing linear-attention backbones such as GLA and Gated DeltaNet.","feed_headline":"Curvature contracts query to boost linear attention retrieval","feed_subtitle":"Key covariance operator focuses reads on high-density memory, lifting long-context performance at low cost","key_machinery":"Curvature-Conditioned Query (CCQ): the linear operator obtained from the Hessian of the softmax log-partition at the isotropic point, which equals the key covariance and contracts the query along high-density memory directions before interaction with the recurrent state.","core_discovery":"The paper establishes that the second-order Taylor expansion of the softmax log-partition at the isotropic-attention point yields a local quadratic model whose curvature coincides with the running key covariance, and that the associated linear operator contracts the query along the high-density directions of memory before reading the recurrent state, supplying a cheap read-time filter that improves in-context retrieval and long-context performance.","pith_inferences":["The covariance-tracking approach could supply analogous contractions for attention variants whose geometry differs from softmax.","Read-time conditioning may turn out to be more efficient than further write-side gating when scaling context length.","Measuring how contraction strength correlates with retrieval accuracy on controlled synthetic tasks would test the mechanism directly."],"forward_implications":["Attaching CCQ to GLA or Gated DeltaNet improves perplexity on long sequences.","CCQ raises zero-shot downstream accuracy and S-NIAH retrieval performance both inside and beyond the training context.","Length-extrapolation perplexity improves when context grows from 4K to 20K.","LongBench accuracy increases while adding only modest computational cost.","The read-side contraction works with any linear-attention write path."],"fun_headline_variants":["Curvature contracts query along key covariance","Key covariance operator conditions linear attention reads","Second-order Taylor yields curvature query contraction","Running covariance contracts queries in recurrent attention","Curvature conditioned query filters memory density reads"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The second-order Taylor expansion of the softmax log-partition at the isotropic point supplies a curvature that usefully identifies high-density key directions for query contraction.","fun_headline_variants_meta":{"raw":{"variants":["Curvature contracts query along key covariance","Key covariance operator conditions linear attention reads","Second-order Taylor yields curvature query contraction","Running covariance contracts queries in recurrent attention","Curvature conditioned query filters memory density reads"]},"model":"grok-4.3","cost_usd":0.00362,"raw_usage":{"total_tokens":1894,"prompt_tokens":677,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":36199500,"prompt_tokens_details":{"text_tokens":677,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1157,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":677,"tokens_out":60,"duration_ms":8464,"temperature":1.0,"reasoning_tokens":1157,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T17:21:14.480731+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If attaching CCQ to GLA or Gated DeltaNet produces no gain or a loss in S-NIAH retrieval accuracy on sequences longer than the training context, the utility of the curvature-derived contraction would be refuted.","supporting_citations":[],"review_version":1}