{"id":"2c502747-098b-4afc-ab97-54af6ce870ca","arxiv_id":"2606.10944","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Express converts non-causal attention approximations to causal versions, achieving log^{3/2}(n)/s error with O(s) memory and O(s^2 log^2(n)) overhead when combined with Thinformer.","lead":"The paper introduces Express, a tool to convert non-causal attention approximations into causal ones while preserving approximation guarantees. When paired with Thinformer, it claims better error bounds for long sequences with low memory use, plus speedups and fixes for four language modeling bottlenecks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Whether the Express conversion preserves non-causal guarantees exactly for causal attention without extra assumptions or error terms","rationale":"The reader's weakest_assumption directly identifies the conversion step as the load-bearing point, which aligns with the structure of the strongest claim. No other internal inconsistency is visible from the abstract-level description of the result; the concern is therefore the same one already flagged.","tokens_in":1633,"tokens_out":347,"duration_ms":13998,"concrete_test":"Locate the section or theorem that defines the Express conversion and proves the error bound; recompute or re-derive the causal error bound from the non-causal Thinformer guarantee using only the steps given in that section. If any additive term appears or if an extra assumption on the attention matrix is invoked, the headline claim does not hold at the stated tightness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Express converting a non-causal approximation (Thinformer) into a causal one while keeping the exact same approximation error bound. This requires that the conversion procedure maps the non-causal construction to a causal one without introducing additional approximation error, without extra assumptions on the attention matrix entries, and without changing the O(s) memory or O(s^2 log^2(n)) overhead. If the conversion step (e.g., any masking, reordering, or recompression to enforce causality) adds even a lower-order term or requires the matrix to satisfy extra properties (such as uniform boundedness away from the diagonal or specific decay), the stated log^{3/2}(n)/s guarantee for causal attention would not follow directly from the non-causal result.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces Express, a conversion tool that transforms non-causal attention approximations into causal ones while preserving the original approximation guarantees. When paired with the Thinformer approximation, Express yields a causal attention guarantee of log^{3/2}(n)/s error using O(s) memory and O(s^2 log^2(n)) compression overhead for sequence length n. The work also provides an I/O-aware Triton implementation demonstrating speedups over FlashAttention 2 and applies the method to address bottlenecks in long-context prefill, KV cache compression, and memory/compute-constrained decoding.","tokens_in":1790,"tokens_out":356,"duration_ms":12352,"significance":"If the conversion procedure indeed maps non-causal guarantees to causal attention without additional error terms or assumptions, the result would strengthen the best-known theoretical bounds for causal attention and enable more efficient long-context language modeling. The combination of improved bounds, low memory overhead, and a practical implementation would be a notable contribution to efficient transformer research.","major_comments":[{"comment":"Abstract: The central claim that Express converts a non-causal approximation (Thinformer) into a causal one while delivering exactly the same log^{3/2}(n)/s error bound (with no extra terms) is load-bearing for the stated improvement over prior causal attention results. The skeptic concern that the conversion step may introduce lower-order error or require additional matrix assumptions (e.g., decay properties away from the diagonal) is not resolved by the abstract alone; without the explicit construction and error analysis, the bound does not follow directly from the non-causal case.","section":"Abstract"}],"minor_comments":[],"recommendation":"uncertain","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful review. We address the single major comment below.","responses":[{"response":"The manuscript provides the explicit construction of Express together with the full error analysis in Sections 3 and 4. The conversion maps any non-causal approximation to a causal one while preserving the original error bound exactly, without introducing lower-order terms. The argument relies only on the standard assumptions already used for the non-causal Thinformer guarantee and does not require additional decay properties away from the diagonal. Consequently the stated causal bound follows directly from the non-causal result via the given mapping.","revision_made":"no","referee_comment":"[Abstract] Abstract: The central claim that Express converts a non-causal approximation (Thinformer) into a causal one while delivering exactly the same log^{3/2}(n)/s error bound (with no extra terms) is load-bearing for the stated improvement over prior causal attention results. The skeptic concern that the conversion step may introduce lower-order error or require additional matrix assumptions (e.g., decay properties away from the diagonal) is not resolved by the abstract alone; without the explicit construction and error analysis, the bound does not follow directly from the non-causal case."}],"tokens_in":1267,"tokens_out":253,"duration_ms":21470,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper introduces Express as a conversion tool that takes a non-causal attention approximation and produces a causal version with the same guarantees. When applied to Thinformer it reports log^{3/2}(n)/s error, O(s) memory, and O(s^2 log^2(n)) overhead for causal attention on length-n sequences. It also supplies a Triton implementation and shows speedups over FlashAttention 2 while targeting four concrete bottlenecks: long-context prefill, KV cache compression, memory-limited decoding, and compute-limited decoding.\n\nThe practical framing is the strongest part. The authors identify real resource limits in current pipelines and tie the new bounds directly to those limits. The implementation detail and reported speedups give something concrete to test even before the theory is fully digested.\n\nThe soft spot is the conversion itself. The abstract states that Express maps the non-causal construction to a causal one without changing the approximation error, memory, or overhead. That claim is load-bearing. If the conversion requires extra assumptions on the attention matrix, introduces a lower-order error term, or changes the compression cost in practice, the stated improvement over prior causal bounds does not follow directly. The paper will stand or fall on whether the construction and proof make this preservation explicit and assumption-free.\n\nThis work is aimed at researchers and engineers who build or optimize long-context transformers and care about attention approximations. Readers who already follow Thinformer-style results will find the most immediate value.\n\nIt deserves a serious referee to examine the conversion math and the experimental verification of the claimed bounds.","headline":"Express claims an exact conversion from non-causal to causal attention approximations that preserves error bounds, but that step is the part that needs the closest check.","tokens_in":2253,"tokens_out":393,"would_cite":false,"duration_ms":14368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Express converts non-causal attention approximations into causal ones while preserving their error guarantees.","keywords":["Express","causal attention","attention approximation","Thinformer","language modeling","long-context","KV cache compression","Triton implementation"],"falsifier":"An explicit attention matrix and sequence where the causal approximation produced by Express exhibits strictly larger error than the original non-causal approximation.","tokens_in":2543,"feed_emoji":"🔄","tokens_out":726,"duration_ms":17515,"temperature":0.7,"pith_summary":"The paper introduces Express as a conversion procedure that takes any non-causal attention approximation and produces a causal version with identical guarantees. Causal attention is required for standard autoregressive language modeling, yet many strong approximation techniques were previously limited to non-causal settings. By matching the guarantees exactly, Express makes the best non-causal methods available for causal use. When paired with the Thinformer approximation it achieves an error bound of log to the three-halves of n divided by s, using only linear memory in the parameter s and quadratic-logarithmic compression cost. The authors also supply a Triton implementation that yields speedups over FlashAttention 2 and removes four specific resource limits in long-context language modeling pipelines.","feed_headline":"Express converts non-causal attention approximations to causal ones","feed_subtitle":"It preserves the original error guarantees and yields log to the 3/2 of n over s causal error with O(s) memory when combined with Thinformer","key_machinery":"Express, the conversion procedure from non-causal to causal attention approximation that preserves the original approximation guarantees exactly.","core_discovery":"Express is a conversion tool that takes a non-causal attention approximation and produces a causal version with exactly the same approximation guarantees. When combined with the state-of-the-art Thinformer approximation, Express improves upon the best known causal attention guarantees, delivering log to the three-halves of n over s approximation error with only O(s) memory and O(s squared log squared of n) compression overhead for a sequence of length n. The method is realized in an efficient I/O-aware Triton implementation that demonstrates substantial speedups over FlashAttention 2 and removes four resource bottlenecks in the language modeling pipeline: long-context prefill, KV cache compr","pith_inferences":["Future non-causal approximation improvements can transfer directly to causal language models via the same conversion.","Separate lines of research on causal-only approximations may become less necessary.","The memory and overhead scaling could support longer context windows under fixed hardware budgets."],"forward_implications":["Causal attention can now achieve the same approximation quality previously available only in non-causal settings.","The error bound improves to log to the three-halves of n over s with linear memory in s.","Compression overhead remains O(s squared log squared of n) for sequence length n.","An efficient Triton implementation produces speedups relative to FlashAttention 2.","Four concrete bottlenecks in long-context language modeling are removed: prefill, KV cache compression, memory-constrained decoding, and compute-constrained decoding."],"fun_headline_variants":["Express converts non-causal attention to causal with matching guarantees","Express improves causal guarantees when paired with Thinformer","Express provides log 3/2 n/s causal error with O s memory","Express converts approximations to enable better causal attention"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The conversion procedure from non-causal to causal approximation preserves the original approximation guarantees exactly, without requiring additional assumptions on the attention matrix or sequence statistics.","fun_headline_variants_meta":{"raw":{"variants":["Express converts non-causal attention to causal with matching guarantees","Express improves causal guarantees when paired with Thinformer","Express provides log 3/2 n/s causal error with O s memory","Express converts approximations to enable better causal attention"]},"model":"grok-4.3","cost_usd":0.004412,"raw_usage":{"total_tokens":2186,"prompt_tokens":627,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":44124500,"prompt_tokens_details":{"text_tokens":627,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1494,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":627,"tokens_out":65,"duration_ms":8282,"temperature":1.0,"reasoning_tokens":1494,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T13:43:26.572004+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An explicit attention matrix and sequence where the causal approximation produced by Express exhibits strictly larger error than the original non-causal approximation.","supporting_citations":[],"review_version":1}