MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

Gaetan Hains; Lenore Mulin

REVIEW 4 major objections 4 minor 11 references

MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

T0 review · 4 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper derives a decode-time attention that skips the K^T buffer and the n×n score matrix, reading K and V once each and claiming the per-token DRAM traffic is provably minimal.

desk verdict The algebra and kernel check out, but the 'proven memory-minimal' claim is borrowed from a companion paper that isn't here; as a standalone paper it's a standard optimization restated in MoA notation. read the letter →

arxiv 2607.19456 v1 pith:IROCJP2G submitted 2026-07-21 cs.LG cs.AI

Lenore Mulin , Gaetan Hains This is my paper · ORCID

classification cs.LGcs.AI

keywords MathematicsofArraysdecodeattentionKVcacheGQA/MQADRAMtrafficOpenACCkernelDNF/ONFmemoryoptimality

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At each autoregressive decoding step, standard attention forms a transposed view of the key cache and an n×n score matrix, costing O(n^2) memory traffic. This paper derives an algebraic reformulation, using the Mathematics of Arrays, in which the current query contracts directly with K stored in row-major order and the output is a weighted sum against V, so K^T never materializes and the score/softmax vectors stay rank-1. The resulting per-step DRAM traffic is (d_k + n d_k + n d_v + d_v)×4 bytes, which the authors assert is information-theoretically minimal by a Storage Theorem proved in a companion paper. The same derivation yields a C/OpenACC kernel with hardware-coalesced access, an O(d_k+d_v) KV-cache append, and GQA/MQA variants whose KV traffic drops by h_q/h_kv; all are verified against PyTorch's scaled dot product attention to within float32 round-off.

What carries the argument

The carrying mechanism is the Mathematics of Arrays ψ/Ω-reduction: the Omega inner product ⃗q (+.×Ω⟨1,2⟩) K replaces the matrix product QK^T by a vector–matrix product that contracts the query with K's rows in memory order, and the matching contraction ⃗a (+.×Ω⟨1,2⟩) V produces the output. The ψ-selection ⟨g_kv⟩ψK maps each query head to a KV slab without copying, and the γ offset arithmetic fixes row-major layout to prove hardware-coalescing in the GPU kernel. The minimality conclusion is carried by the companion paper's Storage Theorem 2.7, which the present text does not reproduce.

What would settle it

Run the supplied OpenACC kernel (or the Python decode) on a GPU with hardware counters that report bytes moved from DRAM for n=4096, d_k=d_v=64, and compare with (64 + 4096·64 + 4096·64 + 64)×4 B; if measured traffic exceeds this by more than the cache-line/coalescing overhead, the minimality claim fails. Alternatively, exhibit an exact attention decoding routine that reads less than this many bytes from DRAM per step (for example by keeping K and V in on-chip memory across consecutive steps) while producing bit-identical outputs—that would falsify the information-theoretic bound as stated.

Watch

Extended reading notes

Core claim

The central claim is that the decode phase of transformer attention can be written as two rank-1 contractions—score vector s = scale(K @ q) and output out = a @ V—and that this form, called the decode DNF, eliminates both the K^T buffer and the n×n score matrix. The ψ-reduction reads K row by row in its natural layout, so every element entering the computation is read once and no intermediate array is stored in DRAM. The authors invoke Storage Theorem 2.7 of the companion paper to claim this is the information-theoretic minimum DRAM traffic, and numerically verify the formula (to error ≤2×10^-7) against PyTorch SDPA for n from 4 to 4096. The same DNF, with query-row index fixed, is shown to

Load-bearing premise

The claim that the decode traffic is information-theoretically minimal rests on Storage Theorem 2.7 of the companion paper, which is cited but not included or proved here; if that theorem does not in fact establish minimal DRAM traffic for exactly this access pattern, the 'proven minimal' assertion collapses.

Editorial extensions

If this is right

A standard single-query decode becomes O(n(d_k+d_v)) byte traffic instead of O(n^2) score-matrix traffic; at n=4096, d=64, that is ~200 MB per step with no n×n materialization.
The KV-cache append is O(d_k+d_v) per step, and total decode traffic over T tokens is O(T^2(d_k+d_v)) with the smallest proven constant, so long-generation inference saves bandwidth quadratically in T.
GQA and MQA fall out of the same DNF by selecting KV slabs; KV DRAM traffic scales with h_kv rather than h_q, a factor h_q/h_kv reduction without materializing the head broadcast.
The C/OpenACC kernel with ONF stride arithmetic achieves exactly the same floating-point result as the sequential reference (∥err∥∞=0), so it can replace existing decode kernels without numerical drift.
Because the derivation is in normal forms, the same decode DNF ports to CPU, GPU, or other targets without re-deriving the memory layout.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the Storage Theorem of the companion paper is accepted, the minimality claim is a bound on DRAM traffic under a synchronous one-pass model; a decoder that keeps the KV cache resident in on-chip memory across steps, or recomputes softmax denominators from cached scores (as FlashAttention-style tiling does), could in principle reduce per-step DRAM traffic below this bound, so the 'information-the
The same ψ-reduction pattern should apply to other attention variants—cross-attention, sliding-window, and linear-attention layers—where eliminating a transpose or broadcast tensor could reduce memory traffic without changing numerics.
The exact arithmetic match of the OpenACC kernel means the MoA transformation is numerically transparent; a natural extension would be to formally relate the gamma-offset coalescing criterion to hardware cache-line sizes, yielding a portable autotuner for arbitrary shapes.
The paper's verification covers n up to 4096 and fixed d; a stress test at long context (n=10^5 or more) would show whether the measured DRAM traffic stays within the claimed (d_k + n d_k + n d_v + d_v)×4 bound as the KV cache grows beyond on-chip capacity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the algebra is right, the kernel is plausible, and the paper is honest about where the big claim comes from — but that big claim is parked in a companion paper we can't see, and without it the paper is a standard optimization restated in a new notation.

What's actually new: very little that I can verify. Avoiding K^T by computing K@q is textbook FlashAttention/vLLM behavior. Pre-allocating the KV cache is standard. GQA/MQA head sharing is reference behavior. What might be new is the MoA formalization itself, and the claim that the Storage Theorem of [1] proves the traffic minimal. That theorem isn't here.

What it does well: the derive-the-identity-in-MoA sections are clear and correct. I checked the math by hand; s = scale(K@q) and a@V are correct. The numerical tables match float32 round-off expectations. The OpenACC kernel has a solid coalescing analysis — the gamma-stride difference argument is a reasonable way to explain why you want d as the vectorized axis. Listings look like they'd compile. So as a 'here's how to write a decode kernel with MoA' tutorial, it's a reasonable artifact.

Soft spots, in proportion: the central proposition (Decode memory minimality) is not proved in this paper. It is a citation to Theorem 2.7 of arXiv:2606.07713v1, by the same authors, which is not available to the reader. That makes the paper's differentiator unverifiable. Even if Theorem 2.7 is true, its model of computation matters: if it assumes a simple one-pass DRAM machine with no recomputation, then it doesn't preempt FlashAttention's recompute-based tradeoff. The paper also conflates 'memory usage' with 'DRAM traffic' in Proposition 1 — the formula is traffic, not footprint. The 'exact IEEE-754' check is a nice sanity test, but it tests arithmetic order, not memory minimality.

The GQA reduction is also stated as 'proven' via the same theorem. The underlying observation is correct — sharing a KV slab across query heads means you don't need to read it once per head — but again, the proof burden is deferred.

Final: I'd read this if you work on MoA or want a simple OpenACC baseline, but I wouldn't cite it as a source for memory optimality. It deserves a referee in the sense that a referee could check the companion paper and ask for a revision that either states the theorem or demotes the claim. As is, I'd lean desk-reject or major revision.

Referee Report

4 major / 4 minor

Summary. The paper applies the Mathematics of Arrays (MoA) framework to single-token decode attention. Starting from the forward-pass DNF of a companion paper, it fixes the query-row index and derives a three-step decode DNF in which the K^T buffer and n×n score matrix are eliminated algebraically. It then presents a C/OpenACC kernel with ONF stride arithmetic, a pre-allocated KV-cache append scheme, and GQA/MQA variants obtained by ψ-selection. The central claim is that the resulting DRAM traffic, M_dec^MoA = (d_k + n·d_k + n·d_v + d_v)×4 B, is 'proven minimal by Storage Theorem 2.7 of [1]'. The paper reports numerical verification against PyTorch SDPA, including exact IEEE-754 equality for the CPU kernel.

Significance. The algebraic core of the paper—that decode attention can be written as s = scale(K@q) and out = a@V, avoiding K^T materialization—is correct and standard. The numerical checks are a useful sanity check, and the OpenACC kernel is clearly presented. However, the paper's differentiator is the claim that the traffic is proven information-theoretically minimal, and that proof is entirely delegated to Theorem 2.7 of the companion paper [1], which is neither stated nor proved here. The GQA/MQA savings claim also depends on an unstated hardware/caching model. If the companion theorem and its applicability to the streaming decode setting were supplied, the paper would be a useful formal derivation; as submitted, its main advertised result is not self-contained and cannot be verified by the reader.

major comments (4)

[§2, Proposition 1] The central claim that M_dec^MoA is 'proven minimal by Storage Theorem 2.7 of [1]' is not verifiable from this manuscript. Theorem 2.7 is neither stated nor proved. The decode DNF is described only as 'eqs. (4)–(6) of [1] with i′ fixed', but the theorem's hypotheses—e.g., machine model, one-pass reading, absence of recomputation—are not given, and their transfer to the streaming decode setting (growing n, KV-cache appends, GQA slabs) is asserted rather than demonstrated. Since the minimality proof is the paper's principal claimed contribution, please include the theorem statement and the decode lower-bound argument, or explicitly weaken the claim to 'the ONF achieves this traffic' without the word 'proven'.
[§1.3, §5.2] The claims that reading the full KV cache once per step is 'unavoidable by any correct implementation' and that M_step(t) = O(t·d_k + t·d_v) is an information-theoretic minimum presuppose a synchronous, non-preemptible machine with no recomputation and no cache hierarchy. FlashAttention-style tiling and recomputation trade compute for DRAM traffic, so any lower bound must be relative to a precisely stated execution model. Without specifying that model and proving the bound within it, the statement is an assumption, not a theorem. Please state the model and prove the lower bound in that model, or qualify the claim.
[§6.2, Figure 3] The 'proven h_q/h_kv reduction' in KV traffic depends on each KV slab being read from DRAM once and then reused from shared memory by all h_q/h_kv query heads that share it. The provided Python implementation (Listing 7) actually reads K[g_kv] once per query head, so the reduction is an architecture-dependent caching effect rather than an algebraic consequence of ψ-selection alone. Storage Theorem 2.7 of [1] is cited but not shown to cover this reuse pattern. Please make the cache/reuse model explicit, or present the factor as a conditional improvement under a stated sharing assumption.
[§5.1–5.2] The comparison with 'standard PyTorch KV-cache' using torch.cat is misleading for the claimed contribution. Production KV-cache implementations typically pre-allocate a buffer and write each new key/value row at an offset, which already achieves the O(d_k + d_v) per-step append cost; the MoA # operator formalizes this same pattern. Proposition 2 is therefore correct as a traffic accounting but is not a new result or a MoA-specific information-theoretic minimum. Please compare against the standard pre-allocated baseline and avoid implying that torch.cat is the standard implementation.

minor comments (4)

[§2, Proposition 1] The proposition states 'memory usage' but the formula and surrounding text concern DRAM traffic. Please use consistent terminology: 'memory traffic' or 'DRAM traffic' rather than 'memory usage'.
[Figure 2] In the 'conventional' column, 'K^T still present' is ambiguous: the Python code uses a transposed view, not a materialized buffer. Clarify that the transpose is a view/layout operation, and that the MoA claim is about avoiding the separate materialized buffer.
[Tables 1–6] The verification tables do not report the random distribution or value ranges used to generate the inputs, beyond dimensions and seed(0) for Table 1. For reproducibility, specify the input distributions (e.g., uniform on [−1,1], normal, etc.) for every table.
[Listing 3] The atomic update in Pass 3 may serialize writes to out[d] if many gangs update the same d; the paper does not report performance measurements, so claims about 'maximal memory saturation' are not substantiated. Consider either adding a performance measurement or softening the wording.

Circularity Check

1 steps flagged · score 6.0 of 10

Decode algebra is correct and verifies against PyTorch, but the paper's central 'proven memory-minimal' claim is imported from the authors' own companion Theorem 2.7, which is neither stated nor proved here; the decode DNF is defined as a specialization of that same self-cited paper.

uniqueness imported from authors [§1.2 Contributions item 1; §2 'Proposition 1'; §7]
"Memory traffic:Mdec MoA = (dk +nd k +nd v +dv)×4 B, is then proven minimal by Storage Theorem 2.7 of [1]. ... This is equations (4)–(6) of [1] with the query-row index i′ fixed to the current token. ... In every case the memory bound is a theorem (Storage Theorem 2.7 of [1]) before code is written."

The paper's central differentiator—that the decode DNF is provably memory-minimal—is not derived in this paper. The minimality bound is asserted to be Storage Theorem 2.7 of [1], a companion paper by the same two authors, and the decode DNF is described as eqs. (4)–(6) of the same [1] with i′ fixed. Thus the load-bearing conclusion 'minimal' is equivalent to accepting the authors' prior unpublished theorem; no statement of the theorem, its execution model, or its proof is included, and the numerical checks only verify the floating-point arithmetic, not DRAM traffic. The paper therefore presents a self-citation chain as the proof of its strongest claim.

full rationale

The algebraic core of the paper is self-contained and sound: s = scale(K@q) is the standard matrix-vector form of q@K.T, softmax is applied to a rank-1 vector, and out = a@V; these are identities, not fitted or data-dependent predictions. The KV-cache append cost and the GQA h_q/h_kv traffic ratio are straightforward counting arguments and are not circular. The circularity, and the reason the score is not 0–2, is confined to the paper's signature claim: 'proven minimal' DRAM traffic. That claim is delegated to Storage Theorem 2.7 of [1] by the same authors, and the decode DNF itself is defined as a specialization of [1]'s equations. The theorem is never stated or proved in the present manuscript, and the quoted proposition merely restates the byte size of q,K,V,out as 'minimal'. Because the paper's differentiator over FlashAttention is exactly this algebraic minimality proof, and that proof is imported by self-citation from an unavailable companion, the central result is partially forced by the authors' own previous work. This is a load-bearing self-citation, not an independent derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper invents no new entities. The unfamiliar item is the MoA algebra itself (ψ, ρ, γ, #, Ω, DNF/ONF), but it is a pre-existing formal system from [3]. The main ledger items are the unprovided Storage Theorem and dimension-lifting machinery from companion [1], plus the modeling assumption that reading all cached rows once per token is unavoidable.

assumptions (4)

domain assumption Storage Theorem 2.7 of companion [1]: every array read once implies traffic is minimal for the decode DNF.
Invoked §1.2, §2, §5.2, §6.2 as the basis for the claimed memory bounds. The theorem is not included in this paper nor machine-checked here, so the claim is load-bearing but unverifiable from this text.
domain assumption Dimension lifting of Section 2.5 of [1] maps the abstract shape to GPU/OpenACC execution and preserves the coalescing and traffic properties.
Invoked §4.2 and §6.2. The kernel's coalescing proof depends on this mapping; the mapping is described but not formally proven in this paper.
domain assumption Reading all cached rows once per decode step is information-theoretically unavoidable ('every token must attend over all previous tokens').
Used in §5.2 and Figure 1 to claim the O(T^2) total and per-step O(t d_k) traffic are unavoidable. In practice, large-context systems use retrieval, chunking, or approximate attention; even exact attention may trade recomputation for traffic. This is an engineering-model assumption, not a proven lower bound.
standard math Numerical stability: subtracting max before exp does not change softmax value.
Standard softmax shift, §2 Step II. Not a concern.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel." pith.science (2026). https://pith.science/paper/IROCJP2G

@misc{pith2026260719456,
  author       = {Pith},
  title        = {Pith review of: MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IROCJP2G}},
  note         = {Machine review of arXiv:2607.19456}
}

abstract

We derive four memory-optimal inference artifacts for transformer attention using the Mathematics of Arrays (MoA), each following directly from the forward-pass Denotational Normal Form (DNF) of with the query-row index fixed to the current decode step. The artifacts are: (1)~a single-query decode DNF in which the $\psi$-reduction eliminates the $K^\top$ buffer algebraically, achieving $(d_k + nd_k+ nd_v+ d_v)\times4\,{B}$ Dynamic Random Access Memory (DRAM) traffic result numerically verified to $\|{err}\|_\leq2\times10^{-7}$; (2)~a C/OpenACC Graphics Processing Unit (GPU) kernel with Operational Normal Form (ONF) stride arithmetic and hardware-coalesced memory access, verified to $\|\mathrm{err}\|_\infty=0$ (exact IEEE-754 floating-point arithmetic); (3)~a multi-step KV-cache with $O(d_k+d_v)$ per-step append via MoA concatenation $\#$; and (4)~Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) derived via $\psi$-selection, achieving a proven $\frac {h_q} { h_{kv} }$ reduction in KV traffic. All programs are verified against PyTorch scaled_dot_product_attention.

Figures

Figures reproduced from arXiv: 2607.19456 by the authors.

**Figure 1.** Realistic nested loops: conventional vs. MoA across training and inference. Solid loops = outer; dashed = inner. Training runs ∼106 total batches once; inference runs 109 requests/day indefinitely, each generating 512 tokens across 96 layers. At n= 4096, L= 96: both conventional and MoA use ≈200 MB per decode step — but only MoA’s is proven minimal (Storage Thm. 2.7). FlashAttention achieves similar traffic via tili… view at source ↗

**Figure 2.** The MoA pipeline: Python → MoA → DNF → ONF → C/Fortran/OpenACC. Starting from conventional Python (q @ K.T), the MoA formulation expresses attention in array algebra. ψ-reduction (DNF) eliminates K⊤ and the softmax Jacobian algebraically. The γ stride map (ONF) fixes the row-major layout and proves coalescing. The resulting C/Fortran 90/OpenACC code is correct by construction (Storage Theorem 2.7). Algorithm 1 MoA d… view at source ↗

**Figure 3.** GQA/MQA via MoA ψ-selection. hq = 8 query heads are grouped into hkv = 2 KV groups. ψ-selection ⟨gkv⟩ψ K maps each query head to its KV slab directly, without materialising the ⟨hq, n, dk⟩ broadcast that repeat interleave would create. Traffic reduction: hq/hkv = 4× here; hq× for MQA (hkv = 1). Proven minimal by Storage Theorem 2.7. Query heads 0 and 1 both evaluate ⟨0⟩ ψ K; heads 2 and 3 both evaluate ⟨1⟩ ψ K. By t… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: MoA inference pipeline. All three compo [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 4 linked inside Pith

[1]

Attention at the Theoretical Minimum: A Mathematics of Ar- rays Framework for Memory-Optimal Trans- former Kernels,

L. Mullin and G. Hains, “Attention at the Theoretical Minimum: A Mathematics of Ar- rays Framework for Memory-Optimal Trans- former Kernels,”arXiv:2606.07713v1 [cs.LG], June 2026. 12

arXiv 2026
[2]

Memory-Optimal Transformer Training: Backward Pass and Fused Kernel via the Mathematics of Arrays,

L. Mullin and G. Hains, “Memory-Optimal Transformer Training: Backward Pass and Fused Kernel via the Mathematics of Arrays,” HAL-05659212, June 2026

2026
[3]

L. M. Mullin,A Mathematics of Arrays, Ph.D. dissertation, Syracuse University, 1988

1988
[4]

Im- proving the Performance of DGEMM with MoA and Cache-Blocking,

S. Thomas, L. Mullin, and K. ´Swirydowicz, “Im- proving the Performance of DGEMM with MoA and Cache-Blocking,” inProc. ACM ARRAY Workshop(PLDI), 2021.https://doi.org/10. 1145/1122445.1122456

arXiv 2021
[5]

From Array Algebra to Energy Effi- ciency on GPUs,

L. Mullin, “From Array Algebra to Energy Effi- ciency on GPUs,”arXiv:2306.11148, 2023

arXiv 2023
[6]

Attention is all you need,

A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Sys- tems (NeurIPS), 2017

2017
[7]

Fast Transformer Decoding: One Write-Head is All You Need,

N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,”arXiv:1911.02150, 2019

arXiv 1911
[8]

GQA: Training Generalized Multi-Query Transformer Models from Multi- Head Checkpoints,

J. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi- Head Checkpoints,” inProc. EMNLP, pp. 4895– 4901, Singapore, 2023

2023

Show all 11 references

[9]

FlashAttention: Fast and Memory- Efficient Exact Attention with IO-Awareness,

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. R´ e, “FlashAttention: Fast and Memory- Efficient Exact Attention with IO-Awareness,” inAdvances in Neural Information Processing Systems (NeurIPS), 2022

2022
[10]

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,

T. Dao, “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” inProc. ICLR, 2024.arXiv:2307.08691

2024 arXiv
[11]

Flash-Decoding for Long- Context Inference,

T. Dao and D. Y. Fu, “Flash-Decoding for Long- Context Inference,”https://crfm.stanford. edu/2023/10/12/flash-decoding.html, 2023. 13

2023

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Attention at the Theoretical Minimum: A Mathematics of Ar- rays Framework for Memory-Optimal Trans- former Kernels,

L. Mullin and G. Hains, “Attention at the Theoretical Minimum: A Mathematics of Ar- rays Framework for Memory-Optimal Trans- former Kernels,”arXiv:2606.07713v1 [cs.LG], June 2026. 12

arXiv 2026

[2] [2]

Memory-Optimal Transformer Training: Backward Pass and Fused Kernel via the Mathematics of Arrays,

L. Mullin and G. Hains, “Memory-Optimal Transformer Training: Backward Pass and Fused Kernel via the Mathematics of Arrays,” HAL-05659212, June 2026

2026

[3] [3]

L. M. Mullin,A Mathematics of Arrays, Ph.D. dissertation, Syracuse University, 1988

1988

[4] [4]

Im- proving the Performance of DGEMM with MoA and Cache-Blocking,

S. Thomas, L. Mullin, and K. ´Swirydowicz, “Im- proving the Performance of DGEMM with MoA and Cache-Blocking,” inProc. ACM ARRAY Workshop(PLDI), 2021.https://doi.org/10. 1145/1122445.1122456

arXiv 2021

[5] [5]

From Array Algebra to Energy Effi- ciency on GPUs,

L. Mullin, “From Array Algebra to Energy Effi- ciency on GPUs,”arXiv:2306.11148, 2023

arXiv 2023

[6] [6]

Attention is all you need,

A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Sys- tems (NeurIPS), 2017

2017

[7] [7]

Fast Transformer Decoding: One Write-Head is All You Need,

N. Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need,”arXiv:1911.02150, 2019

arXiv 1911

[8] [8]

GQA: Training Generalized Multi-Query Transformer Models from Multi- Head Checkpoints,

J. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi- Head Checkpoints,” inProc. EMNLP, pp. 4895– 4901, Singapore, 2023

2023