Pith. sign in

REVIEW 2 major objections 3 minor 1 cited by

HadaCore: Tensor Core Accelerated Hadamard Transform Kernel

T0 review · 2 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HadaCore turns the Fast Walsh-Hadamard Transform into Tensor Core matmuls and runs up to 3.6x faster while preserving FP8-attention accuracy.

desk verdict Genuine tensor-core FWHT kernel, but headline speedups overstate the gain because the baseline lacks the in-place optimization the authors themselves identify. read the letter →

arxiv 2412.08832 v1 pith:SUJKLBRR submitted 2024-12-12 cs.DC cs.AI

Krish Agarwal , Rishi Astra , Adnan Hoque , Mudhakar Srivatsa , Raghu Ganti , Less Wright , Sijia Chen This is my paper · ORCID
classification cs.DCcs.AI
keywords HadamardtransformTensorCoresCUDAkernelLLMquantizationFP8attentioninferenceaccelerationWalsh-Hadamardin-placerotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HadaCore is a CUDA kernel that rewrites the Fast Walsh-Hadamard Transform so that the base case is a 16-by-16 matrix multiply executed on GPU Tensor Cores rather than the usual pairwise butterfly. The paper claims this hardware-aware decomposition keeps the same asymptotic cost while cutting synchronization and data movement enough to run 1.1-1.4x faster on average on A100 and 1.0-1.3x faster on H100, with peak gains of 3.5x and 3.6x, against the existing optimized CUDA implementation. It also reports that the faster kernel does not degrade quantization quality: on Llama-3.1 8B with FP8 attention, HadaCore reaches 65.09 average MMLU accuracy versus 65.45 for the existing kernel and 64.40 with no rotation. The practical stakes are that Hadamard rotations are the standard cheap way to remove activation outliers before low-precision LLM inference, so a faster drop-in transform lowers the overhead of QuaRot- and SpinQuant-style pipelines. The paper additionally identifies an in-place rotation trick that reduces L2-cache pressure for large tensors and attributes part of the largest-size gains to it.

What carries the argument

The central mechanism is the 16-by-16 Hadamard multiply as the base case, executed with the Tensor Core `mma` instruction: two 16x16-by-16x8 matrix operations combine into one 16x16-by-16x16 multiply, and the fragment lives in registers. The governing identity is the Kronecker structure of Sylvester's Hadamard construction: a 2n-size transform can be obtained by applying an n-size transform, permuting elements, and applying it again, which in this kernel is realized by transposing a 16-by-16 chunk so that two 16-size base-case applications separated by the transpose produce a 256-size transform. Non-power-of-16 sizes are handled by tiling a smaller Hadamard along the diagonal of the final 16-by-16 matrix. This machinery does the work of turning four butterfly levels of the textbook algorithm into one Tensor Core matmul, which is why the reported speedups come from reduced synchronization and data exchange rather than reduced arithmetic.

What would settle it

Apply the appendix's one-line change (write the Hadamard result into the input tensor instead of a freshly allocated output) to the baseline kernel, then rerun the 8M- and 16M-element FP16 benchmarks on A100 and H100; if HadaCore's speedup over this matched baseline is near 1.0 at those sizes, the largest reported gains come from cache behavior rather than the Tensor Core base case.

Watch

Extended reading notes

Core claim

The discovery is that the FWHT's recursive stages can be reorganized so that the base case is one 16-by-16 Walsh-Hadamard multiply executed by two Tensor Core `mma` operations, with the data held in registers. Applying that base case to 16-by-16 chunks, transposing, and applying it again produces a 256-element transform, and larger sizes are reached by chunking, a threadblock-level transpose, and another round of base cases. The paper shows this reorganization costs at least twice as many floating-point operations as the textbook algorithm ($4mn\log_2(n)$ versus $2mn\log_2(n)$) yet is faster on A100 and H100 because Tensor Cores give roughly 8x the throughput and because the kernel needs fewer synchronization barriers and less data shuffling. The same kernel in FP16 and BF16 stays numerically accurate, and an end-to-end Llama-3.1 8B run with FP8 attention gives 65.09 average MMLU accuracy with HadaCore rotations versus 65.45 with the baseline kernel and 64.40 without rotations. The claim is thus a better hardware mapping of the same algorithm, not a cheaper algorithm.

Load-bearing premise

The reported speedups assume the published baseline kernel is the correct comparison; the appendix shows the baseline can be made substantially faster with a one-line in-place-output change, and part of HadaCore's biggest gains is attributed to that same change.

Editorial extensions

If this is right

  • HadaCore can replace the Hadamard rotation step in existing quantized-inference pipelines without retraining or changing the quantization scheme, because the MMLU run shows rotated FP8 attention accuracy is preserved.
  • The largest element counts (8M on A100, 16M on H100) benefit most, which the paper attributes to in-place rotation keeping source and destination within L2 cache instead of evicting each other.
  • BF16 users get the same speedup pattern as FP16, with only a small overhead from FP32 accumulation and conversion.
  • Hadamard sizes larger than 256 avoid cross-threadblock synchronization except for one shared-memory transpose, keeping the cost of the serial log-scale recursion small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's own appendix implies that a fair head-to-head at large element counts should use the in-place variant of the baseline; with that one-line change the reported average speedup at 8M and 16M elements is likely to shrink, though the Tensor Core advantage at other sizes may remain.
  • Editorial inference: the 16-by-16 base-case mapping should transfer to other architectures with matrix-multiply-accumulate hardware and to other Kronecker-structured orthogonal transforms, since nothing in the argument is specific to Nvidia's `mma` instruction except the register layout.
  • Editorial inference: kernel fusion is the natural next step, because HadaCore already leaves data in registers and uses FP32 accumulation for BF16, so a fused Hadamard-plus-quantization kernel could avoid the global-memory round trip identified as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. HadaCore is a CUDA kernel that computes the Fast Walsh-Hadamard Transform using 16x16 Tensor Core matrix multiply instructions as the base case, with a recursive decomposition that handles sizes up to 2^15 and non-power-of-16 sizes. The paper reports runtime speedups over the Dao AI Lab fast-hadamard-transform library on A100 and H100 GPUs for FP16 and BF16, and reports an MMLU experiment on Llama-3.1 8B with FP8 attention showing that Hadamard rotations implemented by HadaCore give accuracy close to the Dao AI Lab kernel. The authors provide source code and unit tests.

Significance. The paper addresses a real practical bottleneck: Hadamard rotations in QuaRot/SpinQuant-style quantized LLM inference are often implemented as a dedicated FWHT kernel, and any speedup there reduces end-to-end overhead. The central construction is internally coherent: replacing the size-2 butterfly base case by a 16x16 Tensor Core mma increases arithmetic by about 2x while exposing roughly 8x higher FLOPS, and the claimed reduction in synchronization and data shuffling is plausible. The manuscript has real strengths: parameter-free complexity analysis, unit tests against explicit Hadamard multiplication, and public code. However, the headline speedup claim currently rests on a baseline that Appendix B itself shows can be improved by a one-line in-place change, and the MMLU comparison has no uncertainty quantification. These are fixable benchmarking issues rather than flaws in the algorithm.

major comments (2)
  1. [Section 4.1 and Appendix B, Figures 6/7/10/11] The reported speedups compare HadaCore against the unmodified Dao AI Lab kernel. Appendix B shows that replacing `at::Tensor out = torch::empty_like(x)` with `at::Tensor out = x` in `/csrc/fast_hadamard_transform.cpp` substantially improves the baseline's performance, and Section 4.1 attributes part of HadaCore's 8M/16M-element gains to in-place rotations. Because the main tables and figures use the unmodified kernel, the baseline at large tensor sizes pays for an output allocation and for source/destination L2 cache eviction that HadaCore avoids. The 1.1-1.4x (A100) and 1.0-1.3x (H100) averages, and especially the 3.5x/3.6x peaks, are therefore not cleanly attributable to the Tensor-Core design. The authors should rerun the full comparison against the in-place-patched baseline, or separately report an algorithmic speedup (same memory behavior) and a memory-layout speedup. If the patched baseline closes most of the gap, the central claim must be reframed.
  2. [Section 4.2 (MMLU table)] The paper reports single point estimates of 65.45 (Dao AI Lab kernel) and 65.09 (HadaCore) for FP8 attention with rotation, compared to 64.40 without rotation and 65.38 FP16 baseline. No error bars, number of seeds, or statistical comparison is given. The 0.36-point gap between HadaCore and the Dao kernel, and the 0.29-point gap to the FP16 baseline, are within typical 5-shot MMLU run-to-run variance, so the claim that HadaCore 'maintains comparable quantization error reduction' is not yet supported. Please report multiple runs with standard deviations or confidence intervals, and ideally the same random subsets for a paired comparison.
minor comments (3)
  1. [Appendix B] The sentence 'The fast-hadamard-transform library now outperforms memcpy (which they used as a lower bound in their benchmarks) in some cases' is surprising if memcpy was presented as a lower bound; clarify why a transform can be faster than a copy, or clarify what lower-bound claim is being made.
  2. [Appendix A, Figures 6/7/10/11] The tables and figures would benefit from a statement of measurement methodology (number of repetitions, warm-up, GPU clock locking, and run-to-run variance). At present the reader cannot distinguish real speedups from noise at the 1.0-1.1x level.
  3. [Section 4.2] The MMLU table is not numbered or captioned, and the evaluation setup (number of MMLU subjects, exact prompt template, batch size, whether the same random examples are used) is not described; please add these details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HadaCore's speedup and accuracy claims rest on external empirical comparisons and a parameter-free complexity argument, not on self-referential construction.

full rationale

The paper's central claim is an empirical kernel-speedup comparison against an external library (Dao AI Lab fast-hadamard-transform), supplemented by an end-to-end MMLU accuracy measurement with FP8 attention. The HadaCore algorithm follows the standard FWHT recursion and replaces the size-2 base case with a 16x16 Tensor Core mma base case; the complexity analysis is parameter-free, does not use the reported speedups as inputs, and does not depend on any self-citation. The only notable weakness is benchmarking fairness: Appendix B shows that a one-line change to the baseline, replacing the output tensor with an in-place tensor, materially improves the Dao AI Lab kernel, and the headline tables in Figures 6, 7, 10, and 11 compare against the unmodified baseline. That is a legitimate experimental-design concern, but it is not circularity: the comparison is a measurement against an external artifact, not an equation that reduces to its own inputs. The in-place optimization is disclosed and separately benchmarked in Appendix B. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is relabeled as a new derivation. Therefore no circular step is identified, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on a few standard mathematical identities and one hardware assumption about Tensor Core mma. The only tuned quantity is the threadblock configuration chosen per Hadamard size. No new entities are introduced.

free parameters (1)
  • Threadblock configuration (warps_per_block, num_chunks) = Empirically selected per size
    Section 4.1 states configurations are empirically selected for 8K, 16K, and 32K sizes; this tuning affects measured speedups but not algorithmic correctness.
assumptions (4)
  • standard math The recursive FWHT butterfly correctly computes the Walsh-Hadamard transform in O(mn log n) time.
    Section 2.2 relies on the classic algorithm from Fino and Algazi as a background result.
  • standard math A size-16^k Hadamard transform can be implemented as k stages of size-16 Hadamard transforms with transposes between stages.
    Section 3.4 uses the Kronecker product identity and Sylvester construction to justify the block decomposition.
  • domain assumption Tensor Core mma instructions can be invoked with data in registers to compute 16x16x16x8 matrix multiplications.
    Section 3 assumes Nvidia Tensor Core behavior and inline PTX; correctness depends on this hardware contract.
  • domain assumption The Dao AI lab fast-hadamard-transform library is the appropriate state-of-the-art baseline.
    Section 4.1 benchmarks only against this library; no independent benchmark supports the SOTA label.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HadaCore: Tensor Core Accelerated Hadamard Transform Kernel." pith.science (2026). https://pith.science/paper/SUJKLBRR

@misc{pith2026241208832,
  author       = {Pith},
  title        = {Pith review of: HadaCore: Tensor Core Accelerated Hadamard Transform Kernel},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SUJKLBRR}},
  note         = {Machine review of arXiv:2412.08832}
}
read the original abstract

We present HadaCore, a modified Fast Walsh-Hadamard Transform (FWHT) algorithm optimized for the Tensor Cores present in modern GPU hardware. HadaCore follows the recursive structure of the original FWHT algorithm, achieving the same asymptotic runtime complexity but leveraging a hardware-aware work decomposition that benefits from Tensor Core acceleration. This reduces bottlenecks from compute and data exchange. On Nvidia A100 and H100 GPUs, HadaCore achieves speedups of 1.1-1.4x and 1.0-1.3x, with a peak gain of 3.5x and 3.6x respectively, when compared to the existing state-of-the-art implementation of the original algorithm. We also show that when using FP16 or BF16, our implementation is numerically accurate, enabling comparable accuracy on MMLU benchmarks when used in an end-to-end Llama3 inference run with quantized (FP8) attention.

Figures

Figures reproduced from arXiv: 2412.08832 by the authors.

Figure 1
Figure 1. Transformer block showing online (red) and offline rotations (blue) in QuaRot. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. 1 × 256 vectors (rows) visualized for rotating by a size-256 Hadamard. The batches (number of rows) can be split among warps of the GPU. 3.2 Supporting Sizes Larger than 256 So far, we would only achieve up to a 256-size Hadamard with the above strategy because every contiguous chunk of h elements must be synced to achieve an h-sized Hadamard. Suppose that for a 1 × n row vector we are trying to do an n-size Hadamar… view at source ↗
Figure 3
Figure 3. The 1 × 256 vectors now transposed, allowing us to operate on the factor n 256 . Loading transposed data means that the loads are uncoalesced. We solve this in the following ways: • For sizes 512, 1024, and 2048, due to the Tensor Core register layout, we are able to simply load each chunk coalesced and then use warp-level shuffle operations to get each thread its own data. • For sizes 4096 and above, we can coalesc… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Graphs of runtime and speedup of HadaCore against the Dao AI Lab kernel, measured on an A100-PCIe. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Graphs of runtime and speedup of HadaCore against the Dao AI Lab kernel, measured on an H100-PCIe. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Tables of runtime (in µs) and speedup of HadaCore against the Dao AI Lab kernel, measured on an A100-PCIe. (a) Runtime (b) Speedup [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Tables of runtime (in µs) and speedup of HadaCore against the Dao AI Lab kernel, measured on an H100-PCIe. B In-Place Rotation One simple optimization is modifying the input tensor in-place. If our tensor is 16M elements of FP16 (e.g. 4096×4096), it will fit in ~32MB o…
Figure 8
Figure 8. Figure 8: Graph and table of speedup when the Dao AI Lab [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Graph and table of speedup when the Dao AI Lab [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Graph and table of speedup of HadaCore against the Dao AI Lab kernel for BF16, measured on an A100- [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Graph and table of speedup of HadaCore against the Dao AI Lab kernel for BF16, measured on an H100- [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Data Selection at Scale via Influence Distillation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Influence Distillation selects LLM fine-tuning data by approximating each sample's gradient influence on a target task via landmarks and JVP embeddings, matching or beating RDS+ accuracy at roughly one third the selec...

Reference graph

Works this paper leans on

11 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

    Saleh Ashkboos et al. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs . 2024. arXiv: 2404 . 00456 [cs.LG]. URL: https://arxiv.org/abs/2404.00456

    arXiv 2024

  2. [2]

    CUDA C++ Programming Guide

    Nvidia Corporation. CUDA C++ Programming Guide . https : / / docs . nvidia . com / cuda / cuda - c - programming-guide/index.html#warp-matrix-functions . 2024

    work page 2024

  3. [3]

    Fast Hadamard Transform in CUDA, with a PyTorch interface

    Tri Dao. Fast Hadamard Transform in CUDA, with a PyTorch interface . https : / / github . com / Dao - AILab/fast-hadamard-transform . 2024

    work page 2024

  4. [4]

    Unified Matrix Treatment of the Fast Walsh-Hadamard Transform

    Fino and Algazi. “Unified Matrix Treatment of the Fast Walsh-Hadamard Transform”. In: IEEE Transactions on Computers C-25.11 (1976), pp. 1142–1146. DOI: 10.1109/TC.1976.1674569

    work page arXiv 1976

  5. [5]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks et al. Measuring Massive Multitask Language Understanding . 2021. arXiv: 2009 . 03300 [cs.CY]. URL: https://arxiv.org/abs/2009.03300

    arXiv 2021

  6. [6]

    SpinQuant: LLM quantization with learned rotations

    Zechun Liu et al. SpinQuant: LLM quantization with learned rotations. 2024. arXiv: 2405.16406 [cs.LG] . URL: https://arxiv.org/abs/2405.16406

    arXiv 2024

  7. [7]

    FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

    Jay Shah et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. 2024. arXiv: 2407.08608 [cs.LG]. URL: https://arxiv.org/abs/2407.08608

    arXiv 2024

  8. [8]

    James Sylvester. “LX. Thoughts on inverse orthogonal matrices, simultaneous signsuccessions, and tessel- lated pavements in two or more colours, with applications to Newton’s rule, ornamental tile-work, and the theory of numbers”. In: Philosophical Magazine Series 1 34 (1867), pp. 461–475. URL: https : / / api . semanticscholar.org/CorpusID:118420043

    work page

Show all 11 references
  1. [9]

    CUDA C++ Programming Guide

    Philippe Tillet. CUDA C++ Programming Guide. https://openai.com/index/triton/. 2021

    2021

  2. [10]

    QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

    Albert Tseng et al. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

  3. [2024]

    URL: https://arxiv.org/abs/2402.04396

    arXiv: 2402.04396 [cs.LG]. URL: https://arxiv.org/abs/2402.04396. 7 PRIME AI paper A Graphs and Tables A.1 Graphs Figure 4: Graphs of runtime and speedup of HadaCore against the Dao AI Lab kernel, measured on an A100-PCIe. Each series represents a different rotation size, and ...

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.