Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Deterministic attention backward passes lose up to 37.9% throughput to a scheduling conflict; DASH recovers much of it by reordering GPU tile work.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:41 UTC pith:ZHKYX5CK

load-bearing objection Useful empirics and a sensible DAG framing, but the causal 'optimal' schedule is not optimal under the paper's own model — the formula contradicts its own contiguity constraint. the 3 major comments →

arxiv 2601.21824 v1 pith:ZHKYX5CK submitted 2026-01-29 cs.LG cs.DC

DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training

Xinwei Qiang , Hongmin Chen , Shixuan Sun , Jingwen Leng , Xin Liu , Minyi Guo This is my paper
classification cs.LG cs.DC
keywords deterministic trainingattention backward passGPU schedulingDAG critical pathFlashAttention-3reproducibilityLLM trainingkernel optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

To guarantee bitwise-identical results, deterministic attention kernels force gradient accumulations into a fixed serial order across the GPU's streaming multiprocessors; FlashAttention-3's deterministic backward pass can therefore run up to 37.9% slower than its non-deterministic version. The paper argues this penalty is not inherent to serialization but follows from scheduling tile computation and reduction phases in isolation, leaving SMs idle during pipeline bubbles. DASH models the backward pass as a DAG of compute-plus-reduction tasks, proves a condition (Lemma 1) for adding dependency edges without lengthening the critical path, and designs two schedules—Descending Q-Tile Iteration and the theoretically optimal Shift/Symmetric Shift Scheduling—that balance work and keep reductions conflict-free. On H800 GPUs, DASH raises deterministic backward-pass throughput by up to 1.28× over the FlashAttention-3 deterministic baseline, bringing it close to non-deterministic speed, with 2–10% end-to-end transformer-block gains on LLaMA, Qwen, Mistral, and vision/diffusion models. The authors also report that their deterministic mode yields exactly zero run-to-run gradient deviation in their tests, versus O(10^-4) for non-deterministic.

Core claim

The paper's central claim is that the deterministic attention backward pass can be made nearly as fast as the non-deterministic one, because the loss comes from a mismatch between two ordering constraints: each KV tile's local dK/dV accumulation must stay on one SM, while the global dQ reductions must proceed in a fixed order. DASH formalizes this as a DAG scheduling problem: each tile is a compute-then-reduce chain on one SM, zero-weight edges encode the deterministic accumulation order, and the objective is to minimize the critical path. Lemma 1 states that an added dependency (u,v) preserves the critical path iff depth(u) ≤ depth(v), which effectively forbids parallel execution of tasks t

What carries the argument

The central object is the DAG of tile phases: each (KV_i, Q_j) task is a linear chain of a compute phase C(i,j) of cost c followed by a reduction phase R(i,j) of cost r, with zero-weight edges inserted to enforce the deterministic global reduction order. The paper's Lemma 1 gives the depth condition: such an added edge (u,v) does not lengthen the critical path iff depth(u) ≤ depth(v); this is what makes the schedule construction principled. Shift Scheduling assigns KV tiles to SMs in a cyclic order (SM i visits KV i, i+1, ..., n-1, 0, ..., i-1), staggering reductions so no two SMs reduce the same dQ block concurrently; for causal masks, Symmetric Shift Scheduling pairs the longest and shorte

Load-bearing premise

The paper's model assumes every compute and reduction tile has the same cost and that the inter-SM synchronization signals used to enforce the serialized reduction order cost nothing; its own experiments show that at sequence length 16,384, remote L2 synchronization latency (200–500+ cycles) makes Shift Scheduling slower than the baseline.

What would settle it

Measure the same kernels on a GPU with large register files and low, uniform L2 latency, or with a modified kernel that avoids register spilling; if Shift/Symmetric Shift Scheduling still fails to reach the predicted lower bound (T_full^opt = m·n·(c+r)) and to beat the Descending heuristic where it is predicted to, the zero-cost-synchronization and uniform-tile-cost assumptions are inconsistent with real hardware.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reproducible training need not be slow: if DASH's schedules hold up, deterministic attention backward passes can run within a small factor of their non-deterministic counterparts, removing a major cost objection to bitwise reproducibility in LLM training.
  • The DAG/critical-path formulation turns a kernel-tuning chore into an optimization problem that can be solved analytically, meaning the result extends to other tile sizes, head counts, and sequence lengths within the model's assumptions.
  • Because the paper's deterministic mode achieves zero run-to-run gradient deviation (versus O(10^-4) for the non-deterministic path), DASH makes exact reproduction of training runs possible in practice.
  • The two schedules are complementary: Shift Scheduling is optimal under the model, while the simpler Descending Iteration wins when registers are scarce, so a practical implementation can select per configuration.
  • End-to-end gains of 2–10% on common LLM transformers and ~4% on full-mask vision/diffusion models mean the scheduling fix matters outside microbenchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The depth-monotone scheduling principle is not attention-specific: any kernel that needs deterministic reduction order (split-K/stream-K GEMMs, distributed all-reduce, optimizer-state updates) can borrow the same DAG formulation to hide serialization behind balanced compute.
  • Extending the model with nonzero synchronization costs and heterogeneous tile costs would likely produce schedules that trade some balance for fewer remote-L2 dependencies—directly addressing the seqlen-16384 regression the paper reports.
  • The register-pressure limitation suggests an automatic kernel launcher could choose between the two schedules at runtime based on measured register usage and sequence length, rather than hard-coding one.
  • A sharper falsification of the theory would vary tile granularity and verify that the achieved throughput tracks the predicted lower bound T_full^opt = m·n·(c+r); deviations would pinpoint which of the model's simplifying assumptions (uniform costs, zero-cost edges) is violated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses the throughput penalty of deterministic (bitwise reproducible) attention backward passes in LLM training. It models the deterministic backward pass as a DAG scheduling problem in which per-KV-tile compute and reduction tasks must respect both data dependencies and a serialized, conflict-free reduction order. The authors propose two strategies: Descending Q-Tile Iteration, a heuristic that reverses the query-block traversal for causal masks, and Shift Scheduling / Symmetric Shift Scheduling, which they claim are theoretically optimal for full and causal masks within the DAG model. Experiments on H800 GPUs report up to 1.28× throughput improvement over the FlashAttention-3 deterministic baseline, with end-to-end transformer-block speedups around 2–10%. The paper is written transparently, includes open-source code, and explicitly acknowledges several places where the theoretical model diverges from hardware behavior.

Significance. If the scheduling constructions were rigorously established, the paper would make a useful contribution to reproducible LLM training: it identifies a concrete cause of the deterministic-mode slowdown, gives a principled modeling framework, and provides two practical kernel-level strategies with open-source implementations. The empirical evaluation is clean in its basic setup, and the authors deserve credit for openly discussing the configurations where their 'optimal' schedule is slower than the baseline (e.g., Section 4.2 at seqlen 16384). However, the central theoretical optimality claim for causal masks is not currently supported: the stated formula is not an achievable makespan under the paper's own contiguity constraint, and the workload-folding construction is not specified or proved. The paper also lacks error bars, making the magnitude of the empirical gains difficult to assess precisely. The contribution is promising, but the theoretical centerpiece needs substantial revision.

major comments (3)
  1. [§3.4 (causal optimality, T_causal^opt = m(n+1)(c+r)/2)] The claimed causal optimal makespan is internally inconsistent with the model's own contiguity constraint. For n=2, m=1, KV0 contributes to Q0 and Q1, so those two compute-reduction tasks must run sequentially on one SM; any schedule requires at least 2(c+r). The formula gives 1.5(c+r). More generally, the expression m(n+1)(c+r)/2 is the per-SM average total work per head, i.e., a load-balancing lower bound, not a critical path under unbroken per-KV chains. The 'workload folding' described in §3.4 is only prose: Algorithm 1 merely says 'assigned Q-tile schedule' without defining the folded schedule, and no proof is given that the folding preserves per-KV contiguity, deterministic reduction order, and the depth-monotone condition of Lemma 1. The authors must either provide a formal DAG-level construction with a makespan proof, or qualify the claim (for example, for sufficiently large or e
  2. [§3.4 and Lemma 1] Lemma 1 is proved only for n parallel, isomorphic chains with a single source and a single sink. The causal-mask DAG is not isomorphic: KV_i has n−i tasks, so the chains have different lengths. The paper jumps from Lemma 1 to the symmetric-shift construction via an 'algebraically equivalent' claim, but never states a theorem that maps the causal DAG to the paired/folded schedule or proves that the resulting schedule satisfies the lemma's hypotheses. This is a load-bearing gap: without it, the 'provably optimal' statement in the abstract and Section 3.4 is unsupported. I recommend adding a formal definition of the causal DAG, a precise construction of the symmetric shift schedule, and a proof of its makespan, or explicitly withdrawing the optimality claim for causal masks.
  3. [§4.2–4.3 (empirical evaluation)] The central empirical claim — up to 1.28× throughput improvement — is reported from what appears to be single-run measurements on one GPU model. No error bars, standard deviations, or number of repetitions are given. Given that the authors themselves report one configuration (seqlen 16384, full mask) where the proposed optimal schedule is slower than the baseline, variance information is needed to know whether the observed improvements are robust and to quantify the size of the regression. I would ask for multiple runs (at least 3–5) or a statement about run-to-run variability, especially for the headline speedup and for the causal head_dim=128 inversion.
minor comments (5)
  1. [Algorithm 1] The pseudocode marks '[DASH]' at the loop headers and synchronization points but does not define the assigned Q-tile schedule for Descending Q-Tile Iteration or Shift Scheduling. Since the code is open-sourced this is partly mitigated, but the paper should specify the schedule or cite the exact code artifact so the algorithm is self-contained.
  2. [Figures 6 and 7] The captions and table layouts in Figures 6 and 7 are difficult to parse. In particular, the meaning of the numbers in the 'Assigned SM ID' table and the 'Paired SM Alloc' table is not stated explicitly. Please add a legend or a formal definition of the assignment function.
  3. [§3 (model setup)] The statement 'without loss of generality, we assume that the number of KV tiles equals the number of SMs, denoted by n' needs qualification. Later, m heads are introduced and the schedules are evaluated per head. The interaction between m heads and n SMs — whether heads are processed sequentially, interleaved, or rotated across SMs — should be formalized, as it is central to the causal optimality expression.
  4. [Abstract and §3.4] The word 'optimal' should be qualified as 'optimal within the DAG model' in the abstract and conclusion. The paper already does this in places, but the abstract and Section 3.4 summary state optimality without the qualification, which overclaims given the known zero-cost-edge abstraction and the empirical counterexample at seqlen 16384.
  5. [§4.3] The register-pressure explanation for the head_dim=128 inversion would be more convincing with the actual Nsight Compute register and spill counts rather than a prose description. Please include the measured values.

Circularity Check

0 steps flagged

No significant circularity: DASH's schedules are derived from an explicit DAG model with symbolic costs and are validated independently on hardware.

full rationale

The paper's derivation chain is self-contained. Section 3.1 defines a DAG scheduling model with symbolic compute/reduction costs c and r, explicit zero-weight dependency edges, and a contiguity constraint; Sections 3.3-3.4 construct schedules (Descending Q-Tile Iteration, Shift/Symmetric Shift Scheduling) and evaluate their critical paths against the model's own lower bounds. No parameter is fitted to the measured throughput, so the 'optimality' claims are statements about the model rather than retrospective fits. The empirical 1.28x speedup is a measured benchmark outcome, not a number produced by the model, so there is no fitted-input-called-prediction pattern. The paper contains no load-bearing self-citation: references to FlashAttention-3, RingAttention, StripedAttention, etc. are external, and the cyclic-shift inspiration is explicitly acknowledged rather than imported as proof. The most substantial concerns are internal-consistency/limitation issues, not circularity: the paper itself states that the model 'assumes zero-cost dependency edges' and reports that Shift Scheduling degrades below baseline at seqlen 16384 due to remote L2 latency (Section 4.2); and the causal optimality formula in Section 3.4 relies on an underspecified 'workload folding' construction (Algorithm 1 marks only 'assigned Q-tile schedule' without detailing the folding) and appears to understate the per-SM makespan under the contiguity constraint. These are correctness/completeness concerns, not cases where an output equation is identical to its input by construction; they therefore do not constitute circularity under the specified rubric.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claims rest on a simplified analytical DAG model with uniform tile costs and zero-cost synchronization edges, plus the constraint that each KV tile is processed contiguously on a single SM. No free parameters are fitted to measured data; the model uses symbolic costs c and r. The zero-cost synchronization assumption is acknowledged by the authors to be violated in practice at high parallelism, which limits the generality of the theoretical optimality claims.

axioms (6)
  • standard math Floating-point addition is non-associative; deterministic mode requires a fixed serial accumulation order for dQ reductions
    Foundation of the problem: Section 1 and 2.1; non-associativity is a known property of IEEE-754 FP.
  • domain assumption All tile computation phases have equal cost c and all reduction phases equal cost r
    Section 3.1 'edge weights are constants'; used to derive critical path formulas and to argue workload balance.
  • domain assumption Number of KV tiles equals number of SMs n; other cases are handled by refining/aggregating attention heads
    Section 3 opening; WLOG claim but no concrete mechanism is specified for mismatched counts.
  • domain assumption Dependency edges for synchronization have zero cost
    Section 3.1/3.4; Lemma 1 adds zero-weight edges. Section 4.2 later acknowledges real L2 latency makes this false.
  • domain assumption All operations for a given KV tile must run contiguously on a single SM
    Section 3.1 'Optimization Constraint'; derives from register-resident dK/dV accumulation.
  • domain assumption The DAG abstraction captures the relevant scheduling constraints
    Section 3.1; the paper states the model is a simplified abstraction, not a cycle-accurate predictor.

pith-pipeline@v1.3.0-alltime-deepseek · 18530 in / 10892 out tokens · 106909 ms · 2026-08-03T06:41:52.865196+00:00 · methodology

0 comments
read the original abstract

Determinism is indispensable for reproducibility in large language model (LLM) training, yet it often exacts a steep performance cost. In widely used attention implementations such as FlashAttention-3, the deterministic backward pass can incur up to a 37.9% throughput reduction relative to its non-deterministic counterpart, primarily because gradient accumulation operations must be serialized to guarantee numerical consistency. This performance loss stems from suboptimal scheduling of compute and gradient-reduction phases, leading to significant hardware underutilization. To address this challenge, we formulate the backward pass of deterministic attention as a scheduling problem on a Directed Acyclic Graph (DAG) and derive schedules that minimize the critical path length. Building on this formulation, we present DASH (Deterministic Attention Scheduling for High-Throughput), which encapsulates two complementary scheduling strategies: (i) Descending Q-Tile Iteration, a reversed query-block traversal that shrinks pipeline stalls in causal attention, and (ii) Shift Scheduling, a theoretically optimal schedule within our DAG model that reduces pipeline stalls for both full and causal masks. Our empirical evaluations on NVIDIA H800 GPUs demonstrate that DASH narrows the performance gap of deterministic attention. The proposed strategies improve the throughput of the attention backward pass by up to 1.28$\times$ compared to the baseline, significantly advancing the efficiency of reproducible LLM training. Our code is open-sourced at https://github.com/SJTU-Liquid/deterministic-FA3.

Figures

Figures reproduced from arXiv: 2601.21824 by Hongmin Chen, Jingwen Leng, Minyi Guo, Shixuan Sun, Xin Liu, Xinwei Qiang.

Figure 1
Figure 1. Figure 1: Overview of the deterministic FlashAttention. Left: Tiled computation structure of the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of the Deterministic Scheduling Problem. The Gantt chart (left) shows a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Backward scheduling of FlashAttention-3 for both mask shapes. Each colored segment [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Descending (reverse-order) query tile schedule for the causal mask. Reversing the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustrative example for Lemma 1. Left: Added dependency (zero-weight) edges pre [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Optimal full-mask schedule via cyclic shifting. Left: Cyclic visiting order of [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Optimal causal-mask sched￾ule using symmetric shift and two-phase workload folding. Phase 1 processes the dense lower-left rectangle; Phase 2 folds the remaining triangles into a logical square and traverses it starting from the main diagonal, first covering the upper-left portion before the lower￾right, ensuring each KV block is exe￾cuted contiguously. Summary of Optimal Performance In summary, the propos… view at source ↗
Figure 8
Figure 8. Figure 8: Backward-pass throughput under full attention masks. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Backward-pass throughput under causal attention masks. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: End-to-end performance of a transformer block. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kernel Contracts: A Specification Language for ML Kernel Correctness Across Heterogeneous Silicon

    cs.LG 2026-04 unverdicted novelty 7.0

    Kernel Contracts is a specification language that formalizes correctness requirements for ML kernels to ensure consistent results across heterogeneous silicon platforms.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [4]

    Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li

    URLhttps://arxiv.org/abs/2501.12084. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025. URLhttps:// arxiv.org/abs/2502.09992. NVIDIA. NVIDIA H100 Tensor Core GPU Architecture. Technical report, NVIDIA, mar 2022. URLhttps://www.nvidia.com/content/da...

    Pith/arXiv arXiv 2025

  2. [5]

    Accessed: Nov. 2025. William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Ragan-Kelley. Striped attention: Faster ring attention for causal transformers, 2023. URLhttps://arxiv.org/abs/2311.09431. Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxi...

    Pith/arXiv arXiv 2025

  3. [2024]

    Horace He and Thinking Machines Lab

    URLhttps://arxiv.org/abs/2406.18485. Horace He and Thinking Machines Lab. Defeating nondeterminism in llm infer- ence.Thinking Machines Lab: Connectionism, 2025. doi: 10.64434/tml.20250910. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, ...

    Pith/arXiv arXiv 2025

  4. [2025]

    URLhttp://dx.doi.org/10.1145/3676641

    doi: 10.1145/3676641.3715996. URLhttp://dx.doi.org/10.1145/3676641. 3715996. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything, 2023. URLhttps://arxiv.org/abs/2304.02643. Hao Liu, Matei Zaharia, and Piete...

    arXiv 2023