FOCUS: DLLMs Know How to Tame Their Compute Bound

An Zhong; Hong Xu; Kaihua Liang; Marco Canini; Xin Tan

REVIEW 3 major objections 5 minor 1 cited by

Diffusion large language models can be decoded several times faster by evicting, after two layers, tokens whose attention delta says they will not decode.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:06 UTC pith:IWV33LYC

load-bearing objection Genuinely new compute-centric token eviction idea for block-diffusion LLMs, but the headline throughput claim is not safe until the benchmark's repetition-termination behavior is quantified or removed. the 3 major comments →

arxiv 2601.23278 v2 pith:IWV33LYC submitted 2026-01-30 cs.LG cs.ARcs.CL

FOCUS: DLLMs Know How to Tame Their Compute Bound

Kaihua Liang , Xin Tan , An Zhong , Hong Xu , Marco Canini This is my paper

classification cs.LG cs.ARcs.CL

keywords diffusion language modelsblock diffusiontoken evictionattention importance deltainference accelerationcompute-bound decodingthroughput scaling

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Diffusion large language models (DLLMs) generate many candidate tokens in parallel, but at each denoising step only a small fraction—roughly 10%—are actually decoded; the rest of the block still consumes full attention and feed-forward compute. This paper claims that which tokens will decode is predictable before most of the network runs: the rise in attention importance between the first two transformer layers, the 'importance delta,' correlates strongly with a token's decoding probability. On that basis it builds FOCUS, a training-free inference system that computes only the first two layers for the whole block, evicts low-delta tokens, and runs the remaining layers only on the retained candidates. The paper reports that this removes 65–80% of per-step token processing, restoring throughput scaling at large batch sizes, with up to 3.52x throughput over a state-of-the-art serving engine while preserving or improving benchmark quality.

Core claim

FOCUS establishes that token decodability in block-diffusion language models is not random but can be read off from the change in incoming attention between Layer 0 and Layer 1. Layer 0 attention is dominated by static priors and noise; Layer 1 shows a semantic lift for tokens that are about to decode. The difference, ΔI, separates decodable from non-decodable tokens, so the system can evict the latter after computing only two layers and run the rest of the transformer on a reduced token set. Combined with a dynamic retention budget and a neighbor-aware KV cache that freezes only stabilized tokens, this brings the computation-to-generation ratio close to the 1:1 ideal of autoregressive model

What carries the argument

The central object is the importance delta, ΔI_j = I_j^(Layer1) − I_j^(Layer0), where I_j is the column-wise sum of softmax attention weights received by token j from all query tokens in the block, smoothed by MaxPool1D. The subtraction acts as common-mode rejection, removing Layer 0's positional/noise baseline and isolating the semantic lift in Layer 1. FOCUS uses ΔI to select top-K candidates under a dynamic budget, then applies two structural constraints—retaining each candidate's immediate predecessor and all masked tokens before the rightmost candidate—and an intra-block KV cache with a neighbor-aware stability criterion. This mechanism converts a statistical correlation into a wall-clo

Load-bearing premise

The load-bearing premise is that the Layer 0-to-Layer 1 attention delta reliably separates decodable from non-decodable tokens in every context, not just the tested benchmarks and models; if that correlation weakens, FOCUS evicts tokens the model would have decoded and quality silently drops.

What would settle it

Run the FOCUS eviction policy on long, open-domain conversations or multi-step reasoning chains and compare, token by token, the evicted set against the set the full model actually decodes at the next step. If a substantial fraction (say >5%) of evicted tokens would have been decoded, or if downstream quality degrades measurably despite the same confidence threshold, the importance-delta predictor is not robust enough to support the speedup claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

DLLM serving shifts from compute-bound saturation back to throughput scaling as batch size grows, because effective FLOPs per step drop roughly in proportion to the eviction rate.
The predictor can act as a safety filter, allowing lower confidence thresholds without the quality degradation caused by high-confidence-but-wrong tokens.
Because the technique is training-free and demonstrated across both dense and mixture-of-experts architectures, it can be applied to future block-diffusion models without retraining.
A smaller retained set also reduces the number of KV states to refresh and freeze, shrinking memory traffic and cache-management overhead, not just arithmetic.
Diffusion decoding moves closer to the 1:1 computation-to-generation ratio of autoregressive models, suggesting that DLLM serving efficiency could approach AR engine levels on the compute side.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: if the importance-delta predictor is a universal property of masked diffusion models, it could be folded into training—models could be regularized to make decodability even more separable, or the heuristic budget could be replaced by a learned scheduler.
Beyond the paper: the reported speedup depends on a throughput benchmark that terminates repetitive requests; in open-domain chat or long reasoning where such termination is disallowed, the gains may be smaller and quality effects should be measured directly.
Beyond the paper: a natural test is to apply the same two-layer eviction rule to non-English or multimodal block-diffusion models; the Layer 0 vs Layer 1 attention delta should be re-validated before assuming the correlation transfers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Genuinely new compute-centric token eviction idea for block-diffusion LLMs, but the headline throughput claim is not safe until the benchmark's repetition-termination behavior is quantified or removed.

read the letter

Two things to know. First, the core idea is real and worth engaging with: the paper predicts which masked tokens in a block-diffusion LLM will decode by looking at the change in incoming attention between layers 0 and 1, then evicts the rest from the forward pass. That is a genuine departure from the prior KV-cache and approximate-caching work, and the cheap two-layer delta is a plausible, well-motivated signal. Second, the 3.52× throughput headline is not yet well supported as reported, because the benchmarking script terminates requests with excessive repetition and the paper does not quantify the termination rate or its differential effect on the two systems.

What the paper does well: the systems work is serious. There is a real implementation on top of LMDeploy with custom Triton kernels, 4,000+ lines, and a public repo. The ablations are useful, especially the comparison showing the delta metric beats raw Layer 1 attention. The observation that only ~10% of tokens decode per step is clearly documented across models and block sizes. The design constraints around AR-context preservation and placeholder integrity make sense, and the overhead analysis is at least plausible. I do not see a load-bearing logical flaw. The importance-delta predictor is empirical, which is fine as long as it is not oversold, and the paper mostly does not oversell it.

The soft spots are real but fixable. The repetition-termination issue in Appendix F.1 is the most serious one. Under continuous batching, terminated requests free slots, so even a modest termination rate can inflate measured throughput, and if FOCUS suppresses repetition while the baseline degenerates, the comparison is systematically unfair. This needs to be quantified or the benchmark rerun without truncation. Second, all quality numbers are averages over two runs with no error bars; given the small benchmark set and the fact that α, the confidence threshold, and the MaxPool kernel size are chosen on these same benchmarks, the quality-preservation claim is weaker than the tables suggest. This is mild circularity, not a fatal one. Third, the theoretical safety bound in Appendix D is explicitly conditional on a zero-mean Gaussian model of the importance delta. The paper acknowledges the approximation and argues it is conservative, which is honest, but the bound should not be cited without that caveat. Fourth, the generalization worry is legitimate: the correlation is demonstrated on math, code, and instruction-following, not on long open-domain chat or complex multi-step reasoning. If the delta signal weakens there, eviction silently drops quality.

I would send this to peer review. Ask for the termination statistics, quality confidence intervals, and at least one long-form or open-domain chat evaluation. If those hold up, this is a solid contribution to the LLM-systems literature.

Referee Report

3 major / 5 minor

Summary. The paper proposes FOCUS, a training-free inference system for block-diffusion large language models (DLLMs). The key observation is that the per-token drift in early-layer attention importance (the difference between Layer 1 and Layer 0 incoming attention scores, the 'importance delta') correlates with whether a masked token will be decoded at the current denoising step. FOCUS computes this delta after the first two layers, evicts low-delta tokens before computing the remaining layers, and dynamically sets the retention budget from historical decoding yield and within-block delta statistics. Evaluations on SDAR-8B-Chat and LLaDA2.0-mini report quality matching or exceeding baselines on GSM8K, Math500, HumanEval, MBPP, and IFEval, and throughput gains over LMDeploy up to 2.32x at block size 32 and 3.52x at block size 64.

Significance. If the reported results hold, FOCUS addresses a real bottleneck in DLLM serving: block-wise diffusion recomputes attention/FFN for the full block even though only a small fraction of tokens decode per step. The paper gives a clean, falsifiable hypothesis (early-layer importance delta predicts decodability), a concrete system design with custom Triton kernels, and a public implementation. The claim that a training-free eviction policy can preserve or even improve quality while cutting FLOPs is significant for the growing DLLM inference literature. The main caveats are experimental: a benchmark termination rule that can bias throughput, and limited validation of quality on open-domain chat/long-reasoning settings.

major comments (3)

[Appendix F.1, Section 5.3, Figure 7] The throughput benchmark includes a mechanism that detects and terminates requests with excessive repetition, but the main text does not disclose this, the trigger threshold is not defined, and the frequency/impact is not quantified. DLLM decoding time grows with the number of denoising steps, so removing degenerate long-running requests can disproportionately raise measured tokens/s, and under continuous batching it also frees batch slots. If FOCUS reduces repetition while the LMDeploy baseline does not, the reported 3.52x speedup is inflated. Please report termination rates and discarded-token counts for each system, specify the detection rule, and provide a sensitivity analysis with the mechanism disabled or applied uniformly to all baselines, including Fast-dLLM v2.
[Section 3.2, Table 4, Section 5.1] The same five benchmarks (GSM8K, Math500, HumanEval, MBPP, IFEval) are used to (a) establish the correlation in Figure 4, (b) compare fixed Top/Random/Bottom selection in Table 3, and (c) tune alpha and the confidence threshold in Table 4. This creates a circularity: the reported quality results are on the same distribution used for predictor development and hyperparameter selection. Please split the data into a calibration set and a held-out test set, or add at least one held-out benchmark family (e.g., another reasoning or instruction-following suite), and report tuned-vs-default results.
[Section 5.3, Section 5.2, Appendix B] The throughput experiments use ShareGPT and WildChat, but generation quality is only reported on the five academic benchmarks; there is no quality measurement on open-domain chat or long, multi-step reasoning. Since FOCUS physically evicts tokens, a failure of the importance-delta predictor on such distributions would silently degrade output quality while throughput numbers look good. Please measure quality (e.g., response-level metrics or human/LLM-judge scores) on a sample of ShareGPT/WildChat, and on long MATH samples or another long-chain reasoning set, so the reader can verify that the predictor's generalization is not merely assumed.

minor comments (5)

[Eq. (2)] The softmax in Eq. (2) is written over S^{(h)}_{i,j} without specifying the normalization axis; the surrounding text suggests a row/query-wise softmax, but this should be explicit.
[Section 3.2, Eq. (3)] The 'importance delta' is motivated as Common Mode Rejection, but the statistical properties of the difference (e.g., correlation between Layer 0 and Layer 1 importance) are not analyzed. A sentence quantifying the typical correlation would strengthen the claim.
[Figure 6 and Figure 7] The caption of Figure 6 reports up to 2.32x while the abstract/conclusion emphasize 3.52x from Figure 7 (B=64). Consider stating the block-size dependence in the main-text figure captions to avoid apparent inconsistency.
[Appendix E.3.4] Disabling multi-loop optimization in LMDeploy is an important implementation detail; the resulting scheduler overhead should be reported quantitatively in the main text, not only in the appendix, because it affects the fairness of the speedup comparison.
[Appendix F.1] The paper says results are averaged over two runs, but no variance or error bars are shown. Given that throughput measurements can be noisy, reporting min/max or standard error would help.

Circularity Check

1 steps flagged

Partial circularity confined to in-sample quality hyperparameter selection; the core attention-delta predictor and throughput measurements are otherwise self-contained.

specific steps

fitted input called prediction [Section 5.2 (Table 4), default configuration selection]
"Based on this robustness, we adopt α=1.5 and Conf=0.8 as the default configuration, as it yields an average score of 68.37 across benchmarks—outperforming even the conservative Baseline at Conf=0.9 (67.60)."

The hyperparameters α and the confidence threshold are selected after inspecting generation-quality results on the same five benchmarks used to report the paper's quality-preservation claim. Thus the headline claim 'preserving or improving generation quality across multiple benchmarks' is an in-sample restatement of the selection criterion rather than an out-of-sample prediction. This is a fitted-input-called-prediction pattern, although it is confined to the quality evaluation; the throughput experiments and the attention-delta predictor are not derived from the throughput numbers, so the core efficiency claim remains independent.

full rationale

The paper's central derivation—that early-layer attention deltas predict decodability—is an empirical correlation, not an analytic identity: Eq. 3 defines a free metric, and the decoding probability comes from the model's confidence head. The theoretical bound in Appendix D is conditional on Gaussian assumptions that the paper explicitly labels as a conservative proxy, so it does not make the predictor circular. The throughput claims are measured wall-clock numbers; the repetition-termination mechanism in Appendix F.1 is a potential benchmark confound but not a definitional reduction because no reported value is constructed to equal the termination rule. The only genuine circularity is the selection of α and Conf on the same benchmarks used to assert quality preservation, making that particular claim in-sample. This is mild and does not invalidate the independent efficiency results.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

FOCUS introduces one explicit hyperparameter (α) and relies on the decoding confidence threshold, both tuned on the evaluation benchmarks. The theoretical safety analysis depends on a Gaussian assumption about attention deltas that is asserted, not measured. No new physical or architectural entities are introduced.

free parameters (3)

Expansion factor α = 1.5 (default)
Introduced in Eq. 4 to control retention budget aggressiveness; default chosen from grid {1.2, 1.5, 1.8} in Table 4 based on highest average benchmark score.
Confidence threshold Conf = 0.8 (default)
Used to define decodable tokens; default selected from {0.9, 0.8, 0.7} in Table 4 based on quality, replacing the more common 0.9 default.
MaxPool1D kernel size k = 3
Chosen in Eq. 15 and stated as 'consistent with the configuration defined in Eq. 15'; no sensitivity analysis.

axioms (3)

domain assumption Importance deltas for non-decodable tokens follow a zero-mean Gaussian (Eq. 6), and decodable tokens follow a shifted Gaussian with SNR γ (Eq. 7).
The theoretical safety bound in Appendix D relies on this distributional assumption; the paper acknowledges raw attention is heavy-tailed but asserts the delta cancels skewness, without fitting or formally testing the Gaussian model.
domain assumption The correlation between importance delta and decodability is universal across models, tasks, and steps.
Section 3.2 and Appendix B generalize from three models (SDAR, LLaDA2.0-mini, LLaDA-8B-Instruct) and five benchmarks to a 'universal characteristic intrinsic to MDLMs'; this is an inductive leap.
ad hoc to paper Layer 0 attention reflects static priors/random noise, while Layer 1 provides the first 'semantic lift'.
Section 3.2 hypothesizes this role separation for Layer 0 vs Layer 1 to justify the delta metric; it is an explanatory narrative, not a proven mechanism.

pith-pipeline@v1.3.0-alltime-deepseek · 24865 in / 9009 out tokens · 106126 ms · 2026-08-03T06:06:28.504560+00:00 · methodology

0 comments

read the original abstract

Diffusion Large Language Models (DLLMs) offer a compelling alternative to Auto-Regressive models, but their deployment is constrained by high decoding cost. In this work, we identify a key inefficiency in DLLM decoding: while computation is parallelized over token blocks, only a small subset of tokens is decodable at each diffusion step, causing most compute to be wasted on non-decodable tokens. We further observe a strong correlation between attention-derived token importance and token-wise decoding probability. Based on this insight, we propose FOCUS, an inference system designed for DLLMs. By dynamically focusing computation on decodable tokens and evicting non-decodable ones on-the-fly, FOCUS increases the effective batch size, alleviating compute limitations and enabling scalable throughput. Empirical evaluations demonstrate that FOCUS achieves up to 3.52$\times$ throughput improvement over the production-grade engine LMDeploy in large-batch settings, while preserving or improving generation quality across multiple benchmarks.

Figures

Figures reproduced from arXiv: 2601.23278 by An Zhong, Hong Xu, Kaihua Liang, Marco Canini, Xin Tan.

**Figure 1.** Figure 1: Comparison of Inference Paradigms. (Left) ARLLMs generate tokens one token at a time. (Right) DLLMs process an entire block in parallel, yet decode only a subset at every step. tensity fundamentally breaks the conventional assumption of LLM inference being memory-bound (Pope et al., 2023) and shifts it to a compute-bound regime. This shift reveals a critical efficiency gap: while block-wise parallelism ena… view at source ↗

**Figure 2.** Figure 2: Distribution of decoded tokens per step across benchmarks. Using the SDAR-8B-Chat model with block size B = 32 and conf idence threshold = 0.9, the data reveals that the mean proportion of successfully decoded tokens is typically only around 10%, indicating that ∼ 90% of the block-wise computation is redundant. Please see Appendix A for more results. computes the entire block despite decoding only a subset… view at source ↗

**Figure 4.** Figure 4: Importance Delta vs. Decodability. Decodable tokens (green) cluster at high deltas, distinct from Non-Decodable ones (grey). Settings match [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: FOCUS design overview. The workflow centers on (Middle) Token Eviction, which performs early filtering after the Layer 1 Q/K projections (QKProj). It prioritizes decodable token candidates based on the Importance Delta (∆I in Eq. 3) from (Right) Delta Calculation, and retaining only the top candidates within the adaptive token budget determined by (Left) Dynamic Budgeting. represents the count of tokens wi… view at source ↗

**Figure 6.** Figure 6: Throughput Scaling. FOCUS achieves up to 2.32× throughput improvement over the LMDeploy baseline. FOCUS breaks the computational wall, enabling sustained throughput growth at larger batch sizes for both SDAR and LLaDA2.0 models. settings. Since the inference process operates as a Markov chain (Cheng et al., 2025), selecting incorrect tokens as decodable candidates—as frequently occurs in Random and especia… view at source ↗

**Figure 8.** Figure 8: Throughput breakdown on ShareGPT and SDAR. While Intra-Block KV cache aids efficiency, the full FOCUS system provides substantial speedup without compromising quality. by CPT-adapted models. Our Neighbor-Aware strategy (DC+) successfully restores stability. Crucially, however, the complete FOCUS system achieves generation quality that surpasses even the original Baseline. This reaffirms that FOCUS acts as… view at source ↗

**Figure 9.** Figure 9: Decoded token distribution for SDAR across block sizes B ∈ {8, 16, 64}. Histograms show the number of decoded tokens per step across all benchmarks. Rows correspond to different block sizes, while columns represent different datasets. 1 11 22 32 Number of Decoded Tokens 0 0.2 0.4 0.6 0.8 1 Proportion Mean: 3.09 Std: 4.50 Median: 1.00 B=32 GSM8K 1 11 22 32 Number of Decoded Tokens Mean: 2.29 Std: 3.50 Media… view at source ↗

**Figure 10.** Figure 10: Decoded token distribution for LLaDA2.0-mini (B = 32). Histograms showing the number of decoded tokens per step across all evaluated benchmarks. 1 11 22 32 Number of Decoded Tokens 0 0.2 0.4 0.6 0.8 1 Proportion Mean: 2.26 Std: 3.08 Median: 1.00 B=32 GSM8K 1 11 22 32 Number of Decoded Tokens Mean: 2.82 Std: 4.09 Median: 1.00 Math500 1 11 22 32 Number of Decoded Tokens Mean: 3.76 Std: 5.63 Median: 1.00 Hum… view at source ↗

**Figure 11.** Figure 11: Decoded token distribution for LLaDA-8B-Instruct (B = 32). Histograms showing the number of decoded tokens per step across all evaluated benchmarks. A. Extended Analysis of Decoded Token Statistics We conduct an extended analysis of the decoding statistics across three representative DLLMs, including the SDAR family (Cheng et al., 2025), LLaDA 2.0-mini (Bie et al., 2025), and LLaDA-8B-Instruct (Nie et al.… view at source ↗

**Figure 12.** Figure 12: Importance Delta vs. Decodability across all benchmarks. Columns represent individual datasets, while rows correspond to LLaDA-8B-Instruct (Full-Diffusion) and LLaDA2.0-mini (Block-Diffusion), respectively. [0, 20) [20, 40) [40, 60) [60, 80) [80, 100] Percentile (%) 0 10 20 30 40 50 60 70 Percentage (%) 10.3 9.4 13.5 17.7 49.1 17.4 19.0 23.7 21.5 18.4 GSM8K Decodable Non-Decodable [0, 20) [20, 40) [40, 60… view at source ↗

**Figure 13.** Figure 13: Layer 1 Importance vs. Decodability across all benchmarks. Instead of using importance delta, we use the raw importance scores at Layer 1 directly. Settings match [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

**Figure 14.** Figure 14: Distribution of Importance Delta (∆I) across benchmarks. Decodable tokens (green) consistently exhibit higher ∆I values than non-decodable tokens (grey). The dashed line indicates the σ threshold (+Std) used for dynamic budgeting. Settings match [PITH_FULL_IMAGE:figures/full_fig_p016_14.png] view at source ↗

**Figure 15.** Figure 15: Throughput Comparison with Fast-dLLM v2 on SDAR. FOCUS significantly outperforms Fast-dLLM v2, achieving up to 9.04× the throughput on WildChat dataset. Throughput Benchmarks. Given that semantic context affects the decoding speed of DLLMs (Cheng et al., 2025), we integrate a mechanism within the benchmarking script to detect and terminate requests exhibiting excessive repetition. Although we have impleme… view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation
cs.AI 2026-07 conditional novelty 6.0

A diffusion multimodal LLM's early-layer MLP sparsity at denoising step 0 pinpoints the semantic boundary, enabling one-shot suffix truncation that raises throughput up to ~31x.

Reference graph

Works this paper leans on

7 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

18 FOCUS: DLLMs Know How to Tame Their Compute Bound

Preparation Phase:The scheduler aggregates per-sequence metadata (local indices, processed pointers, and decoding averages). 18 FOCUS: DLLMs Know How to Tame Their Compute Bound
[2]

Host-Side Packing:Values are packed into pre-allocated pinned buffers to eliminate allocation overhead during the forward pass
[3]

Asynchronous Transfer:Data is copied to the device via non-blocking operations, recording CUDA events for synchronization
[4]

GPU Execution:Kernels atomically update the block progress state and compute importance metrics without host intervention
[5]

State Persistence and ResetWhen a block completes processing (all tokens become cached), the scheduler resets per-sequence states: •FocusState.rightmost processedis reset to−1

Async Readback:The scheduler waits on the recorded event before reading back updated progress, ensuring consistency without blocking the computation stream. State Persistence and ResetWhen a block completes processing (all tokens become cached), the scheduler resets per-sequence states: •FocusState.rightmost processedis reset to−1. •DelayedCacheState.unca...

2022
[2021]

Jawahar, G., Sagot, B., and Seddah, D

URL https://openreview.net/forum? id=7Bywt2mQsCe. Jawahar, G., Sagot, B., and Seddah, D. What does BERT learn about the structure of language? InProceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, 2019. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Am...

Pith/arXiv arXiv 2019
[2024]

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K

URL https://openreview.net/forum? id=mZn2Xyh9Ec. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, pp. 4171–4186, 2019. Fu...

Pith/arXiv arXiv 2019

[1] [1]

18 FOCUS: DLLMs Know How to Tame Their Compute Bound

Preparation Phase:The scheduler aggregates per-sequence metadata (local indices, processed pointers, and decoding averages). 18 FOCUS: DLLMs Know How to Tame Their Compute Bound

[2] [2]

Host-Side Packing:Values are packed into pre-allocated pinned buffers to eliminate allocation overhead during the forward pass

[3] [3]

Asynchronous Transfer:Data is copied to the device via non-blocking operations, recording CUDA events for synchronization

[4] [4]

GPU Execution:Kernels atomically update the block progress state and compute importance metrics without host intervention

[5] [5]

State Persistence and ResetWhen a block completes processing (all tokens become cached), the scheduler resets per-sequence states: •FocusState.rightmost processedis reset to−1

Async Readback:The scheduler waits on the recorded event before reading back updated progress, ensuring consistency without blocking the computation stream. State Persistence and ResetWhen a block completes processing (all tokens become cached), the scheduler resets per-sequence states: •FocusState.rightmost processedis reset to−1. •DelayedCacheState.unca...

2022

[6] [2021]

Jawahar, G., Sagot, B., and Seddah, D

URL https://openreview.net/forum? id=7Bywt2mQsCe. Jawahar, G., Sagot, B., and Seddah, D. What does BERT learn about the structure of language? InProceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, 2019. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Am...

Pith/arXiv arXiv 2019

[7] [2024]

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K

URL https://openreview.net/forum? id=mZn2Xyh9Ec. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, pp. 4171–4186, 2019. Fu...

Pith/arXiv arXiv 2019