{"id":"f9f44000-60b3-4e69-a498-e8d8a7211254","arxiv_id":"2608.18469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ERASE improves training throughput by up to 9.51% on a CTR model by using Forward-Forward-style stop-gradient detachment to launch each block's backward pass early on separate CUDA streams.","lead":"ERASE speeds up training of recommendation models by detaching blocks of a network and running each block's backward pass early on a separate CUDA stream, overlapping it with later forward work. It reports up to 9.51% higher throughput on a click-through-rate model, though the quality gap for the best configuration is not reported.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CTR experiment never specifies local objectives for the six detached subarchitectures, so the 9.51% speedup may reflect untouched subgraphs rather than early-backward overlap; the only NE gap is also for a non-best configuration.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my independent read reinforces it without moving it. The paper supplies some independent evidence: the NanoChat single-batch trace demonstrates real kernel overlap, and the MNIST sanity check shows a timing reduction; neither, however, validates the mechanism on the CTR model, since the trace reports no speedup and the MNIST check reports no accuracy. The main empirical support is Table 1 and the NE curve, and those omit exactly what the central claim needs. The most serious gap is that the local losses ℓ_i required by Eq. (2) are never specified for the six detached dense subarchitectures. Without them, the early backward passes would compute zero gradients, making the throughput gain a freezing result rather than a scheduling result. This is a missing-support problem, not an observed contradiction, so it does not justify rejection. The second gap, already identified by the reader, is that the only NE gap is measured on row 1 while the headline gain is row 0; the quality claim is therefore not attached to the speedup claim. Both issues can be settled by a concrete instrumentation check and a single additional NE measurement. The paper should also provide error bars and code, but those are secondary to the two omissions above. I therefore leave the verdict at CONDITIONAL, matching the reader's assessment.","tokens_in":5565,"tokens_out":8594,"duration_ms":83701,"concrete_test":"Instrument the CTR training run by hooking the six detached subgraphs and, at each early-backward launch, printing the gradient norm of that subgraph's parameters and the top-level loss node(s) that produced it. If any detached subgraph receives zero gradient, or if the gradient is not traceable to an explicit local loss ℓ_i in the code/loss equation, the claimed 9.51% gain cannot be attributed to ERASE scheduling. If the gradients are nonzero and local, the remaining check is to run row 0 (blocking, FUP=True) with the same NE protocol as Figure 3 and report the gap; the central claim stands only if that gap remains comparable to baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ERASE's core mechanism is only meaningful if each detached subgraph has a local objective: in Eq. (2), the backward of block b depends on ℓ_b because downstream gradients are cut by sg. Section 3.2.1 lists six detached dense subarchitectures but never states what ℓ_i are, their weights, or how they are constructed from the CTR label; Section 5 even says 'without FF's goodness objective', leaving ambiguous what gradient the early backward computes. If ℓ_i are absent, ∂L/∂θ_i = 0 for those subarchitectures, so the +9.51% row would be evidence that six subgraphs can be left untrained, not that early backward overlaps with forward. Separately, the only NE gap reported (≈1.38%, Figure 3) belongs to row 1 (non-blocking + CUDA Graphs, FUP=False), while the headline +9.51% is row 0 (blocking, FUP=True). The conclusion's '5–9% with small NE gap' therefore assumes row 0's NE behavior without measuring it. Both omissions are load-bearing for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ERASE, a scheduling scheme that uses Forward-Forward-style detachment to make each block's backward pass ready as soon as the block's forward pass finishes, then launches that backward pass early on a separate CUDA stream so it overlaps with subsequent forward work. The authors argue this improves accelerator utilization on small-kernel workloads, where conventional forward-then-backward scheduling leaves capacity idle. They demonstrate the intended overlap with a single-batch execution trace on a small transformer (NanoChat), note that a device-saturating fused attention kernel prevents further overlap, and then measure training throughput on a large-scale click-through-rate model with six detached dense subarchitectures. The reported result is a p90 QPS improvement of up to 9.51% over baseline with a 'small' normalized-entropy gap, although the only NE gap reported is about 1.38% for a configuration that achieves +7.38%.","tokens_in":5731,"tokens_out":8026,"duration_ms":72493,"significance":"If the empirical claims held, the scheduling idea would be a practical, inexpensive way to improve throughput in small-kernel training workloads. The conceptual derivation in Section 2 is clear and does not rely on fitted constants: the readiness of a block's backward pass follows directly from the stop-gradient construction, and the NanoChat execution trace is a useful sanity check that the intended overlap actually occurs. However, the manuscript currently does not specify the local objectives used for the detached CTR subarchitectures, leaving the meaning of the measured gradients ambiguous, and it does not report a quality metric for the best-throughput configuration. These two gaps weaken the central claim. The paper also does not provide code or a detailed reproducibility protocol, and the measurements are single-run values with no variance reporting.","major_comments":[{"comment":"The six detached subarchitectures are never assigned explicit local objectives. Eq. (2) defines the aggregate loss as L = L_main + Sum_i lambda_i ell_i, and the early backward for block i is only meaningful if ell_i is defined, but the CTR section never states what these ell_i are, how they are derived from the CTR label, what weights lambda_i were used, or which subgraphs receive them. Section 5 even says the scheme works 'without FF's goodness objective,' leaving unclear what gradient the early backward computes. As written, the +9.51% row could be consistent with the six subgraphs receiving zero gradient and therefore not being trained at all, making the throughput gain a comparison against an incompletely trained model. Please specify the per-subarchitecture losses or explain how the subgraph gradients are defined.","section":"Section 3.2.1 / Eq. (2) / Section 5"},{"comment":"The best-throughput row (row 0, blocking + FUP=True, +9.51%) has no reported NE gap; the only quality measurement is approximately 1.38% for row 1 (non-blocking + CUDA Graphs, FUP=False). Since FUP explicitly changes which parameters participate in gradient synchronization, the NE behavior of row 0 cannot be assumed identical to row 1. The conclusion that ERASE gives '5-9% ... with a small NE gap' therefore extrapolates beyond the measured configuration. Please report the NE gap, and ideally the full NE curve, for row 0 and all other rows, or restrict the conclusion to the configuration for which quality was actually measured.","section":"Table 1 / Sections 4 and 5"},{"comment":"The throughput numbers are single values with no repeated runs, confidence intervals, or variance across seeds or replicas. The p90 statistic is taken within one training run; no information is given on how many steps or how many replicas were used. The smallest gain (row 3, +0.37%) is within typical run-to-run noise of distributed training, so the ablation ordering in Table 1 cannot be distinguished from noise. Please run each configuration multiple times, report mean and standard deviation (or per-replica statistics), and state the number of steps used for the p90 computation.","section":"Table 1 / Section 3.2.2"},{"comment":"The CTR experiment description is too incomplete to reproduce: no model architecture (depth and width of the six detached subarchitectures), feature cardinality, dataset, optimizer, learning rate, batch size, data-parallel configuration on the eight H100 GPUs, or training duration is given. Without these details, it is impossible to determine whether the 5-9% speedup is tied to a particular kernel mix or model shape, and the broader claim about 'modern recommendation systems' is unsupported. Please add the missing experimental setup or clearly scope the claim.","section":"Section 3.2.1"}],"minor_comments":[{"comment":"The dagger (†) on row 0 is never explained; if it flags the reported best configuration or refers to a footnote about quality, please state it explicitly. The row ordering (3, 2, 1, 0) is also confusing; reorder by gain or label rows consistently with the text.","section":"Table 1"},{"comment":"The MNIST sanity check is a single sentence with no hyperparameters, model size, or measurement protocol; please provide at least the setup and define what 'compute-matched backward time' means.","section":"Section 3.1"},{"comment":"The statement that 'FUP controls which parameters participate in gradient synchronization' should be made precise, since find_unused_parameters in DDP normally concerns gradient reduction for unused parameters; clarify the exact mechanism and why FUP=True reduces the parameters in the final aggregate backward.","section":"Section 4"},{"comment":"Please clarify why only four of the six detached subarchitectures receive separate CUDA streams; the scheduling effect, and hence the measured gain, may depend on which subgraphs are assigned streams.","section":"Section 3.2.1"},{"comment":"The execution trace would be more informative with axis labels and a time scale; currently it reads as a schematic illustration.","section":"Figure 2"},{"comment":"The phrase 'normalized entropy close to the baseline' is vague; consider reporting the NE gap quantitatively in the abstract or defining 'close' in terms of the observed 1.38% gap.","section":"Abstract / Section 5"},{"comment":"The 'approximately 1.38%' NE gap is ambiguous between percentage points and relative change; please state the baseline NE value and define the gap in absolute and relative terms.","section":"Figure 3 / Section 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The core scheduling idea is plausible and the paper is readable, but the experimental section needs substantial strengthening before this can be accepted as a systems contribution. In particular, the missing local objectives and the missing quality measurement for the headline configuration are load-bearing gaps. I would not reject the paper, since the issues are fixable by adding detail and rerunning the key measurements, but the current version is not yet ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nRead ERASE (arXiv:2608.18469). The core scheduling idea is genuinely nice: FF-style stop-gradient detachment makes each block's backward independent, so you can run it early on a separate CUDA stream and overlap it with downstream forward work. The NanoChat trace shows the overlap clearly, and the observation that saturated kernels leave no room for concurrency is honest and correct. The MNIST check, though brief, is a useful sanity check. This is a real engineering contribution for small-kernel training, not a fundamentally new algorithm.\n\nThat said, the CTR experiment does not support the headline claim as written. Two things are load-bearing and both are missing. First, the six 'detached subarchitectures' are never given local objectives. Eq. (2) defines the aggregate loss with lambda_i * l_i, but Section 3 never tells us what l_i are, what lambda_i are, or how they are computed from the CTR label. Section 5 even says 'without FF's goodness objective'. If there is no local loss, the detached blocks receive zero gradient, and the 9.51% throughput gain is evidence that you can skip training six subgraphs, not that early backward overlaps. The paper must specify these losses or the mechanism does not parse. Second, the NE gap of roughly 1.38% is reported only for row 1 (non-blocking + CUDA Graphs), not for row 0 (blocking, FUP=True), which is the +9.51% configuration. The conclusion's '5–9% with a small NE gap' assumes row 0's quality without measuring it. That is a straightforward omission and fixable.\n\nSmaller issues: no error bars or repeated runs, and the '†' in Table 1 is never explained. Those are minor but contribute to the impression that the paper was rushed.\n\nWho is this for? People working on systems-level training optimizations for recommender models, and anyone interested in Forward-Forward ideas as a scheduling tool rather than as a learning algorithm. It deserves a serious referee with a request for major revision: specify the local losses, report NE for the best row, add variance estimates, and ideally release code. If those come back, the result would be credible and worthwhile. As it stands, I would be reluctant to build on the CTR numbers.\n\nRecommendation: send to peer review, but with the clear expectation that the missing experiment specifications are fixed. It is not a desk reject.","headline":"A smart scheduling idea undercut by a CTR experiment that never specifies the local losses, leaving the headline speedup ambiguous.","tokens_in":6285,"tokens_out":3790,"would_cite":false,"duration_ms":33898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that Forward-Forward-style detachment can be used purely as a scheduling trick: each block's backward pass starts on a separate CUDA stream as soon as its forward pass ends, yielding up to 9.51% higher training…","keywords":["early backward propagation","Forward-Forward detachment","CUDA streams","CUDA Graphs","training throughput","click-through-rate model","normalized entropy","gradient scheduling"],"falsifier":"Run the blocking, FUP=True configuration on the same CTR model and measure its normalized-entropy gap over a full training run; if it is much larger than approximately 1.38%, the paper's headline claim that the best configuration keeps quality close to baseline is contradicted.","tokens_in":5306,"feed_emoji":"⚡","tokens_out":6520,"duration_ms":56245,"temperature":0.7,"pith_summary":"The paper tries to establish that cutting a network into block-local subgraphs can speed up training without changing the mathematics of the gradients inside each subgraph. Its idea is that detaching a block's output with a stop-gradient makes that block's backward pass ready the moment its own forward pass and local objective finish, so the backward work can be launched early on a separate CUDA stream and overlapped with the forward pass of the next block. On a large click-through-rate model, ERASE reports p90 throughput gains up to 9.51% while keeping normalized entropy close to the baseline, and on a lightweight transformer it shows the overlap directly in an execution trace. The paper also marks the limit of the idea: a fused kernel that saturates the device leaves no spare capacity, so overlap helps only where kernels are small enough to leave the accelerator underutilized.","feed_headline":"Early-backward trick lifts CTR training throughput by up to 9.5%","feed_subtitle":"Each detached block's backward pass runs on its own CUDA stream, hidden behind later forward work, with little quality loss.","key_machinery":"The load-bearing mechanism is stop-gradient detachment used as a scheduling primitive. In the Forward-Forward-style graph, the output of block b is passed to block b+1 through a stop-gradient, so no gradient crosses block boundaries; with a local loss on the block's own output, the only inputs needed for that block's gradient are its own forward results and the local objective. ERASE launches each such subgraph's backward pass on a separate CUDA stream, using stream events to preserve the remaining dependencies, and in the non-blocking variant captures the affected subgraphs as CUDA Graphs to fix launch order across ranks. This is what turns idle capacity during the remainder of the forward pass into useful gradient work.","core_discovery":"The central claim is that reverse-mode training does not have to wait for the full forward pass before any backward work begins. Once a block's output is detached, its parameters and local loss form an independent subgraph, and ERASE dispatches that subgraph's backward pass as soon as its forward returns, accumulating gradients while later blocks still run forward. On a large click-through-rate model with six detached dense subarchitectures, the non-blocking configuration with CUDA Graphs raises p90 QPS to 198,145.28 (a 7.38% gain) with a normalized-entropy gap of about 1.38%, and the blocking configuration with FUP=True reaches 202,078.43 (a 9.51% gain). The paper's conclusion is that ERASE yields a 5-9% p90 QPS improvement with a small NE gap when cross-rank collective order is kept deterministic.","pith_inferences":["If the mechanism exploits spare capacity rather than reducing arithmetic, the gain should track the fraction of kernels that do not saturate the device; embedding-heavy and small-GEMM workloads are the natural place to look for larger effects.","The reported NE gap of about 1.38% belongs to the non-blocking + CUDA Graphs row; the paper does not report the NE gap for the best-throughput row, so the quality claim for that configuration is an extension from a nearby measurement, not a stated result.","A direct test of the saturation explanation: shrink hidden width or batch size on a small transformer and observe whether the overlap-induced speedup grows until the device-filling kernel becomes the bottleneck."],"forward_implications":["Small-kernel workloads that leave the accelerator underutilized can convert otherwise idle capacity into training throughput without changing the loss or the per-subgraph gradients.","A device-saturating kernel, such as a fused multi-head attention backward, removes the free capacity that overlap depends on; the benefit is inherently workload-dependent.","Deterministic, rank-synchronized collective order is a practical requirement for the gain: blocking with FUP=True and non-blocking with CUDA Graphs both recover throughput, while plain non-blocking dispatch without fixed order does not.","On the tested click-through-rate model, ERASE improves p90 throughput by roughly 5-9% with a small normalized-entropy gap under deterministic collective order."],"supporting_citations":[{"why":"Supplies the stop-gradient detachment and block-local objective mechanism that ERASE reinterprets as a scheduling primitive.","marker":"[9]"},{"why":"Provides CUDA Graphs as the execution mechanism that fixes asynchronous launch order in the non-blocking variant.","marker":"[19]"},{"why":"Demonstrates kernel batching with CUDA Graphs, the execution-level acceleration that ERASE combines with detachment.","marker":"[6]"},{"why":"Supplies compiler support for robust CUDA Graph capture, used to keep the early-backward dispatch order deterministic.","marker":"[7]"},{"why":"Provides the lightweight transformer proxy on which the single-batch execution trace demonstrates the intended overlap.","marker":"[11]"},{"why":"Supplies the MNIST dataset for the small-scale MLP sanity check that measured 58% lower backward time and 30% lower total batch time.","marker":"[5]"}],"fun_headline_variants":["ERASE overlaps backward and forward passes for 9.5% faster CTR training","Backward passes run early on separate CUDA streams for 9.5% boost","Detach blocks, run backward early: 9.5% higher CTR throughput","ERASE frees hidden GPU capacity, lifting CTR training by up to 9.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the configuration with the highest throughput, which is not the one whose normalized-entropy gap was reported, keeps its NE gap close to the roughly 1.38% measured for the non-blocking CUDA Graphs row; if that gap grows materially with the blocking/FUP=True configuration, the promised combination of speed and quality is not established.","fun_headline_variants_meta":{"raw":{"variants":["ERASE overlaps backward and forward passes for 9.5% faster CTR training","Backward passes run early on separate CUDA streams for 9.5% boost","Detach blocks, run backward early: 9.5% higher CTR throughput","ERASE frees hidden GPU capacity, lifting CTR training by up to 9.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1626,"prompt_tokens":906,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":522,"tokens_out":720,"duration_ms":6395,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T19:56:04.794244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the blocking, FUP=True configuration on the same CTR model and measure its normalized-entropy gap over a full training run; if it is much larger than approximately 1.38%, the paper's headline claim that the best configuration keeps quality close to baseline is contradicted.","supporting_citations":[{"cited_title":"Accelerating PyTorch with CUDA graphs","cited_arxiv_id":null,"evidence_quote":"Provides CUDA Graphs as the execution mechanism that fixes asynchronous launch order in the non-blocking variant."},{"cited_title":"Boosting performance of iterative applications on gpus: Kernel batching with cuda graphs","cited_arxiv_id":null,"evidence_quote":"Demonstrates kernel batching with CUDA Graphs, the execution-level acceleration that ERASE combines with detachment."}],"review_version":1}