{"id":"34e28636-78d3-40ea-b9b4-c5bb34a006ce","arxiv_id":"2509.00925","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A learned router sends about 90% of tokens through a linear projection path instead of quadratic attention, keeps the MLP for every token, and matches dense Transformer accuracy at roughly 85% of the FLOPs.","lead":"Researchers built DTRNet, a Transformer that lets each token choose between full self-attention and a cheaper projection-only update at every layer, while all tokens still pass through the MLP. After training, only about 10% of tokens use attention in its routing layers, yet accuracy stays close to a dense Transformer with lower compute and less KV-cache memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Long-context FLOPs advantage rests on an unverified assumption: the hard-argmax router keeps routing ~10% of tokens under 20K YaRN extrapolation; Figure 4 fixes k=0.10 theoretically, but the actual routing rate at 20K is never measured.","rationale":"The reader's weakest assumption identifies both the information-preservation premise of the bypass path and the transfer of soft-training/hard-inference routing from 2K to 20K. I agree that the long-context piece is the most load-bearing: the paper's headline efficiency gain at 20K is computed from a fixed k=0.10, but no experiment confirms that the trained router actually routes ~10% of tokens under YaRN extrapolation. The standard-length experiments, ablations, and matched-FLOPs comparisons are internally consistent and provide reasonable support for the 2K claims, so I do not see grounds to reject. The concern is a specific missing measurement that could be resolved by instrumenting the released checkpoint; it reinforces the reader's CONDITIONAL verdict rather than moving it. I keep the verdict unchanged because the reader already conditioned on exactly this kind of unverified extrapolation, and the proposed test would settle whether the condition is actually violated.","tokens_in":15020,"tokens_out":17793,"duration_ms":231639,"concrete_test":"Run the released DTRNet-1.3B checkpoint on PG-19 (or all six long benchmarks) at lengths 2K, 8K, 14K, and 20K with the paper's YaRN factor 10.0. Log the hard router decisions per DTR layer and compute the mean attention-token fraction at each length. Then recompute the 20K FLOPs ratio using the measured fractions instead of k=0.10. Accept the long-context efficiency claim only if the measured fraction stays within [0.08, 0.12] and the recomputed ratio is approximately 0.785; if the measured fraction is materially higher (e.g., >=0.15) or strongly length-dependent, Figure 4 does not represent actual DTRNet operation and the long-context FLOPs advantage is unsubstantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The advertised long-context gain is the most exposed part of the central claim. Figure 4 plots FLOPs ratio versus sequence length under the fixed assumptions k=0.10 for DTRNet, k=0.70 for MoD, and k=0.84 for D-LLM, and the text states: 'At a 20K sequence length, DTRNet operates at a FLOPs ratio of 0.785.' However, k=0.10 is the routing fraction measured at standard evaluation lengths (Figure 5). The router is a two-layer MLP trained on 2K sequences with soft path weighting and used with hard argmax at inference; under YaRN(10.0) extrapolation, hidden-state statistics at 8K-20K can shift. The paper reports no measurement of the actual hard-argmax attention-token fraction at those lengths. If the router sends more than ~10% of tokens to attention as context grows, or if the fraction becomes layer-dependent, the 0.785 figure is a theoretical lower bound rather than an operating point, and the claimed efficiency advantage over MoD/D-LLM at long context can shrink or vanish. The perplexity results in Figure 3 are not in question; what is at stake is the 'efficiency gains scale with sequence length' contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DTRNet, a Transformer variant in which each layer contains a learned two-layer router that sends each token either to the full self-attention path or to a linear projection path x W_V W_O followed by the shared MLP. Training uses soft weighting of both paths plus a load-balancing penalty; inference uses hard argmax routing. The architecture interleaves standard Transformer layers with DTRNet layers, with full attention in the first and last layers. Experiments at 360M and 1.3B scales on language modeling and downstream tasks report that DTRNet matches dense Transformer performance at about 0.84–0.85 FLOPs ratio, outperforms MoD and D-LLM under matched FLOPs, lowers KV-cache memory, and has a theoretical FLOPs ratio that decreases with sequence length (0.785 at 20K). The paper also includes ablations on routing strategy, architecture variants, and the necessity of the W_V W_O bypass path.","tokens_in":15283,"tokens_out":7497,"duration_ms":90604,"significance":"If the central claim holds, DTRNet offers a simple and practical way to reduce the quadratic cost of attention while retaining most of the model's quality: a token-local linear update plus the full MLP is enough for ~90% of tokens at each layer. The paper's strengths are concrete: code is released, the main comparisons are done under a fixed FLOPs budget, multiple ablations (A1–A5) probe design choices, and the long-context evaluation goes up to 20K tokens. The 10%-routing result is non-obvious and, if verified, would be a useful contribution to the growing literature on dynamic token routing. However, the efficiency and memory claims rest on an internal inconsistency in the attention definition and on an unverified assumption about routing behavior at extrapolated lengths, so the claims are not yet established in the submitted form.","major_comments":[{"comment":"There is a direct contradiction between the two formal definitions of the attention path. Eq. (4) defines Attn(x_i) using K = X W^K and V = X W^V, where X is the full token sequence, so an attention-routed token attends to all tokens (subject to the causal mask). Eq. (6) defines the effective mask as M = \\tilde{g} \\tilde{g}^T, which restricts attention to interactions only among tokens routed to attention. These are different operations: the former requires computing and caching K,V for every token, while the latter only needs K,V for the ~10% routed tokens. The KV-cache savings in Figure 6 and the FLOPs ratios in Table 1 and Figure 4 presuppose the Eq. (6) semantics, since otherwise no KV-cache is saved. The paper must specify which semantics is actually implemented, align Eqs. (4) and (6), and re-derive the FLOPs/memory accounting accordingly. This is load-bearing for the paper's main","section":"Methodology, Computation Paths, Eq. (4) and Eq. (6)"},{"comment":"The statement 'At a 20K sequence length, DTRNet operates at a FLOPs ratio of 0.785' is based on a theoretical calculation that fixes k=0.10 for DTRNet throughout. The 10% figure is measured at the standard evaluation lengths (Figure 5), not at 8K–20K under YaRN(10.0). The router is a two-layer MLP trained on 2K sequences with soft path weights, so its hard-argmax decisions at extrapolated positions could shift. No measurement of the actual hard-routing fraction at 4K, 8K, 12K, or 20K is reported. If the fraction grows with length, 0.785 is a lower bound rather than an operating point, and the claimed long-context advantage over MoD/D-LLM may shrink. Please report measured routing fractions by layer at the extrapolated lengths, or substantially soften the claim.","section":"Results and Analysis, Figure 4"},{"comment":"The paper criticizes MoD for the inconsistency between its auxiliary router and the true routing decisions, but DTRNet itself uses soft scores to weight both paths during training (Eq. 3 and Eq. 5, with the soft score g) and hard argmax at inference (Eq. 2). This is a soft-to-hard mismatch of the same kind, and the paper does not evaluate it. The sensitivity analyses in Appendix A3 and A5 show that the composition of the two paths strongly affects accuracy (going from 44.36 to 42.50 without attention, and to 41.99 without W_V W_O), so the router's hard decisions are not a trivial detail. Please report at minimum the agreement rate between soft argmax and hard decisions during validation, or an ablation using straight-through/soft routing at inference. Without this, the claim that the trained model routes to 10% attention while preserving accuracy is not fully established for the deployed","section":"Methodology, Training and Regularization; Results, Figure 5"}],"minor_comments":[{"comment":"Mixture-of-Depths is attributed to '(Zhou et al. 2022)', but the reference list contains Zhou et al. as 'Mixture-of-Experts with Expert Choice Routing'. The MoD paper is Raposo et al. 2024. Please correct the citation.","section":"Related Work"},{"comment":"The loss in Eq. (7) is hard to parse: α_l is defined with δ (hard decisions) while the norm uses G (soft scores), and the text says 'penalize the product of the load and the aggregate attention scores'. Please rewrite the equation with explicit notation and explain the role of α_l. Also, f_att^l is a count but is later used in a normalized fraction; clarify the index ranges.","section":"Methodology, Eq. (7)"},{"comment":"The notation for DTRNet-BiLayer and DTRNet-TriLayer is ambiguous. 'T-D-T-D-T-D' and 'T-D-D-T-D-D' do not make it clear how the 32-layer 360M and 24-layer 1.3B models are composed, particularly with first/last layers always being full Transformer layers. Please give the exact layer-by-layer pattern for the actual model sizes.","section":"Experiments, DTRNet configurations"},{"comment":"All results appear to come from a single training run with no error bars or multiple seeds. Given the small differences between DTRNet and the dense baseline (e.g., 44.36 vs 44.23 at 360M), a statement about run-to-run variance would help the reader judge the stability of the comparison.","section":"Results and Analysis, Figure 3 and Table 1"},{"comment":"The y-axis label '1e 1' is confusing; the plotted FLOPs ratios are around 0.8, not 10. Please use a normal 0–1 axis or clearly explain the scaling.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting empirical result and a clean architectural idea, but the submitted version has a load-bearing inconsistency between Eq. (4) and Eq. (6) that directly affects the FLOPs and KV-cache accounting. The long-context FLOPs claim is also based on an unverified constant routing fraction. These are fixable with clarifications and additional measurements, but until then the efficiency claims are not reliable. The soft/hard routing mismatch deserves a direct experiment rather than a reference to common practice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nDTRNet is a decent empirical paper with a design worth knowing: it routes tokens to either full attention or a linear bypass path that applies the shared W_V W_O projections followed by the MLP. That is what separates it from MoD and D-LLM, which skip the whole block. The paper shows the combination works: at 360M, DTRNet-BiLayer averages 44.36 versus 44.23 for the dense SmolLM baseline, and at 1.3B it gets 51.02 versus 51.53, beating both routing baselines at matched FLOPs. The ablations (A3 and A5) are well chosen and demonstrate that both the attention path and the W_V W_O update contribute meaningfully. The authors also reimplement the original lower-FLOPs MoD and D-LLM variants in the appendix, which is fair practice.\n\nThe main soft spot is the long-context efficiency claim. Figure 4's FLOPs curve assumes DTRNet keeps routing 10% of tokens to attention at 20K, but that is the fraction measured at standard lengths. The router is a two-layer MLP trained with soft path weighting at 2K and used with hard argmax under YaRN(10.0); the paper never measures the actual routing rate at 8K–20K. If the fraction climbs with sequence length, the 0.785 ratio is a lower bound rather than an operating point, and the advertised advantage over MoD and D-LLM shrinks. The perplexity curves in Figure 3 are empirical, but the FLOPs curve is arithmetic, and the two are not tied together in the text. That should be fixed before the claim is trusted.\n\nOther issues are smaller. All numbers are single-run with no error bars. The 1.3B DTRNet is 0.5 accuracy points below dense, so \"comparable\" is accurate but \"better\" is not. The training uses soft weighting while inference hard-argmaxes, the same mismatch the paper criticizes in MoD, and no ablation tests the effect. The regularization strength lambda is hand-chosen, and the authors concede tuning difficulty in the conclusion. None of these sink the core result, which is: at standard 2K context, a transformer that routes ~10% of tokens through attention but keeps an MLP for everyone can match dense accuracy at about 15% lower FLOPs and lower KV-cache memory. That is a useful, modest contribution to efficient inference.\n\nFor peer review: yes, send it out. The core claim is empirically grounded and the architecture is genuinely different from the baselines. The referee should ask for routing-rate measurements under length extrapolation, error bars, and an ablation on the soft/hard mismatch. This paper is worth a reading group slot for people working on token routing or efficient transformers.","headline":"A genuinely different token-routing design with solid 2K-context results, but the long-context FLOPs advantage is computed, not measured, and the paper should be revised to verify routing rates under YaRN.","tokens_in":15936,"tokens_out":3233,"would_cite":true,"duration_ms":34678,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learned router can send about one token in ten through full self-attention while still matching dense Transformer accuracy.","keywords":["dynamic token routing","sparse attention","transformer efficiency","mixture-of-depth","KV-cache memory","long-context extrapolation","token-choice routing","language modeling"],"falsifier":"Take a trained 360M DTRNet and retrain a variant where the router is forced to pick a random 10% of tokens per layer, keeping the same training budget and architecture. If the random-10% model matches DTRNet's 44.36 average accuracy instead of dropping, then the learned router is not what preserves performance; if it drops, the router's token selection is load-bearing.","tokens_in":14800,"feed_emoji":"⚡","tokens_out":10697,"duration_ms":117607,"temperature":0.7,"pith_summary":"DTRNet is an attempt to break the quadratic cost of Transformer attention without making tokens skip their updates. The paper's central claim is that a small learned router at each layer can send roughly 10% of tokens to full multi-head attention and route the remaining 90% through a cheap, token-local projection path, while keeping language-model accuracy at the level of a dense Transformer. The key to holding accuracy is that bypassed tokens still pass through shared value/output projections and then the MLP, so every token is updated at every layer. Against routing baselines that skip whole layers, DTRNet reports higher average accuracy at the same FLOP budget, lower KV-cache memory, and a growing FLOPs advantage as sequence length increases. The authors take this as evidence that attention and token updates can be decoupled, with most tokens not needing cross-token mixing most of the time.","feed_headline":"Only 10% of tokens get full attention; model accuracy holds","feed_subtitle":"The routed model matches a dense Transformer on benchmarks while cutting FLOPs and KV-cache memory.","key_machinery":"The DTRNet layer, interleaved with full Transformer layers. Its router is a two-layer SiLU MLP outputting softmax scores over two paths; a hard argmax picks the path at inference. The attention path is standard causal multi-head self-attention. The bypass path is the core mechanism: x_i W_V W_O, a token-local update using the same value and output projections as attention, followed by the shared MLP. Because the bypass uses the same W_V and W_O, skipped-token representations stay in the same space as attended-token representations. In matrix form, routing induces a dynamic sparse attention mask M = delta delta^T, which restricts attention to the routed tokens and turns the quadratic part of","core_discovery":"The paper's core discovery claim is that a Transformer can keep its accuracy while making self-attention a sparse, token-selective operation rather than a uniform one. In each DTRNet layer, a two-layer router scores each token for one of two paths: full multi-head attention, or a linear path x W_V W_O followed by the shared MLP. At inference the router makes a hard argmax choice; during training it uses the soft scores to weight both paths so gradients reach the router. With an L1 penalty on attention load designed to balance routing across layers, a trained 360M model routes only about 10% of tokens to attention per DTRNet block and averages 44.36 accuracy across eight benchmarks, versus 44","pith_inferences":["Implicit consequence: if most tokens genuinely need only a token-local projection plus MLP, the same idea could stack with linear-attention or sparse-window mechanisms on the attended path, pushing the quadratic share even lower.","The layerwise cosine-similarity redundancy the authors measure on a dense model suggests an adaptive budget policy: allocate more attention to boundary layers and less to the high-similarity middle layers, rather than a fixed interleaving pattern.","A testable extension is to make the routing threshold per-layer and per-input-difficulty at inference, since the trained router already outputs continuous scores that could be thresholded more aggressively under latency constraints.","If the 10% selection is truly driven by token content, DTRNet-style routers might transfer across domains or tasks after fine-tuning, though the paper does not test transfer."],"forward_implications":["At matched FLOPs (about 0.84-0.85 of a dense Transformer), DTRNet reaches or slightly beats the dense baseline's average accuracy on eight zero-shot tasks, so compute can be cut without a corresponding accuracy cut.","Because only about 10% of tokens are attended per DTRNet layer, the quadratic share of FLOPs shrinks as sequence length grows; the paper reports a ratio of 0.785 at 20K tokens.","KV-cache memory drops because unselected tokens never create key/value entries; the reported memory advantage grows with input length, unlike the masking-based eviction strategy used by D-LLM.","Uniform routing load across layers avoids D-LLM's layer starvation and MoD's train/inference router mismatch, making the efficiency stable at scale.","The design principle of retaining the MLP and giving bypassed tokens an explicit W_V W_O update is what preserves performance; ablations show removing the projections drops average accuracy from 44.36 to 41.99."],"supporting_citations":[{"why":"Establishes the Transformer and self-attention architecture whose quadratic cost DTRNet targets and whose layer structure DTRNet modifies.","marker":"Vaswani et al. 2017"},{"why":"Supplies the Mixture-of-Depths baseline that DTRNet compares against and the alternating-layer routing pattern DTRNet adapts.","marker":"Raposo et al. 2024"},{"why":"Supplies the D-LLM baseline whose per-layer gating and layer-starvation failure mode motivate DTRNet's load-balancing penalty.","marker":"Jiang et al. 2024"},{"why":"Supports the claim that attention retrieves while MLP memorizes, motivating the design choice to keep the MLP for all tokens.","marker":"Dong et al. 2025"},{"why":"Provides WikiText, the corpus used for the layerwise cosine-similarity redundancy analysis and for perplexity evaluation.","marker":"Merity et al. 2016"},{"why":"Provides FineWeb-Edu, the training corpus used for all DTRNet and baseline models.","marker":"Penedo et al. 2024"},{"why":"FlashAttention-2's variable-length attention function is what makes efficient batched training of DTRNet possible.","marker":"Dao 2023"},{"why":"YaRN is applied to extrapolate the models to 20K-token sequences for the long-context evaluation.","marker":"Peng et al. 2023"},{"why":"Cited as the basis for weighting both paths by soft router scores during training so gradients reach the router.","marker":"Shazeer et al. 2017"},{"why":"Supplies the expert-choice routing variant used in the ablation that motivates DTRNet's token-choice design.","marker":"Zhou et al. 2022"}],"fun_headline_variants":["Skip attention for 90% of tokens and keep accuracy","Transformer routes 10% of tokens to attention, saves FLOPs","Dynamic token routing cuts quadratic cost, keeps quality","Let most tokens skip attention: DTRNet efficiency","Sparse attention via routing: only 1 in 10 tokens needs it"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The design assumes most tokens in a layer can get what they need from a token-local projection followed by the shared MLP, without cross-token mixing, and that the router learned under soft weighting at training length still picks correctly under hard routing at much longer sequences.","fun_headline_variants_meta":{"raw":{"variants":["Skip attention for 90% of tokens and keep accuracy","Transformer routes 10% of tokens to attention, saves FLOPs","Dynamic token routing cuts quadratic cost, keeps quality","Let most tokens skip attention: DTRNet efficiency","Sparse attention via routing: only 1 in 10 tokens needs it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1092,"prompt_tokens":744,"completion_tokens":348,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":263}},"tokens_in":488,"tokens_out":348,"duration_ms":4430,"temperature":1.0,"reasoning_tokens":263,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:05:22.453360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained 360M DTRNet and retrain a variant where the router is forced to pick a random 10% of tokens per layer, keeping the same training budget and architecture. If the random-10% model matches DTRNet's 44.36 average accuracy instead of dropping, then the learned router is not what preserves performance; if it drops, the router's token selection is load-bearing.","supporting_citations":[{"cited_title":"N.; Kaiser, L.; and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Establishes the Transformer and self-attention architecture whose quadratic cost DTRNet targets and whose layer structure DTRNet modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the D-LLM baseline whose per-layer gating and layer-starvation failure mode motivate DTRNet's load-balancing penalty."},{"cited_title":"M.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q","cited_arxiv_id":null,"evidence_quote":"Cited as the basis for weighting both paths by soft router scores during training so gradients reach the router."},{"cited_title":"M.; Le, Q","cited_arxiv_id":null,"evidence_quote":"Supplies the expert-choice routing variant used in the ablation that motivates DTRNet's token-choice design."}],"review_version":1}