{"id":"abd2205d-2f91-4ecb-94be-332ec08a5fc3","arxiv_id":"2605.23911","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.5,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A pure-Triton fused MoE dispatch kernel reaches 89–131% of CUDA Megablocks throughput at inference batch sizes and passes all tests on both NVIDIA A100 and AMD MI300X with zero code changes.","lead":"TritonMoE is a portable Triton kernel that fuses MoE routing, token permutation, expert GEMMs and weighted combination without any CUDA. It matches or beats Megablocks at small inference batches and runs unchanged on AMD MI300X, showing vendor-locked MoE kernels are no longer required for competitive serving.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the paper's own stated limitation on fixed BLOCK_M under skew.","rationale":"The strongest claim is a carefully scoped systems result: pure-Triton fused MoE dispatch reaches 89–131% of Megablocks at inference batch sizes and is portable to AMD with zero code changes. The experimental support (ablation Table 4, roofline, controlled skew study, 162 tests) is transparent; the authors themselves surface the fixed-BLOCK_M failure mode and list it as future work. Because that limitation is already measured and does not falsify the headline numbers under the distributions used for the main claim, no further load-bearing concern arises. The reader's CONDITIONAL verdict already reflects exactly this residual caveat plus missing AMD performance numbers; no adjustment is warranted.","tokens_in":9562,"tokens_out":503,"duration_ms":6546,"concrete_test":"Re-run the exact Table 2 / Figure 3 protocol on Mixtral-8x7B and Qwen2-MoE at 512 tokens with the same synthetic Zipfian assignments, but replace the static schedule with a simple dynamic block-to-expert map (recompute blocks after sorting expert loads); if the α=2.0 Qwen2 speedup recovers above 0.95× while Mixtral numbers stay within 5%, the paper's own diagnosis is confirmed and the CONDITIONAL caveat is precisely localized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption correctly flags the fixed BLOCK_M schedule (Algorithm 1, §3.2) as the softest point: under Zipfian α=2.0 on Qwen2-MoE (64 experts) the relative speedup falls from 1.03× to 0.70× because Megablocks consolidates the dominant expert while Triton latency stays flat (§4.7). That regime is already quantified and scoped by the authors; it does not undermine the central claim of 89–131% of Megablocks at ≤512 tokens under the natural/near-uniform routing that dominates the main tables, nor the zero-change AMD correctness result. No deeper internal inconsistency, missing control, or unacknowledged failure mode appears to be load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents TritonMoE, a complete MoE forward-pass dispatch pipeline (router scoring, token permutation, expert GEMMs, weighted unpermutation) implemented solely in OpenAI Triton. The main technical contributions are a block-scheduled grouped GEMM that maps program blocks to (expert, token-offset) pairs (Algorithm 1) and a fused gate+up SwiGLU kernel that shares L2-cached input tiles and performs SiLU in registers, with a byte-count argument claiming ~35% less global memory traffic. On A100 the system reports 89–131% of Megablocks throughput at inference batch sizes (≤512 tokens) across Mixtral-8x7B/22B, Qwen2-MoE and (unfused/fused) DeepSeek-V3 configurations; all 162 correctness tests pass unchanged on AMD MI300X. A controlled Zipfian-skew study (§4.7) quantifies the regime (64+ experts, extreme skew) where fixed BLOCK_M scheduling loses to Megablocks’ block-sparse layout.","tokens_in":9703,"tokens_out":1016,"duration_ms":8402,"significance":"If the reported numbers hold, the work shows that a pure-Triton MoE dispatch stack can match or beat a mature CUDA baseline at the batch sizes that matter for interactive serving, while remaining portable to AMD with zero source changes. That combination is practically useful: it lowers the barrier to multi-vendor MoE inference and supplies a self-contained, open implementation (code link provided) rather than a framework-internal kernel. The fusion ablation, roofline breakdowns and explicit skew characterization are concrete engineering contributions that other portable-kernel efforts can build on. The fixed-tile limitation under extreme imbalance is already scoped by the authors and does not erase the small-batch / near-uniform result.","major_comments":[{"comment":"§4.2 / Tables 2–3: the headline claim of 89–131% of Megablocks is supported for Mixtral and Qwen2, but DeepSeek-V3 (256 experts) has no Megablocks baseline at all—only an unfused-vs-fused Triton comparison. The abstract and introduction still list DeepSeek-V3 among the configurations for which the relative-throughput claim is made. Either obtain a runnable Megablocks (or equivalent CUDA) number for that configuration, or rephrase the claim so that the 89–131% range applies only to the configurations that actually have a Megablocks comparator.","section":null},{"comment":"§4.6: cross-platform validation is correctness-only (162 tests on MI300X). The central portability claim is therefore only half-demonstrated; without even a single latency or bandwidth number on AMD it is impossible to know whether the same fused schedule remains competitive once the memory hierarchy and tensor-core mapping change. A minimal performance table (or an explicit statement that AMD performance is out of scope and why) is needed before the portability result can be treated as load-bearing.","section":null}],"minor_comments":[{"comment":"§3.3 memory-traffic analysis: the unfused formula counts 8TF + 4Td bytes; a short parenthetical confirming the exact read/write sequence (including the final intermediate write) would make the 35% figure easier to reproduce.","section":null},{"comment":"Table 5: d_ffn is scaled down as expert count rises “to approximate constant total compute.” State the exact FLOP target used so readers can verify the scaling.","section":null},{"comment":"Figure 1 is referenced as a two-panel roofline but the manuscript text does not include axis units or peak lines; ensure the camera-ready figure is self-contained.","section":null},{"comment":"Related Work: vLLM’s Triton fused MoE is mentioned; a one-sentence note on whether its kernel is available outside the serving stack (or how TritonMoE differs in schedule/fusion) would help readers place the contribution.","section":null},{"comment":"Limitations (5): the host-side block-schedule construction is listed; quantify the CPU time or the resulting host–device sync cost relative to kernel time so the overhead is not left qualitative.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Solid systems paper with open code and honest self-critique of the fixed-BLOCK_M regime. The two major points are presentation/scope fixes rather than soundness failures; once the DeepSeek claim is tightened and AMD performance is either measured or explicitly scoped out, the manuscript is ready. Fit for a systems / MLSys venue is good; pure-theory venues would find it too engineering-heavy."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a solid systems artifact paper. The new thing is a self-contained pure-Triton MoE forward path (router, permute, fused gate+up, down, unpermute) that hits 89–131% of Megablocks at ≤512 tokens on Mixtral/Qwen configs and passes 162 correctness tests on both A100 and MI300X with zero code changes. That cross-platform result is real and useful; most of the literature is still CUDA-locked.\n\nWhat they do well: the fused gate+up design is clean (shared A-tile loads, in-register SiLU, clear 35% traffic math), the block-scheduled grouped GEMM is a practical way around Triton’s missing grouped primitive, and the experimental suite is careful—ablation, roofline, expert scaling, and a controlled Zipfian skew study. They do not hide the failure mode: under α=2.0 on 64-expert Qwen2 their latency stays flat while Megablocks accelerates, so relative speedup drops to 0.70×. That is exactly the fixed BLOCK_M schedule limitation, and they flag dynamic block-to-expert assignment as future work. DeepSeek-V3 is reported only against their own unfused baseline because Megablocks is fragile there; that is honest rather than evasive. Code is shipped.\n\nSoft spots are proportionate, not fatal. AMD is correctness-only (no perf numbers). The schedule is static and host-side. Fusion stops before the down projection. None of these overturn the central claim for the batch sizes that matter for interactive serving under near-uniform routing. Novelty is moderate—vLLM already has a Triton MoE kernel—but this one is standalone, fully instrumented, and portable. Citations look standard; no circularity.\n\nThis is for people who care about MoE serving kernels, vendor lock-in, and AMD datacenter paths. It deserves a serious referee. I would engage with it, cite the portable numbers and the skew characterization, and keep an eye on the dynamic-scheduling follow-up.","headline":"Clean portable Triton MoE dispatch that is competitive with Megablocks at serving batch sizes and actually runs unchanged on MI300X; the fixed-tile schedule is the known soft spot the authors already measure.","tokens_in":10348,"tokens_out":518,"would_cite":true,"duration_ms":5834,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A fully portable Triton MoE dispatch kernel matches CUDA Megablocks at inference batch sizes and runs unchanged on AMD.","keywords":["Mixture-of-Experts","Triton","MoE dispatch","cross-platform GPU","fused GEMM","SwiGLU","inference kernels","block-sparse scheduling"],"falsifier":"Measure end-to-end latency of TritonMoE versus Megablocks on a 64-expert top-4 configuration under Zipfian α=2.0 routing at 128–512 tokens; if the Triton kernel’s relative speedup stays near or above 1.0 instead of collapsing toward 0.7\times, the fixed-tile assumption holds.","tokens_in":10417,"feed_emoji":"⚙️","tokens_out":927,"duration_ms":10281,"temperature":0.7,"pith_summary":"Mixture-of-Experts layers are the backbone of most frontier language models, but their inference path is messy: tokens must be scored, reordered onto experts, run through variable-size GEMMs, then scattered back with weights. Almost every high-performance implementation of that path is written in CUDA and therefore locked to NVIDIA GPUs. This paper shows that the same complete forward path can be written entirely in OpenAI Triton, using only portable primitives. The resulting TritonMoE kernel reaches 89–131 percent of the throughput of the leading CUDA baseline (Megablocks) at the batch sizes typical of interactive serving, while every correctness test passes on both NVIDIA A100 and AMD MI300X with zero source changes. The decisive engineering move is a fused gate-plus-up projection that reuses the same L2-resident input tiles and finishes the SiLU activation in registers, cutting roughly 35 percent of global memory traffic. The work therefore argues that vendor-specific CUDA is no longer a prerequisite for competitive MoE inference kernels.","feed_headline":"Portable Triton MoE kernel matches CUDA at serving batches","feed_subtitle":"Runs unchanged on NVIDIA and AMD; fused gate+up cuts 35% memory traffic","key_machinery":"The fused gate+up GEMM: both SwiGLU projections are computed from a single shared load of each input tile, with the SiLU activation performed in registers so that the two intermediate buffers never touch global memory; the rest of the pipeline is a block-scheduled grouped GEMM that maps fixed-size program blocks onto (expert, token-offset) pairs.","core_discovery":"TritonMoE, a five-kernel MoE dispatch pipeline written solely in portable Triton, delivers 89–131 percent of CUDA Megablocks throughput at inference batch sizes (≤512 tokens) across Mixtral, DeepSeek-V3 and Qwen2-MoE configurations, and the identical source passes all 162 correctness tests on both NVIDIA A100 and AMD MI300X.","pith_inferences":["Once the same source is proven correct on both vendors, the remaining performance gap on AMD becomes pure optimization work rather than a portability research problem.","The same fused-tile pattern should transfer directly to other dual-projection activations (e.g., GEGLU) with only the element-wise op changed.","If dynamic block assignment closes the skew gap, portable Triton could become the default MoE backend even for training-scale expert counts."],"forward_implications":["Interactive LLM serving stacks can ship a single MoE kernel binary that runs on both NVIDIA and AMD accelerators without platform forks.","At the small-to-medium batch sizes that dominate chat serving, portable Triton kernels can already match or beat hand-tuned CUDA dispatch.","The 35 percent memory-traffic reduction from gate+up fusion is available to any SwiGLU MoE that can share input tiles.","Beyond roughly 64 experts under heavy skew, fixed tile schedules become a liability, so future kernels must adopt dynamic block-to-expert assignment."],"fun_headline_variants":["Triton MoE matches CUDA Megablocks at ≤512-token batches","Portable fused Triton MoE equals CUDA on NVIDIA and AMD","Cross-platform Triton MoE hits 89-131% of CUDA throughput","Fused gate+up Triton MoE rivals CUDA under serving loads","Same Triton MoE source passes tests on A100 and MI300X"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That a compile-time fixed tile size and a static expert-to-block schedule remain competitive when real production traffic is heavily skewed toward a few experts.","fun_headline_variants_meta":{"raw":{"variants":["Triton MoE matches CUDA Megablocks at ≤512-token batches","Portable fused Triton MoE equals CUDA on NVIDIA and AMD","Cross-platform Triton MoE hits 89-131% of CUDA throughput","Fused gate+up Triton MoE rivals CUDA under serving loads","Same Triton MoE source passes tests on A100 and MI300X"]},"model":"grok-4.5","effort":"low","cost_usd":0.00375,"raw_usage":{"total_tokens":1251,"prompt_tokens":843,"num_sources_used":0,"completion_tokens":79,"cost_in_usd_ticks":37500000,"prompt_tokens_details":{"text_tokens":843,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":329,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":843,"tokens_out":79,"duration_ms":4461,"temperature":1.0,"reasoning_tokens":329,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T09:08:35.554634+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Measure end-to-end latency of TritonMoE versus Megablocks on a 64-expert top-4 configuration under Zipfian α=2.0 routing at 128–512 tokens; if the Triton kernel’s relative speedup stays near or above 1.0 instead of collapsing toward 0.7\times, the fixed-tile assumption holds.","supporting_citations":[],"review_version":1}