arxiv: 2605.14438 · v1 · submitted 2026-05-14 · 💻 cs.AI

Recognition: no theorem link

BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE

Juntong Wu , Jialiang Cheng , Qishen Yin , Yue Dai , Yuliang Yan , Fuyu Lv , Ou Dan , Li Yuan

Authors on Pith no claims yet

Pith reviewed 2026-05-15 01:55 UTC · model grok-4.3

classification 💻 cs.AI

keywords Mixture of ExpertsDynamic RoutingInference EfficiencyBinary MasksSparse ActivationLarge Language ModelsvLLM Integration

0 comments

The pith

Trainable binary masks let MoE models pick experts token-by-token, cutting expert-layer FLOPs by up to 85 percent while keeping more than 98 percent of original accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard MoE routing activates a fixed number of experts for every token, which leaves substantial unused computation even when many experts are redundant for that token. BEAM replaces the fixed top-k rule with per-token binary masks that are learned end-to-end; a straight-through estimator lets gradients pass through the discrete decisions, and an auxiliary loss pushes the masks toward sparsity. The masks remain active at inference, so the same dynamic selection that was trained also runs during decoding. A custom CUDA kernel makes the sparse path fast enough to integrate directly with vLLM. The result is large reductions in FLOPs and wall-clock time with only marginal accuracy loss.

Core claim

BEAM shows that token-adaptive expert activation can be induced by optimizing binary masks inside the existing MoE training loop. The masks are produced by a lightweight gating network whose outputs are binarized; the straight-through estimator plus an auxiliary sparsity penalty lets the model discover which experts are truly needed for each token while the base weights stay largely unchanged. At inference the same masks deliver genuine dynamic routing, yielding the reported FLOPs and latency gains.

What carries the argument

Trainable binary expert activation masks, realized through a straight-through estimator and an auxiliary regularization loss that together replace fixed top-k routing.

If this is right

Expert-layer compute becomes proportional to the actual number of useful experts per token rather than a fixed budget.
Decoding latency improves without any architectural change to the base model or costly retraining.
The same trained masks can be reused across different inference back-ends once the CUDA kernel is in place.
Throughput gains scale with batch size because fewer expert parameters are touched per forward pass.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The approach could be stacked with existing weight pruning or quantization pipelines to compound savings.
If the masks prove stable across domains, the technique might reduce the need for separate sparse fine-tuning stages.
Extending the mask training to also learn which experts to drop entirely could produce permanent model compression.

Load-bearing premise

The sparsity pattern the masks learn on the training distribution will continue to be effective and stable on new inputs seen only at inference.

What would settle it

Measure the fraction of tokens whose activated expert set changes when the same prompt is fed through the model once on training data and once on held-out data; if the change exceeds a few percent while accuracy drops more than two points, the central claim fails.

Figures

Figures reproduced from arXiv: 2605.14438 by Fuyu Lv, Jialiang Cheng, Juntong Wu, Li Yuan, Ou Dan, Qishen Yin, Yue Dai, Yuliang Yan.

**Figure 2.** Figure 2: Vanilla Top-K vs. BEAM: BEAM learns a binary mask over Top-K candidates for token-adaptive activation. In this work, we propose BEAM (Binary Expert Activation Masking), a novel dynamic routing framework designed to achieve extreme expert sparsity and inference speedups in MoE models. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Figure 3: The illustration of our proposed BEAM method with 4 experts and K=3 as an example. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Comparison of TPOT, TTFT, and throughput across different methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 5.** Figure 5: The average number of activated experts per token in BEAM: Qwen3-30B-A3B. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

**Figure 6.** Figure 6: Layer-wise sparsity and position-wise masking analysis. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Training curves of BEAM (β = 0.1) on three MoEs. Blue: language modeling loss (left axis). Orange: expert active rate (right axis). Gray dashed line: SFT baseline loss without BEAM [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗

**Figure 8.** Figure 8: Layer-wise masking rank analysis across three MoE models. The shaded region between [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

**Figure 9.** Figure 9: Expert load balance visualization of MoE models before and after BEAM fine-tuning. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

**Figure 10.** Figure 10: Per-token and per-layer expert activation heatmap for DeepSeekV2-Lite. Each cell [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗

**Figure 11.** Figure 11: Per-token and per-layer expert activation heatmap for Qwen1.5-MoE-A2.7B. Each cell [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗

**Figure 12.** Figure 12: Per-token and per-layer expert activation heatmap for Qwen3-30B-A3B. Each cell indicates [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗

read the original abstract

Mixture-of-Experts (MoE) architectures enhance the efficiency of large language models by activating only a subset of experts per token. However, standard MoE employs a fixed Top-K routing strategy, leading to redundant computation and suboptimal inference latency. Existing acceleration methods either require costly retraining with architectural changes or suffer from severe performance drop at high sparsity due to train-inference mismatch. To address these limitations, we propose BEAM (Binary Expert Activation Masking), a novel method that learns token-adaptive expert selection via trainable binary masks. With a straight-through estimator and an auxiliary regularization loss, BEAM induces dynamic expert sparsity through end-to-end training while maintaining model capability. We further implement an efficient custom CUDA kernel for BEAM, ensuring seamless integration with the vLLM inference framework. Experiments show that BEAM retains over 98\% of the original model's performance while reducing MoE layer FLOPs by up to 85\%, achieving up to 2.5$\times$ faster decoding and 1.4$\times$ higher throughput, demonstrating its effectiveness as a practical, plug-and-play solution for efficient MoE inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

BEAM adds trainable binary masks via straight-through estimator for dynamic MoE sparsity, but the train-inference consistency on expert selection remains the unverified link.

read the letter

The main thing to know is that BEAM trains binary masks on expert activations so each token picks its own sparse subset, using a straight-through estimator plus an auxiliary loss to keep the model capable during end-to-end training. It then hardens the masks for inference and ships a custom CUDA kernel that drops into vLLM. That combination is the concrete new piece: prior work either retrains heavily or accepts big drops at high sparsity, while this tries to get token-adaptive sparsity without those costs. The reported numbers are the part that matters if they hold—over 98% performance retention, 85% MoE FLOPs cut, 2.5x faster decoding, 1.4x throughput. The implementation detail on the kernel is useful because it shows the method is meant to be practical rather than just theoretical. The soft spot is exactly the one the stress-test flags. Straight-through estimators give biased gradients for the binary decisions, so the masks that look good in the training forward pass can select different experts once they are frozen at inference. The abstract gives no overlap statistics between training and inference activations, no ablation on the auxiliary loss weight, and no check on whether the claimed sparsity survives hardening. That leaves the central efficiency claims resting on an assumption that may not be fully closed. The free parameter for the auxiliary loss weight also hints that some tuning was required, which could limit how plug-and-play it really is across models. This is for researchers and engineers who already run large MoE models and need lower inference cost without full retraining. A reader who cares about measurable speedups on real hardware would get value from the kernel and the benchmark setup. It deserves a serious referee because the idea is straightforward to implement and the claims are falsifiable with the right controls. I would send it to peer review but ask the authors to add explicit measurements of mask consistency and sensitivity to the auxiliary loss before acceptance.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes BEAM, a method that learns token-adaptive binary masks for expert selection in MoE layers via end-to-end training with a straight-through estimator and auxiliary regularization loss. It includes a custom CUDA kernel for vLLM integration and claims retention of over 98% of baseline performance while cutting MoE-layer FLOPs by up to 85%, yielding up to 2.5× faster decoding and 1.4× higher throughput as a plug-and-play inference optimization.

Significance. If the central empirical claims hold under scrutiny, BEAM would supply a lightweight, training-compatible route to dynamic sparsity in MoE models that avoids full retraining or architectural overhaul. The reported speedups and high performance retention would be practically relevant for latency-sensitive deployment of large MoE LLMs.

major comments (3)

[§3] §3 (Method), straight-through estimator description: the paper does not quantify the train-inference activation mismatch (e.g., via overlap statistics or KL divergence between training soft masks and hardened inference selections), which directly bears on whether the reported 85% FLOPs reduction and >98% performance retention survive when masks are frozen at inference.
[§5] §5 (Experiments), performance tables: the >98% retention figures are presented without standard deviations across runs, number of random seeds, or statistical significance tests, and no ablation is shown for the auxiliary-loss weight (the sole free hyperparameter), leaving the robustness of the sparsity-performance trade-off unverified.
[§5.3] §5.3 (Inference results): the 2.5× decoding and 1.4× throughput claims rest on the custom CUDA kernel, yet no measurement is given of actual expert activation overlap or FLOPs realized when the learned binary masks are applied in their hardened inference form, undermining the central claim that STE plus auxiliary loss closes the train-inference gap.

minor comments (2)

[Abstract] Abstract: the 'up to' qualifiers for FLOPs reduction and speedups are not tied to specific model sizes, sparsity targets, or benchmark suites, making the headline numbers difficult to interpret.
[§3] Notation: the binary-mask formulation uses an undefined symbol for the temperature parameter in the STE; a brief definition or reference to the exact estimator equation would improve clarity.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. The comments highlight important areas for strengthening the empirical validation of BEAM. We address each major comment below and will incorporate the suggested revisions to improve the manuscript.

read point-by-point responses

Referee: [§3] §3 (Method), straight-through estimator description: the paper does not quantify the train-inference activation mismatch (e.g., via overlap statistics or KL divergence between training soft masks and hardened inference selections), which directly bears on whether the reported 85% FLOPs reduction and >98% performance retention survive when masks are frozen at inference.

Authors: We agree that explicitly quantifying the train-inference mismatch would provide stronger support for the claims. In the revised manuscript, we will add overlap statistics (e.g., Jaccard similarity or activation agreement rate) and KL divergence between the soft masks during training and the hardened binary selections at inference. These metrics will demonstrate that the auxiliary regularization loss effectively minimizes the discrepancy, thereby validating that the reported FLOPs reduction and performance retention hold under the hardened inference regime. revision: yes
Referee: [§5] §5 (Experiments), performance tables: the >98% retention figures are presented without standard deviations across runs, number of random seeds, or statistical significance tests, and no ablation is shown for the auxiliary-loss weight (the sole free hyperparameter), leaving the robustness of the sparsity-performance trade-off unverified.

Authors: We acknowledge the need for greater statistical rigor. We will update all performance tables to report means and standard deviations over multiple random seeds (at least three) and include p-values from appropriate statistical tests for the retention figures. We will also add a dedicated ablation subsection varying the auxiliary-loss weight across a range of values to verify the stability of the sparsity-performance trade-off. revision: yes
Referee: [§5.3] §5.3 (Inference results): the 2.5× decoding and 1.4× throughput claims rest on the custom CUDA kernel, yet no measurement is given of actual expert activation overlap or FLOPs realized when the learned binary masks are applied in their hardened inference form, undermining the central claim that STE plus auxiliary loss closes the train-inference gap.

Authors: The speedups are measured using the custom CUDA kernel that applies the hardened masks at inference. To directly address the concern, we will include new measurements in §5.3 of the realized expert activation rates (overlap with training soft masks) and the actual FLOPs computed under the hardened inference masks. These will confirm that the STE and auxiliary loss close the train-inference gap sufficiently to support the reported throughput gains. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical training and evaluation

full rationale

The paper proposes an empirical training procedure for binary expert masks using straight-through estimator and auxiliary loss, then reports experimental results on performance retention and FLOPs reduction against standard MoE baselines. No mathematical derivation chain exists that reduces predictions or uniqueness claims to fitted inputs by construction, nor any load-bearing self-citation of prior author work that substitutes for independent verification. Claims rest on external benchmarks and measured speedups rather than self-referential definitions or renamed known results.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The paper relies on standard assumptions from machine learning optimization and introduces binary masks as the main new element without new physical entities.

free parameters (1)

auxiliary loss weight
The regularization loss for inducing sparsity likely requires tuning a hyperparameter.

axioms (1)

standard math The straight-through estimator provides a valid gradient approximation for binary mask training
This is a standard technique in training binary neural networks and discrete variables.

pith-pipeline@v0.9.0 · 5519 in / 1130 out tokens · 49818 ms · 2026-05-15T01:55:47.778168+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 9 internal anchors

[1]

Da-moe: Towards dynamic expert allocation for mixture-of-experts models.arXiv preprint arXiv:2409.06669,

Maryam Akhavan Aghdam, Hongpeng Jin, and Yanzhao Wu. Da-moe: Towards dynamic expert allocation for mixture-of-experts models.arXiv preprint arXiv:2409.06669,

work page arXiv
[2]

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

Walaa Amer, Fadi Kurdahi, et al. Conflayers: Adaptive confidence-based layer skipping for self- speculative decoding.arXiv preprint arXiv:2604.14612,

work page internal anchor Pith review Pith/arXiv arXiv
[3]

Qwen Technical Report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

work page internal anchor Pith review Pith/arXiv arXiv
[4]

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv preprint arXiv:1308.3432,

work page internal anchor Pith review Pith/arXiv arXiv
[5]

BoolQ: Exploring the surprising difficulty of natural yes/no questions

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

work page 2019
[6]

Training Verifiers to Solve Math Word Problems

Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300/. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv prepr...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/n19-1300
[7]

Introducing longcat-flash-thinking: A technical report.arXiv preprint arXiv:2509.18883,

Anchun Gui, Bei Li, Bingyang Tao, Bole Zhou, Borun Chen, Chao Zhang, Chengcheng Han, Chenhui Yang, Chi Zhang, et al. Introducing longcat-flash-thinking: A technical report.arXiv preprint arXiv:2509.18883,

work page arXiv
[8]

Dynamic mixture of ex- perts: An auto-tuning approach for efficient transformer models.arXiv preprint arXiv:2405.14297,

Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. Dynamic mixture of ex- perts: An auto-tuning approach for efficient transformer models.arXiv preprint arXiv:2405.14297,

work page arXiv
[9]

Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021a

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.Proceedings of the International Conference on Learning Representations (ICLR), 2021a. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt....

work page arXiv
[10]

Mixtral of Experts

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088,

work page internal anchor Pith review Pith/arXiv arXiv
[11]

Sparsity-controllable dynamic top-p moe for large foundation model pre-training.arXiv preprint arXiv:2512.13996,

Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohiremen Dibua, Yifan Gong, Yan Kang, and Dimitris N Metaxas. Sparsity-controllable dynamic top-p moe for large foundation model pre-training.arXiv preprint arXiv:2512.13996,

work page arXiv
[12]

Moe++: Accelerating mixture-of-experts methods with zero-computation experts.arXiv preprint arXiv:2410.07348,

Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan. Moe++: Accelerating mixture-of-experts methods with zero-computation experts.arXiv preprint arXiv:2410.07348,

work page arXiv
[13]

Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

work page internal anchor Pith review Pith/arXiv arXiv
[14]

Learning to skip the middle layers of transformers.arXiv preprint arXiv:2506.21103,

Tim Lawson and Laurence Aitchison. Learning to skip the middle layers of transformers.arXiv preprint arXiv:2506.21103,

work page arXiv
[15]

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding.arXiv preprint arXiv:2006.16668,

work page internal anchor Pith review Pith/arXiv arXiv 2006
[16]

Cmmlu: Measuring massive multitask language understanding in chinese, 2023a

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023a. Jiamin Li, Qiang Su, Yitao Yang, Yimin Jiang, Cong Wang, and Hong Xu. Adaptive gating in mixture-of-experts based language models.arXiv preprint arXiv:2310.07188, 2023b. Pingzhi ...

work page arXiv
[17]

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.334. URL https://aclanthology.org/2024. acl-long.334/. 11 Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538,

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2024.acl-long.334 2024
[18]

Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts.arXiv preprint arXiv:2407.09816,

Zhenpeng Su, Zijia Lin, Xue Bai, Xing Wu, Yizhe Xiong, Haoran Lian, Guangyuan Ma, Hui Chen, Guiguang Ding, Wei Zhou, et al. Maskmoe: Boosting token-level learning via routing mask in mixture-of-experts.arXiv preprint arXiv:2407.09816,

work page arXiv
[19]

CommonsenseQA: A question answering challenge targeting commonsense knowledge

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, ...

work page 2019
[20]

Qwen3 Technical Report

Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https: //aclanthology.org/N19-1421. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025a. Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Y...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/n19-1421
[21]

Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts

Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruning in sparse mixture-of-experts. InFindings of the Association for Computational Linguistics: ACL 2025, pages 86–102,

work page 2025
[22]

We do not identify any societal impacts specific to the proposed method beyond those already associated with the general use and deployment of large language models

12 A Appendix on Method A.1 Impact Statements This work focuses on improving the computational efficiency of Mixture-of-Experts models. We do not identify any societal impacts specific to the proposed method beyond those already associated with the general use and deployment of large language models. A.2 Limitations Our work has several limitations. First...

work page 2025
[23]

*/ ) { 29//

? o r i g i n a l _ e x p e r t : -1; 25} 26 27te mp lat e < t yp en ame scalar_t , t yp en ame token_cnts_t > 28_ _ g l o b a l _ _ void m o e _ a l i g n _ b l o c k _ s i z e _ k e r n e l ( /* ... */ ) { 29// ... thread setup ... 30for ( int i = s t a r t _ i d x ; i < end_idx ; ++ i ) { 31int64_t e x p e r t _ i d = to pk _i ds [ i ]; 32if ( e x p e ...

work page 2023
[24]

B.1.2 Hyper-Parameters Tables 6 summarize the main configurations for all MoE models studied in this work. All trainings and evaluations are performed on NVIDIA H20 GPUs B.1.3 Benchmarks For accuracy comparison, we select a diverse set of tasks from the OpenCompass [Contributors, 2023] benchmark, covering multiple domains: Reasoning (MATH [Hendrycks et al...

work page 2023
[25]

For acceleration comparison, we use vLLM [Kwon et al., 2023] as the inference framework. Each model is deployed on a single GPU, and we record theTime per Output Token(TPOT, in ms) across differentQueries per Second(QPS), theTime To First Token(TTFT, in ms) in 32 QPS (high- computing scenarios), and the offlineThroughput(samples/s). The input and output s...

work page 2023