Pith. sign in

REVIEW 4 major objections 6 minor 31 references

A single learned draft expert per layer restores cheap, high-acceptance speculative decoding when MoE experts live off the accelerator.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 14:57 UTC pith:U7ZOKLQL

load-bearing objection Solid draft-expert idea and clean ablations, but the 1.45× rests partly on a no-cache AR baseline that DraftExpert itself is allowed to escape. the 4 major comments →

arxiv 2607.24434 v1 pith:U7ZOKLQL submitted 2026-07-27 cs.LG cs.AI

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Dengke Han This is my paper
classification cs.LG cs.AI
keywords Mixture-of-Expertsspeculative decodingexpert offloadingself-distillationend-device inferenceprefetchingdraft experts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On phones and consumer GPUs, large Mixture-of-Experts language models keep most expert weights in slower memory and load them only when needed. That loading cost breaks ordinary speculative decoding: a better draft loads more experts, a cheap draft is rarely accepted, and checking a block of candidates can pull in a large union of experts instead of one cheap step. DraftExpert trains one small accelerator-resident draft expert per layer to stand in for the missing experts, using only residual, token, and router signals from the frozen target model. At run time it drafts with a fixed shared+top-1+draft-expert path, stops early when a low-confidence token would expand the verifier’s expert set too much, and prefetches the experts the draft router predicts. Final tokens are still checked exactly by the full target. Across two models and both CPU-to-GPU and Flash-to-NPU offload, this raises acceptance into the mid-80s, hits most prefetches, and lifts decode throughput by about 1.45× over plain offloaded decoding.

Core claim

Under expert offload, speculative speed is governed by accepted tokens per expert-set expansion, not tokens per target forward. A fixed-footprint shared+top-1+draft-expert drafter, trained by residual/logit/token and router-agreement self-distillation from the frozen MoE, restores cheap drafting and high acceptance; the same router alignment then drives confidence–expansion truncation and target-expert prefetch so exact verification stays controllable.

What carries the argument

The fixed-footprint shared+top-1+draft-expert path: one lightweight accelerator-resident draft expert per MoE layer approximates the residual of omitted top-k routed experts, so draft cost stays near top-1 while self-distillation recovers acceptance and router agreement for truncation and prefetch.

Load-bearing premise

One small learned draft expert per layer, trained only on residual, token, and router signals from unlabeled text, is enough to replace the missing experts so acceptance and prefetch stay high without loading more routed experts during drafting.

What would settle it

On the same DeepSeek-V2-Lite and Moonlight setups, measure draft acceptance and end-to-end offloaded TPS when the draft expert is removed or trained without residual and router losses; if acceptance falls back near the top-1/top-3 baselines (~22–46%) and the 1.45× speedup disappears, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • End-device MoE runtimes can keep routed experts in CPU or Flash and still use exact speculative decoding without a second draft model.
  • Draft cost need not grow with top-r: accuracy can come from a resident draft expert instead of loading more offloaded experts.
  • Router agreement becomes a first-class training target because it directly sets prefetch hit rate and verify-side expansion control.
  • Dynamic draft length should gate on predicted new verifier experts, not confidence alone.
  • Reported gains hold across CPU→GPU BF16 and Flash→NPU Q4_0 single-user decode, with acceptance 84–87% and prefetch hits 86–88%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same residual-plus-router distillation recipe may transfer to other sparse modules whose activation sets expand under multi-token verify, not only MoE FFNs.
  • If draft experts remain tiny relative to the routed pool, memory-constrained devices can treat them as permanent residents much like shared experts and attention.
  • Serving stacks that already prefetch experts could plug in draft-router scores as a drop-in lookahead without changing target weights.
  • Longer contexts or MoE topologies with weaker residual structure would be the natural stress test for whether one draft expert per layer saturates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies speculative decoding for MoE models on end devices where routed experts are offloaded (CPU→GPU or Flash→NPU) and expert movement, not arithmetic, dominates decode latency. It argues that in this regime speculative decoding's three classical conditions (cheap draft, near-free verification, high acceptance) are governed by expert-set expansion rather than token counts. The proposed system, DraftExpert, (i) trains one small accelerator-resident draft expert per MoE layer via self-distillation from the frozen target (residual, logit/token, and router-agreement losses), enabling a fixed-footprint shared+top-1+draft-expert drafter; (ii) truncates draft blocks using a confidence–marginal-expansion controller; and (iii) prefetches predicted verifier experts using the draft router. On DeepSeek-V2-Lite and Moonlight-16B-A3B across both offload hierarchies, the authors report 1.45x average decode throughput over autoregressive expert-offloaded decoding, 84–87% draft acceptance, and 86–88% prefetch hit rates, with exact target-model outputs preserved by full verification.

Significance. If the results hold, the paper makes a useful contribution to an increasingly relevant deployment regime: single-user, latency-critical MoE inference where routed experts live off-accelerator. The expert-set-expansion reframing of the three speculative-decoding conditions is conceptually clean and the fixed-footprint draft expert plus router-agreement distillation is a natural, well-motivated design. The work has concrete strengths: exactness is preserved by construction (all emitted tokens are verified by the frozen target MoE), the evaluation spans two models and two genuinely different memory hierarchies (PCIe CPU→GPU BF16 and Flash→NPU Q4_0 on a real Hexagon HTP), and the loss ablation (Table 5) cleanly separates residual, logit/token, and router contributions, including the non-obvious finding that router-agreement loss is what drives prefetch hit rate from 73% to 88% while leaving acceptance flat. The truncation and prefetch policies are simple, exactness-preserving, and deployable. However, the quantitative significance of the 1.45x headline cannot be assessed until the baseline issue in Major Comment 1 is resolved, since the resident-memory AR reference on MN (14.36 TPS vs.

major comments (4)
  1. [Setup and Metrics / Table 2] The 1.45x headline speedup (Table 2) is normalized against an AR-offload baseline that is explicitly denied any persistent expert cache: 'Routed target experts are not kept in a persistent accelerator cache unless an experiment explicitly studies the resident-memory reference' (Setup). DraftExpert's own Algorithm 1 (lines 2, 13-14) maintains a cache C and in-flight prefetch set P. The manuscript's own data suggest this asymmetry may account for much of the gain: Figure 5 shows ~59% cumulative layer-expert coverage at K=16 (expert reuse is high, so an LRU cache warms quickly), and on MN the resident-memory AR reference reaches 14.36 TPS versus DraftExpert's 15.47 TPS (Table 1/Table 2) — once expert movement is hidden by any mechanism, the learned-drafter contribution on that platform appears to be ~8%. The cited offload systems (MoE-Infinity, EdgeMoE, Fiddler) all use expert caching/prefe
  2. [Tables 3 and 5] The acceptance numbers for the top-1 drafter are inconsistent across tables: Table 3 reports 22.5% (DS) / 21.8% (ML) acceptance at K=4, while Table 5 reports the 'Top-1' row at 47.7% (DS) / 45.0% (ML). Since the headline '84-87% acceptance' claim is supported by these same tables (85.0%/83.0% in Table 3; 87.1%/84.5% in Table 5), the discrepancy is not cosmetic. Plausibly Table 5 measures per-token top-1 agreement on held-out text while Table 3 measures block-level accepted/drafted ratio, but this is never stated, and the two metrics are conflated in the text ('acceptance' is used for both). Please define each metric precisely, state the evaluation protocol (prompts, K, teacher-forced vs. free-running), and reconcile the numbers.
  3. [Self-Distillation (Eqs. 5-11)] The distillation procedure is underspecified for a learned-method paper. Eq. (11) introduces weights alpha, beta, eta and temperature T, and Eq. (5) introduces per-layer scale gamma_l and a draft-expert architecture, but none of these values, the draft-expert capacity/parameter count, the training corpus size, training steps, or wall-clock training cost are reported anywhere. Given that the added per-layer draft expert is a new architectural component whose footprint matters for the central 'fixed-footprint' claim, its size (absolute and relative to one routed expert) must be reported, and a sensitivity check on the loss weights (at least showing robustness of the Table 5 ranking) is needed.
  4. [Experimental Evaluation / Tables 2-4] Throughput numbers in Tables 2-4 are single point estimates with no error bars, repetition counts, or multi-seed variance, despite known run-to-run variance in offload pipelines (PCIe contention, Flash staging, NPU scheduling) and in distillation training. The DS CG margin (2.99 vs 2.19 TPS) is comfortable, but the ML CG margin (2.50 vs 1.94 TPS) and the comparison against the 14.36 TPS resident-memory reference deserve variance quantification. Please report means and standard deviations over multiple runs, and at least two training seeds for the distilled checkpoint.
minor comments (6)
  1. [Cost-Aware Dynamic Truncation (Eq. 14)] B_i in Eq. (14) is described as 'a fixed layer-expert budget or a budget derived from the current expert cache,' but the experiments never state which was used or its value. Please specify the budget policy and threshold tau used in Tables 2 and 4.
  2. [Table 2] Table 2's 'top-r TPS' column is labeled the 'best training-free shared+top-r exact self-drafting setting,' yet all entries are slower than AR offload (e.g., 1.19 vs 2.19 on DS CG), which means the best training-free baseline is effectively AR itself. This is a legitimate finding but should be stated explicitly, including which r won the sweep per platform.
  3. [Algorithm 1] Algorithm 1, line 7: Delta_i is computed against the not-yet-updated U-hat, which matches Eq. (13), but the variable reuse (U-hat before vs. after update) is easy to misread; a brief comment or renaming would help.
  4. [Figure 1] Figure 1(b) reports 'decode-128 profiled time' but the profiling protocol (prompt mix, cold vs. warm cache state) is not described; given the cold-cache emphasis in Figure 2, clarify whether Figure 1(b) is cold-start or steady-state.
  5. [Table 4] Prefetch waste (12-14%, Table 4) is only reported as hit/waste rate; a sentence quantifying its bandwidth/latency cost on the MN platform, where Flash bandwidth is the bottleneck, would strengthen the verify-side analysis.
  6. [References] Several references are dated 2026 with arXiv numbers (e.g., Bang et al. 2026, McDanel et al. 2026, Li et al. 2026); please verify these are publicly accessible so reviewers and readers can check the positioning claims in Table 6.

Circularity Check

0 steps flagged

No significant circularity: throughput, acceptance, and prefetch claims are empirical measurements, not identities of the training losses or self-cited premises.

full rationale

DraftExpert is an empirical systems paper. Its load-bearing claims (1.45× decode TPS vs AR offload, 84–87% acceptance, 86–88% prefetch hits; Tables 2–5) are measured end-to-end against frozen-target verification and training-free top-r baselines, not algebraically forced by the residual/logit/token/router losses. Self-distillation uses the frozen full MoE as teacher on unlabeled text (Eqs. 6–11); acceptance and router hit rate remain free outcomes of that training, as the ablation (Table 5) shows stepwise gains rather than tautologies. Fixed-footprint drafting (Eq. 5), confidence–expansion truncation (Eq. 14), and draft-router prefetch are design choices evaluated experimentally; none redefine the reported metrics in terms of themselves. Citations (Leviathan et al., MoE-Infinity, EdgeMoE, Fiddler, etc.) are external prior art, not author-overlapping uniqueness theorems that force the result. Baseline-fairness concerns (no-cache AR vs cached/prefetched DraftExpert) affect attribution of speedup, not circularity of the derivation. The paper is self-contained against external benchmarks with no fitted-input-as-prediction or self-definitional loop.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central empirical claim rests on standard transformer/MoE algebra, the domain premise that routed experts dominate memory and are loaded on demand, and several paper-specific modeling choices (one draft expert per layer, particular multi-term distillation objective, confidence–expansion gate). No physical constants or deep unproved lemmas; free parameters are the usual training and controller knobs that are only partially reported.

free parameters (4)
  • loss weights α, β, η
    Coefficients balancing logit, token, residual, and router-agreement terms in Eq. (11); values not stated.
  • confidence threshold τ and expansion budget Bi
    Controller stops drafting when qi < τ and Δi > Bi (Eq. 14); concrete schedules or values omitted.
  • draft-expert capacity / scale γl
    Width, initialization, and optional per-layer scale of the resident draft expert are unspecified beyond ‘lightweight’.
  • distillation temperature T
    Temperature in the KL logit loss (Eq. 8); standard but unreported.
axioms (5)
  • domain assumption Routed experts dominate parameter count and must be staged on demand from CPU or Flash while non-routed weights stay accelerator-resident.
    Stated in Introduction and Figure 1; underpins the entire cost model.
  • standard math Standard MoE forward: y = shared + Σ g_e E_e(h) over top-k router selections.
    Eq. (4); conventional sparse MoE formulation.
  • ad hoc to paper A single learned draft expert can approximate the residual of the omitted top-k experts well enough for high token and router agreement.
    Core modeling choice in Eqs. (5)–(7) and training objective; validated only by the reported ablations.
  • ad hoc to paper Draft-router predictions of verifier expert sets remain accurate enough after distillation to support useful prefetch and truncation.
    Enables the verify-side policies; supported by Table 5 hit-rate column but not proved in general.
  • domain assumption Evaluation under cold expert cache / single-user latency-critical decode is the relevant regime.
    Explicitly adopted for Figures 2–3 and throughput tables; warm-cache or batched settings may differ.
invented entities (2)
  • Per-layer accelerator-resident draft expert Ed_l no independent evidence
    purpose: Recover omitted routed-expert residual at fixed memory/compute footprint so drafting stays cheap while acceptance rises.
    Introduced in Fixed-Footprint Drafting; trained via self-distillation; no existence claim beyond the engineered module.
  • Confidence–expansion truncation controller no independent evidence
    purpose: Stop drafting when a low-confidence token would add too many new verifier experts.
    Defined by Eq. (14) and Algorithm 1; policy invention rather than a physical entity.

pith-pipeline@v1.2.0-grok45-kimik3 · 17805 in / 3556 out tokens · 53511 ms · 2026-07-31T14:57:29.868956+00:00 · methodology

0 comments
read the original abstract

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU. In this setting, self-speculative decoding faces a new bottleneck: increasing the draft expert set improves accuracy but triggers extra expert loading, while cheap small-footprint drafts have low acceptance; moreover, verifying a multi-token block activates the union of target experts and is no longer close to one target step. We propose DraftExpert, an expansion-aware self-speculative decoding framework for expert-offloaded MoE inference. DraftExpert trains one lightweight accelerator-resident draft expert per layer by self-distilling residual, logit/token, and router-agreement signals from the frozen target MoE. At inference time, it uses a fixed-footprint shared+top-1+draft-expert drafter together with confidence--expansion truncation and target-expert prefetching, while final tokens are still exactly verified by the target model. On DeepSeek-V2-Lite and Moonlight-16B-A3B across CPU-GPU and Flash-NPU offload, DraftExpert improves decode throughput by 1.45x on average, raises draft acceptance to 84~87%, and achieves 86~88% prefetch hit rates.

Figures

Figures reproduced from arXiv: 2607.24434 by Dengke Han.

Figure 1
Figure 1. Figure 1: Motivation for expert-offloaded MoE inference on end devices. (a) Routed experts dominate DeepSeek-V2-Lite weight [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top-r acceptance–cheapness trade-off under cold expert offloading. Increasing router top-r improves draft ac￾ceptance, but it also makes drafting less cheap: draft latency rises relative to top-1, while draft speedup over the full target falls from 4.7× to 2.0×. Condition 3: Acceptance Requires Learned Drafting One might keep draft and verify expansion small by using only shared+top-1 drafting and short fi… view at source ↗
Figure 3
Figure 3. Figure 3: Draft-side expert-set expansion. Shared+top- [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Dense versus MoE verification under offloaded [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Expansion-aware DraftExpert overview. Expert-offloaded MoE breaks the three classical speculative decoding con [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 6 linked inside Pith

  1. [1]

    2023 , eprint=

    Fast Inference from Transformers via Speculative Decoding , author=. 2023 , eprint=

    2023

  2. [2]

    2023 , eprint=

    Accelerating Large Language Model Decoding with Speculative Sampling , author=. 2023 , eprint=

    2023

  3. [3]

    International Conference on Learning Representations , year=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. International Conference on Learning Representations , year=

  4. [4]

    Journal of Machine Learning Research , volume=

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity , author=. Journal of Machine Learning Research , volume=

  5. [5]

    2602.16052 , archivePrefix=

    Bradley McDanel and Steven Li and Sruthikesh Surineni and Harshit Khaitan , year=. 2602.16052 , archivePrefix=

    arXiv

  6. [6]

    2025 , eprint=

    Utility-Driven Speculative Decoding for Mixture-of-Experts , author=. 2025 , eprint=

    2025

  7. [7]

    Self-Speculative Decoding for On-device

    Peirong Zheng and Wenchao Xu and Haozhao Wang , booktitle=. Self-Speculative Decoding for On-device. 2026 , doi=

    2026

  8. [8]

    2604.10152 , archivePrefix=

    Jehyeon Bang and Eunyeong Cho and Ranggi Hwang and Jinha Chung and Minsoo Rhu , year=. 2604.10152 , archivePrefix=

    Pith/arXiv arXiv

  9. [9]

    2505.19645 , archivePrefix=

    Zongle Huang and Lei Zhu and Zongyuan Zhan and Ting Hu and Weikai Mao and Xianzhi Yu and Yongpan Liu and Tianyu Zhang , year=. 2505.19645 , archivePrefix=

    arXiv

  10. [10]

    2025 , eprint=

    Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding , author=. 2025 , eprint=

    2025

  11. [11]

    2510.10302 , archivePrefix=

    Liangkun Chen and Zijian Wen and Tian Wu and Xiaoxi Zhang and Chuan Wu , year=. 2510.10302 , archivePrefix=

    arXiv

  12. [12]

    2511.14102 , archivePrefix=

    Wenfeng Wang and Jiacheng Liu and Xiaofeng Hou and Xinfeng Xia and Peng Tang and Mingxuan Zhang and Chao Li and Minyi Guo , year=. 2511.14102 , archivePrefix=

    arXiv

  13. [13]

    2603.09983 , archivePrefix=

    Shuhuai Li and Jianghao Lin and Dongdong Ge and Yinyu Ye , year=. 2603.09983 , archivePrefix=

    arXiv

  14. [14]

    Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via

    Haiduo Huang and Fuwei Yang and Zhenhua Liu and Yixing Xu and Jinze Li and Yang Liu and Xuanwu Yin and Dong Li and Pengju Ren and Emad Barsoum , year=. Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via. 2502.06282 , archivePrefix=

    Pith/arXiv arXiv

  15. [15]

    Xin Cheng and Xingkai Yu and Chenze Shao and Jiashi Li and Yunfan Xiong and others , year=

  16. [16]

    Semantic Parallelism: Redefining Efficient

    Yan Li and Zhenyu Zhang and Zhengang Wang and Pengfei Chen and Pengfei Zheng , booktitle=. Semantic Parallelism: Redefining Efficient. 2026 , eprint=

    2026

  17. [17]

    Dmitry Lepikhin and HyoukJoong Lee and Yuanzhong Xu and Dehao Chen and Orhan Firat and Yanping Huang and Maxim Krikun and Noam Shazeer and Zhifeng Chen , booktitle=

  18. [18]

    Nan Du and Yanping Huang and Andrew M. Dai and Simon Tong and Dmitry Lepikhin and Yuanzhong Xu and Maxim Krikun and Yanqi Zhou and Adams Wei Yu and Orhan Firat and Barret Zoph and Liam Fedus and Maarten Bosma and Zongwei Zhou and others , booktitle=

  19. [19]

    2024 , eprint=

    Mixtral of Experts , author=. 2024 , eprint=

    2024

  20. [20]

    Damai Dai and Chengqi Deng and Chenggang Zhao and R. X. Xu and Huazuo Gao and Deli Chen and Jiashi Li and Wangding Zeng and Xingkai Yu and Y. Wu and Zhenda Xie and Y. K. Li and Panpan Huang and Fuli Luo and Chong Ruan and Zhifang Sui and Wenfeng Liang , booktitle=. 2024 , doi=

    2024

  21. [21]

    Muon is Scalable for

    Jingyuan Liu and Jianlin Su and Xingcheng Yao and Zhejun Jiang and Guokun Lai and Yulun Du and Yidao Qin and Weixin Xu and Enzhe Lu and Junjie Yan and Yanru Chen and Huabin Zheng and Yibo Liu and Shaowei Liu and Bohong Yin and Weiran He and Han Zhu and Yuzhi Wang and Jianzhou Wang and Mengnan Dong and Zheng Zhang and Yongsheng Kang and Hao Zhang and Xinra...

    Pith/arXiv arXiv

  22. [22]

    Xupeng Miao and Gabriele Oliaro and Zhihao Zhang and Xinhao Cheng and Zeyu Wang and Zhengxin Zhang and Rae Ying Yee Wong and Alan Zhu and Lijie Yang and Xiaoxiang Shi and Chunan Shi and Zhuoming Chen and Daiyaan Arfeen and Reyna Abhyankar and Zhihao Jia , booktitle=

  23. [23]

    Lee and Deming Chen and Tri Dao , booktitle=

    Tianle Cai and Yuhong Li and Zhengyang Geng and Hongwu Peng and Jason D. Lee and Deming Chen and Tri Dao , booktitle=. Medusa: Simple

  24. [24]

    2401.15077 , archivePrefix=

    Yuhui Li and Fangyun Wei and Chao Zhang and Hongyang Zhang , year=. 2401.15077 , archivePrefix=

    Pith/arXiv arXiv

  25. [25]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages=

    Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages=. 2024 , doi=

    2024

  26. [26]

    Aly and Beidi Chen and Carole-Jean Wu , booktitle=

    Mostafa Elhoushi and Akshat Shrivastava and Diana Liskovich and Basil Hosmer and Bram Wasti and Liangzhen Lai and Anas Mahmoud and Bilge Acun and Saurabh Agarwal and Ahmed Roman and Ahmed A. Aly and Beidi Chen and Carole-Jean Wu , booktitle=. 2024 , doi=

    2024

  27. [27]

    2401.14361 , archivePrefix=

    Leyang Xue and Yao Fu and Zhan Lu and Luo Mai and Mahesh Marina , year=. 2401.14361 , archivePrefix=

    Pith/arXiv arXiv

  28. [28]

    2308.14352 , archivePrefix=

    Rongjie Yi and Liwei Guo and Shiyun Wei and Ao Zhou and Shangguang Wang and Mengwei Xu , year=. 2308.14352 , archivePrefix=

    Pith/arXiv arXiv

  29. [29]

    Fiddler:

    Keisuke Kamahori and Tian Tang and Yile Gu and Kan Zhu and Baris Kasikci , booktitle=. Fiddler:

  30. [30]

    llama.cpp:

    Georgi Gerganov , year=. llama.cpp:

  31. [31]

    NIPS Deep Learning and Representation Learning Workshop , year=

    Distilling the Knowledge in a Neural Network , author=. NIPS Deep Learning and Representation Learning Workshop , year=