Pith. sign in

REVIEW 3 major objections 6 minor 12 references

A hybrid MoE parent model can be cut to 75B total / 9B active parameters and still serve about twice as many tokens per second under tight per-user latency targets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 19:40 UTC pith:NVICEFTE

load-bearing objection Solid systems paper: ~2× interactive throughput and 1→8 1M-context concurrency on one H100, with accuracy mostly held vs Super; Iterative Puzzle is incremental but the measurements are real. the 3 major comments →

arxiv 2607.04371 v2 pith:NVICEFTE submitted 2026-07-05 cs.AI

Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

Akhiad Bercovich , Talor Abramovich , Daniel Afrimi , Shay Aharon , Nir Ailon , Vladimir Anisimov , Omer Ullman Argov , Maor Ashkenazi
show 62 more authors
Tomer Asida Nave Assaf Tomer Bar Natan Alexander Bukharin Grzegorz Chlebus Marcin Chochowski Eric Chung Mohammad Dabbah Carlo del Mundo Ewa Dobrowolska Ido Galil Yaniv Galron Amnon Geifman Yonatan Geifman Izik Golan Alex Gronskiy Tomasz Grzegorzek Netanel Haber Lior Kadoch Grzegorz Karch Tomer Keren Abhinav Khattar Amir Klein Tugrul Konuk Roi Koren Daniel Korzekwa Shaun Kotek Konstantinos Krommydas Itay Levy Ofri Masad Yoav Miron Pavlo Molchanov Shahar Mor Zach Moshe Saurav Muralidharan Najeeb Nabwani Besmira Nushi Mostofa Patwary Omri Puny Johannes Rausch Tomer Ronen Sepehr Sameni Itamar Schen Elad Segal Daniel Serebrenik Ido Shahaf Soumye Singhal Daniil Sorokin Sharath Turuvekere Sreenivas Marta Stepniewska-Dziubinska Ali Taghibakhshi Nima Tajbakhsh Oren Tropp Dor Tzur Anna Warno Yi-Fu Wu Michal Zawalski Jiaqi Zeng Yian Zhang Ran Zilberstein Amit Zuker Ran El-Yaniv
This is my paper
classification cs.AI
keywords hybrid MoE compressionIterative Puzzleknowledge distillationMamba pruninginteractive servingmulti-token predictionNVFP4 quantizationlong-context concurrency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that large hybrid mixture-of-experts language models can be made much cheaper to serve without giving up most of their reasoning, coding, long-context, and agentic skill. Starting from a 120B-total / 13B-active parent, the authors build a 75B-total / 9B-active student by repeatedly pruning MoE experts and Mamba state, healing each intermediate model with short knowledge distillation, then finishing with longer distillation, reinforcement learning, quantization, and multi-token prediction. On a single eight-GPU node the compressed model roughly doubles total server throughput at the same per-user token rate; on one GPU it raises the number of simultaneous million-token requests from one to eight. The claim is practical: production systems can keep near-parent quality while meeting interactive and ultra-long-context budgets that the parent cannot.

Core claim

Heterogeneous, iterative structural compression of a hybrid MoE-Mamba model—jointly choosing per-layer expert width, number of active experts, and Mamba state size, then recovering with distillation and reinforcement learning—yields a student that delivers approximately 2× Pareto-optimal server throughput versus the parent at matched user-throughput targets and multiplies 1M-context concurrency eight-fold while retaining strong accuracy across a broad evaluation suite.

What carries the argument

Iterative Puzzle: a sequential neural-architecture search that alternates moderate hardware-aware pruning of MoE and Mamba layers with short knowledge-distillation recovery so that replacement scores are recomputed on the current compressed model rather than only on the original parent.

Load-bearing premise

The method assumes that after each short healing step the quality of individual layer replacements remains roughly additive, so the optimizer can still pick a near-optimal capacity layout for the final target.

What would settle it

Run the same three-stage compression budget as a single-shot Puzzle search (no intermediate recovery) and check whether the final model still matches the reported suite-average accuracy and the ~2× throughput gains at fixed user throughput on the 8×B200 node; a large drop would falsify the value of the iterative loop.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript presents Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-oriented compression of Nemotron-3-Super from 120.7B total / 12.8B active parameters to 75.3B total / 9.3B active. Compression uses Iterative Puzzle (sequential hardware-aware NAS over heterogeneous MoE intermediate size and top-k, plus uniform Mamba SSM state pruning 128→96), interleaved with short KD, then longer-context KD, SWE-focused RL, PTQ (FP8/NVFP4), and a transferred/continued shared MTP head. On a single 8×B200 node at matched NVFP4, the model reports roughly 1.6–2.14× Pareto-optimal total throughput versus Super at fixed user-throughput thresholds (UT≥100/125/150) in 50K/2K and 8K/64K regimes (Table 7, Fig. 6); on a single H100 at 1M context it raises concurrency from 1 to 8. Suite accuracy is largely retained relative to Super across reasoning, coding, long-context, multilingual, and agentic benchmarks (Table 3), with supporting ablations on iterative vs single-shot Puzzle (Table 6), recovery progression (Fig. 4), MTP acceptance (Table 5), and disaggregated prefill (Fig. 5).

Significance. If the reported Super→Puzzle comparisons hold under independent reproduction, this is a strong systems contribution: it shows that hybrid Mamba–Attention–MoE models can be non-uniformly compressed for interactive serving and ultra-long-context memory limits while keeping most parent capability. Strengths include matched-quantization Pareto sweeps over TP/EP/batch size, explicit UT-constrained serving metrics (not only peak TPS), public Hugging Face checkpoints, and concrete ablations (iterative Puzzle +0.57 avg, training-stage recovery, MTP acceptance lengths, prefill-only disaggregation). The heterogeneous active MoE capacity map (Fig. 2, Table 1) and contribution-based SSM channel selection (Appendix A) are useful methodological artifacts for the community. Significance is primarily empirical and stack-specific rather than a new general theory of MoE compression.

major comments (3)
  1. Appendix C still contains an explicit unfinished placeholder: “TODO: drop in the headline MTP boost numbers (Super-Turbo-MTP-best vs. Super-Turbo-no-MTP at UT≥100, both scenarios) from the latest report.” The abstract and §3.1/Table 4 advertise large MTP-compounded gains, but the main interactive Pareto analysis in §3.3/Table 7 is single-step only, and the MTP overlay (Fig. 8) is incomplete. For the deployment claim to be fully load-bearing, the manuscript needs finalized, matched-quantization Pareto numbers for Puzzle+MTP vs Super+MTP at the same UT thresholds used in Table 7, with draft length and acceptance-length assumptions stated.
  2. External baselines beyond the Nemotron Super/Nano family are essentially absent from the accuracy–efficiency comparison (Fig. 1, Table 4). The central Super→Puzzle 2× claim is internally well supported, but the broader conclusion that “large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability” would be much stronger with at least one independent open model of similar active-parameter or throughput class (or a published compressed MoE baseline) under the same UT-constrained Pareto protocol. Without that, generality remains an open risk even if the parent comparison is correct.
  3. §2.1.2 and Table 6: Iterative Puzzle is presented as the core methodological advance, yet the only architecture-search ablation is a +0.57 unweighted average over a modest benchmark subset versus single-shot Puzzle at the same final budget. That gain supports preferring the iterative procedure, but does not establish near-optimality of the heterogeneous ρ_l allocation (Fig. 2) under residual higher-order layer interactions. A load-bearing clarification is needed: either (i) state clearly that the paper claims measured efficiency of the final architecture, not global optimality of the search, or (ii) add a stronger control (e.g., uniform capacity at matched active params, or a second iterative path with different stage budgets) so readers can separate search quality from recovery quality.
minor comments (6)
  1. Abstract “approximately 2×” should be qualified by regime: Table 7 shows ~1.60–1.79× on prefill-heavy 50K/2K and ~2.03–2.14× on decode-heavy 8K/64K at the listed UT points.
  2. Table 4 “TPS×Super” and “Rel. Req./min” columns are easy to misread against the Super MTP row (3.04× vs text mentioning 3.57× elsewhere). Align all Super+MTP multipliers with one consistent measurement protocol.
  3. Fig. 6 legend uses “Turbo Pareto” while the model is named Puzzle-75B-A9B throughout; unify naming to avoid implying a different product.
  4. §2.2 RL: “the impact of RL training in our experiments was small” (Fig. 4) is important; consider moving a one-sentence statement into the main recovery narrative so readers do not over-attribute SWE recovery to RL.
  5. Appendix A/B/D are valuable but dense; a short pointer in §2.1 to which pruning axes entered the final MIP (and which were explored then dropped, e.g. latent dimension) would improve navigability.
  6. Typos/consistency: “Nemotron-3-Puzzle” vs “Nemotron-Labs-3-Puzzle”; “Super Turbo” vs “Puzzle-75B-A9B”; ensure Table 3/9 benchmark names and tool-use settings match the parent Super paper’s protocol citations.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose ~2× throughput and accuracy-retention claims are measured, not derived by construction from self-defined inputs.

full rationale

Nemotron-Labs-3-Puzzle-75B-A9B is a post-training compression and serving paper. Its central claims (≈2× Pareto-optimal server throughput vs Super at matched UT on 8×B200 NVFP4; 1→8 concurrency at 1M on H100; suite accuracy near Super in Table 3) are hardware measurements and external-benchmark scores, not first-principles predictions forced by definition or by a fitted parameter renamed as a forecast. Iterative Puzzle, KD from Super, Super Stage-2 RL, Super MTP transfer, and Super-style PTQ are methodological self-citations typical of a distillation/compression lineage; they supply the parent teacher and recovery recipe but do not make the reported TPS/UT or AIME/GPQA/RULER/SWE numbers equal to their inputs by construction. Ablations (Table 6 iterative vs 1-step; Fig. 4 training progression; disaggregated prefill) and Pareto sweeps (Table 7, Fig. 6) are independent empirical checks. No uniqueness theorem is imported to forbid alternatives; no ansatz is smuggled as a derivation of the 2× result; no known empirical pattern is merely renamed. Residual dependence on the self-produced Super stack affects generality and teacher quality, not internal circularity of the measured Super→Puzzle comparison. Score 1 only for ordinary same-group parent/framework citation that is not load-bearing for the throughput claim.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central claims rest on engineering choices and empirical recovery assumptions rather than free physical constants. Load-bearing free parameters are the staged compression budgets, KD/RL schedules, and serving operating points chosen to hit the 2× / 8-concurrency targets. Domain axioms include Puzzle’s local-score + MIP formulation, transferability of Super logits/MTP/PTQ recipes, and fairness of matched-quantization Pareto comparisons. Invented entities are methodological constructs (Iterative Puzzle procedure, contribution-based SSM ranking, FastPuzzle prefill variants), not new physical objects; independent evidence is limited to the paper’s own ablations and released weights.

free parameters (6)
  • Stage-wise MoE capacity targets (75% → 60% weights; final 50% activated routed-expert budget)
    Hand-chosen incremental compression budgets that define the final architecture search path (§2.3); different budgets would yield different models and possibly different throughput/accuracy tradeoffs.
  • Mamba SSM state size reduction 128→96 (75%)
    Uniform prune ratio selected under inference-framework constraints; contribution ranking chooses which channels, but the target size is a design choice (§2.1.1, Table 1).
  • Per-stage KD token budgets (24B / 43.2B / 52.8B; recovery up to 100B tokens)
    Training compute allocated to heal after each Puzzle step; recovery quality and verbosity depend on these budgets (Fig. 4).
  • RL learning-rate sweep and weight averaging (1e-7 to 5e-6)
    Post-hoc multi-run averaging chosen after observing SWE sensitivity; final policy depends on this selection (§2.2).
  • Interactive operating point UT=100 tok/s (and 125/150) and scenarios 50K/2K, 8K/64K
    Deployment targets that define the Pareto comparison and the claimed ~2× boost; other latency/throughput targets could reorder models (Table 7).
  • PTQ calibration sets (256 samples; 4K or 65K tokens) and per-operator precision policy
    Calibration and keep-in-BF16/FP32 choices determine quantized accuracy and memory (§2.4, Table 2).
axioms (5)
  • domain assumption Local Puzzle block-replacement quality scores are approximately additive under a MIP constraint, and recomputing them after moderate prune+KD sufficiently captures higher-order interactions.
    Core of Iterative Puzzle tractability (§2.1, §2.1.2); only partially validated by the +0.57 iterative-vs-1-step ablation.
  • domain assumption Knowledge distillation from Super logits on a 30/70 pretrain/SFT mix recovers most capabilities lost to structural pruning, with SWE RL fixing residual software-engineering damage.
    Recovery pipeline premise (§2.2); Fig. 4 shows strong KD recovery but small RL effect.
  • domain assumption Matched NVFP4/FP8 quantization and identical hardware isolate architectural compression gains from numeric-format confounds.
    Stated methodology for serving comparisons (§3.3).
  • ad hoc to paper Contribution-based ranking of SSM channels (average |contrib| over validation tokens) identifies near-optimal channels to prune.
    Appendix A method; beats random under aggressive prune but gap shrinks after KD at the moderate 75% setting used in the final model.
  • standard math Standard mixed-integer programming and NAS search-space enumeration are valid optimizers for discrete layer choices.
    Background optimization machinery inherited from Puzzle/Puzzletron.
invented entities (3)
  • Iterative Puzzle procedure no independent evidence
    purpose: Sequential prune–score–KD loop to adapt architecture search to changing student representations.
    Primary methodological construct; evidence is internal ablation vs single-step Puzzle, not independent external replications.
  • Puzzle-75B-A9B heterogeneous MoE capacity map (ρ_l = k_l d_l / k_T d_T) independent evidence
    purpose: Layer-wise active routed-expert allocation under a global active budget.
    Concrete architecture product of the search (Fig. 2, Table 1); falsifiable via released weights and serving benchmarks.
  • FastPuzzleV1/V2 prefill-only pruned models no independent evidence
    purpose: Disaggregated prefill compression while keeping full decode model.
    Appendix B constructs; quality/throughput claims rest on joint Star Elastic training and the paper’s own common-benchmark average.

pith-pipeline@v1.1.0-grok45 · 28989 in / 4231 out tokens · 49508 ms · 2026-07-11T19:40:34.724821+00:00 · methodology

0 comments
read the original abstract

We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8xB200 node, Puzzle-75B-A9B achieves approximately 2x higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability. Our model is publicly available on Hugging Face.

Figures

Figures reproduced from arXiv: 2607.04371 by Abhinav Khattar, Akhiad Bercovich, Alexander Bukharin, Alex Gronskiy, Ali Taghibakhshi, Amir Klein, Amit Zuker, Amnon Geifman, Anna Warno, Besmira Nushi, Carlo del Mundo, Daniel Afrimi, Daniel Korzekwa, Daniel Serebrenik, Daniil Sorokin, Dor Tzur, Elad Segal, Eric Chung, Ewa Dobrowolska, Grzegorz Chlebus, Grzegorz Karch, Ido Galil, Ido Shahaf, Itamar Schen, Itay Levy, Izik Golan, Jiaqi Zeng, Johannes Rausch, Konstantinos Krommydas, Lior Kadoch, Maor Ashkenazi, Marcin Chochowski, Marta Stepniewska-Dziubinska, Michal Zawalski, Mohammad Dabbah, Mostofa Patwary, Najeeb Nabwani, Nave Assaf, Netanel Haber, Nima Tajbakhsh, Nir Ailon, Ofri Masad, Omer Ullman Argov, Omri Puny, Oren Tropp, Pavlo Molchanov, Ran El-Yaniv, Ran Zilberstein, Roi Koren, Saurav Muralidharan, Sepehr Sameni, Shahar Mor, Sharath Turuvekere Sreenivas, Shaun Kotek, Shay Aharon, Soumye Singhal, Talor Abramovich, Tomasz Grzegorzek, Tomer Asida, Tomer Bar Natan, Tomer Keren, Tomer Ronen, Tugrul Konuk, Vladimir Anisimov, Yaniv Galron, Yian Zhang, Yi-Fu Wu, Yoav Miron, Yonatan Geifman, Zach Moshe.

Figure 1
Figure 1. Figure 1: Accuracy–efficiency tradeoff on a single 8 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise active routed Experts capacity relative to the teacher. For each MoE layer [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MTP acceptance rate by draft token index on the SPEED-Bench qualitative split using a draft length of 7. 3.2 Ablations Single-shot vs. iterative compression. To isolate the effect of iterative compression, we compare the full three-stage Iterative Puzzle procedure against a single-step Puzzle baseline. The single-step baseline applies the compression targets in one Puzzle run, whereas Iterative Puzzle reac… view at source ↗
Figure 4
Figure 4. Figure 4: Training progression of Puzzle-75B-A9B. Accuracy recovers sharply after the final Puzzle iteration [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Disaggregated prefill ablation. FastPuzzleV1 and FastPuzzleV2 are faster derivatives of Puzzle-75B-A9B that are pruned further to accelerate the prefill stage. Using a faster model only for prefill barely changes accuracy as long as Puzzle-75B-A9B is still used for decode, whereas using the faster model for both prefill and decode substantially degrades accuracy. Numbers above bars show the exact plotted v… view at source ↗
Figure 6
Figure 6. Figure 6: Pareto frontiers underlying Table 7, on the 8 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The effect of the SSM channel-selection method. Under aggressive pruning, our contribution-based [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Super Turbo Pareto frontier with MTP overlaid on the single-step curves from Figure 6, same [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pareto frontiers of total vs. user throughput on a single 8 [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

  1. [1]

    URLhttps://arxiv.org/abs/2604.09557. Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ull...

    Pith/arXiv arXiv 2025

  2. [2]

    Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, et al

    URLhttps://arxiv.org/abs/2602.11937. Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, et al. Latentmoe: Toward optimal accuracy per flop and parameter in mixture of experts.arXiv preprint arXiv:2601.18089,

    arXiv

  3. [3]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias

    doi: 10.1515/crll.1883.94.41. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machin...

    work page doi:10.1515/crll.1883.94.41

  4. [4]

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov

    URL https://proceedings.mlr.press/v202/leviathan23a.html. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation.ArXiv, abs/2407.14679,

    Pith/arXiv arXiv

  5. [5]

    org/CorpusID:271328221

    URLhttps://api.semanticscholar. org/CorpusID:271328221. NVIDIA. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning.CoRR, abs/2512.20848,

    arXiv

  6. [6]

    URL https://doi.org/10

    doi: 10.48550/ARXIV.2512.20848. URL https://doi.org/10. 48550/arXiv.2512.20848. NVIDIA. Puzzletron implementation. https://github.com/NVIDIA/Model-Optimizer/tree/main/ modelopt/torch/puzzletron,

    doi:10.48550/arxiv.2512.20848

  7. [7]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro

    URLhttps://arxiv.org/abs/2604.12374. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism.CoRR, abs/1909.08053,

    Pith/arXiv arXiv 1909

  8. [8]

    Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan, et al

    URLhttp://arxiv.org/abs/1909.08053. Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan, et al. Star elastic: Many-in-one reasoning llms with efficient budget control.arXiv preprint arXiv:2605.07182,

    Pith/arXiv arXiv 1909

  9. [9]

    Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun

    URLhttps://arxiv.org/abs/2203.05482. Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-aware singular value decomposition for compressing large language models.ArXiv, abs/2312.05821,

    Pith/arXiv arXiv

  10. [10]

    17 Appendix A Mamba SSM Pruning A.1 Method We chose which SSM channels to prune by estimating their contribution to the Mamba layer output in the following manner

    URL https://api.semanticscholar.org/CorpusID:266162471. 17 Appendix A Mamba SSM Pruning A.1 Method We chose which SSM channels to prune by estimating their contribution to the Mamba layer output in the following manner. Given an input vectorxt, the SSM layer computes the following transformation: h(i) t =A th(i) t−1 +B (i) t xt y(i) t =C (i) t ⊤ h(i) t yt...

    2026

  11. [11]

    Per-request prefill of the 990K-token prompt is roughly1.2×faster on Super Turbo than on Super

    At this concurrency, Super Turbo delivers approximately4× the aggregate decode throughput of Super (Super Turbo at bs=8: ∼400 tok/s, derived from 20.1ms median inter-token latency across 8 concurrent requests; Super at bs=1: ∼94 tok/s, derived from 10.7ms TPOT). Per-request prefill of the 990K-token prompt is roughly1.2×faster on Super Turbo than on Super...

    2026

  12. [12]

    The projection-based method outperforms the coordinate-selection baselines at each tested latent size

    Projection-based method1.050 0.780 75.00Minitron 1.234 0.738 Table 8: Zero-shot comparison of latent-dimension pruning methods on an earlier Super checkpoint, with no KD or recovery. The projection-based method outperforms the coordinate-selection baselines at each tested latent size. Random and reverse-Minitron are diagnostic coordinate-selection control...

    2024