Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Selective neuron loading and computation let a 3B-parameter LLM run on an 8 GB edge device with about 1.5× the decode speed of 4-bit quantization, at a 13% memory reduction—no retraining required.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:10 UTC pith:FUUEC52Q

load-bearing objection New combination of offline neuron profiling with selective loading and selective computation, but the paper's own tables contradict the 'preserving task performance' claim, and the unmeasured prefill cost undermines the speedup. the 4 major comments →

arxiv 2607.18081 v1 pith:FUUEC52Q submitted 2026-07-20 cs.LG cs.AI

SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

Huzaifa Shaaban Kabakibo , Eric Schniedermeyer , Artem Burchanow , Lin Wang This is my paper
classification cs.LG cs.AI
keywords selective neuron loadingselective computationoffline LLM profilingactivation sparsityedge inferencememory reductiontask-specific neuronsmodel-specific neurons
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SelectInfer tries to establish that a large language model's feed-forward neurons have a dual structure—a stable, task-agnostic core and task-specific subsets—and that knowing this in advance, from an offline profiling pass, is enough to run a 3-billion-parameter model on a memory-poor edge device without retraining. It makes two moves: selective loading keeps only the profiled important neurons in memory, and selective computation computes only a further narrowed set at runtime, mixing always-on task neurons with prompt-chosen dynamic ones. On an 8 GB shared-memory device, the paper reports that Llama3.2-3B reaches about 9.85 tokens/s decoding, 1.53× over 4-bit quantization, with peak memory dropping from 6.88 GB to 5.97 GB, while translation accuracy far exceeds the prompt-only sparse baseline. The practical stakes are that memory footprint and compute can become two independent, continuously tunable knobs rather than a single quantization dial.

Core claim

The paper's central claim is that offline neuron profiling—scoring each FFN neuron by the sum of absolute post-activation values over many prompts—reveals two robust groups: task-specific 'base' neurons (high overlap within a task, low overlap across tasks) and general-purpose 'secondary' neurons (high overlap across all tasks). Using these rankings, loading the top 70% of neurons per modified layer and computing only the top 40% at runtime preserves competitive accuracy on QA, translation, and summarization, while enabling 3B models to fit on a device that cannot hold the full model. The comparison that carries the paper: replacing profiling-guided loading with random loading collapses accu

What carries the argument

The scoring identity is s_n = Σ_{t∈T} |a_n(t)|, the sum over tokens of the absolute activation of neuron n at the input to the FFN's down-projection; it converts per-prompt activations into ranked neuron indices that can be merged across tasks. On top of it, two selection rules do the work. The load-time rule picks the top δ task-specific neurons as base neurons, then fills to γ with top model-specific neurons. The runtime rule precomputes, for every prefill token, the activation values of all loaded neurons, counts how often each neuron appears in the top α per token, and uses the top β most frequent as 'dynamic' neurons, combined with the static base until the total compute ratio φ is reac

Load-bearing premise

The load-bearing premise is that ranking neurons by summed absolute activation during profiling—and again during each prompt's prefill—identifies the neurons that actually matter for output quality; the paper's speed claim is measured only during decoding, so the prefill scoring cost is not counted.

What would settle it

Measure wall-clock time from prompt submission to completion for a long-prompt workload (e.g., 2048-token summarization) on the same 8 GB board, comparing SelectInfer against a 4-bit-quantized run. If SelectInfer is not faster end-to-end, the 1.53× decode-only claim does not generalize to prefill-dominated workloads.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A 3B-parameter LLM can run on an 8 GB shared-memory edge device—where the full model does not fit—by loading only about 70% of its feed-forward neurons in the layers where the method is applied.
  • Decoding throughput for Llama3.2-3B is about 9.85 tokens/s, roughly 1.53× faster than 4-bit quantization and more than 13× faster than disk offloading on the same board.
  • Memory savings follow a closed-form formula, ΔM = N_ffn × (1 − r_load) × r_layer × s, and the measured reduction for Llama3.2-3B is 13.2% (6.88→5.97 GB).
  • Profiling-guided selection is essential: random neuron loading collapses accuracy, while offline-ranked loading preserves a large share of the full model's performance.
  • Memory and compute can be tuned separately by changing the loading ratio γ and the computation ratio φ, giving finer-grained trade-offs than 4-bit vs 8-bit quantization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported 9.85 tokens/s is decoding-only; the selection step scores every loaded neuron for every prefill token. For long-prompt tasks such as document summarization, including prefill cost in an end-to-end measurement could shrink or erase the gap over quantization—a direct test of the throughput claim.
  • Because loaded weights stay at full float16 precision, a natural extension is to combine selective loading with low-precision quantization of only the loaded neurons; the two savings should compound, at some accuracy cost the paper does not quantify.
  • The offline profiler needs task-labeled data; the paper's own future-work note about a lightweight task classifier suggests the framework could be extended to unseen tasks at runtime, but until then a new deployment domain requires its own profiling run.
  • If the same model-specific secondary neurons reappear across tasks, sharing one merged profile across many task files could reduce duplication on-device; the paper does not explore this storage-level consequence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SelectInfer, a framework that uses an offline profiler to identify task-specific and model-specific neurons in LLM FFN layers, and then applies two runtime optimizations: selective loading (keeping only a fraction of neurons in memory) and selective computation (computing only a subset of loaded neurons during decoding). The method is evaluated on Llama3.2-1B/3B and Qwen2.5-3B on the Jetson Orin Nano, with claims of reduced memory, improved decoding throughput, and preserved accuracy. My assessment is that the central accuracy claim is contradicted by the paper's own baseline tables, and the throughput claim is incomplete because prefill-phase costs are not reported.

Significance. The dual-knob design — decoupling memory loading from compute — is a sensible and potentially useful idea, and the offline profiler cost (5–5.5 GPU-hours for 3B models) is modest. However, the paper's own results show that SelectInfer is substantially less accurate than both 70%-dense and 4-bit quantized baselines on the same datasets, which directly refutes the abstract's claim of 'preserving task performance.' The evaluation also suffers from a circularity in how neuron files are constructed from the evaluation datasets. If these issues were corrected, the method might still be of interest as a memory-saving technique with controlled accuracy loss, but the current manuscript does not support its main claims.

major comments (4)
  1. [§6.5, Tables 3 and 4] The central claim of accuracy preservation is contradicted by the paper's own measurements. For Llama3.2-3B, SelectInfer achieves 25.68 BLEU on WMT16 de–en and 22.55 EM on SQuADv2, whereas the 70%-dense baseline in Table 4 achieves 34.91 and 26.41, and 4-bit quantization achieves 37.64 and 30.14, respectively. Similar gaps appear for Llama3.2-1B and Qwen2.5-3B on most tasks. Section 6.8 states that SelectInfer reaches 'comparable accuracy' to 4-bit quantization, but a 5–12 BLEU drop is not comparable. The 70%-dense configuration is the natural control for selective computation, and the large gap to this baseline undermines the conclusion that selective computation 'boosts performance'.
  2. [§6.2] The evaluation is circular. Task-specific neuron files are generated 'using all datasets within a task,' which includes the datasets later used for accuracy evaluation in Table 3 (SQuADv2, WMT16, XSUM, CNN/DailyMail). Model-specific neurons are generated using XSum, which is also an evaluation task. This means the reported accuracy is on data that contributed to the offline neuron selection, so the numbers do not demonstrate generalization to unseen inputs of the same task. The manuscript should at minimum evaluate on held-out datasets that were not used in profiling, or clearly disclose this overlap and its impact.
  3. [§5, §6.7, Fig. 11] The speedup claim is based solely on decoding-phase tokens/s, but the selective-computation mechanism requires computing activations for all loaded neurons for every prefill token (§5). This prefill overhead is never measured or reported. For tasks with long prompts, such as summarization (CNN/DailyMail, XSUM) and QA (SQuADv2), the prefill phase can dominate end-to-end latency, so the reported 1.5× decoding speedup over 4-bit quantization may not translate to a user-visible end-to-end speedup. The paper should report end-to-end latency or prefill + decode breakdown for representative prompt lengths.
  4. [§6.5, Table 3] The claim that 'SelectInfer consistently outperforms' CoreInfer + Selective Loading is false. On Qwen2.5-3B, CoreInfer + Selective Loading achieves 35.69 EM on SQuADv2 while SelectInfer achieves 24.32; on MLQA en–en, SelectInfer scores 0.53 vs 0.48 (a small difference); and on CNN/DailyMail, SelectInfer scores 0.15 vs 0.16. Thus the contribution of selective computation over selective loading is not consistent, and the largest observed advantage is on translation tasks where the absolute BLEU scores are low for both methods. This weakens the paper's claim that the dynamic selection component is broadly beneficial.
minor comments (5)
  1. [§6.7] The reference to the speed–accuracy trade-off figure is broken: 'Illustrated in Fig ??.' Figure 11 also labels the SelectInfer bar as 'PartInfer' in the legend, which appears to be a typo.
  2. [§6.2 vs Table 3] Dataset naming and splits are inconsistent. Table 3 reports WMT16 ro–en and MLQA en–en, but §6.2 lists WMT16 de-en/en-de and WMT14 fr-en/en-fr, and does not mention SQuADv2 explicitly. Clarify the exact datasets, splits, and evaluation metrics used for each reported cell.
  3. [Abstract and Fig. 11] The abstract states 'about 11 tokens/s' for Llama3.2-3B, but Fig. 11 reports 9.85 tokens/s. Please reconcile.
  4. [Figures 5 and 6] Figure 5 (selective computation) appears in §5, but Figure 6 (selective loading) is referenced in §4; as a result, Figure 6 appears before Figure 5 in the text. Reorder or renumber for clarity.
  5. [Throughout] Several typos and minor wording issues: 'eventhough' (§6.5), 'Xsum' vs 'XSUM' capitalization, 'prefilling and decoding stage' should be 'prefill and decode stages' (§6.7).

Circularity Check

2 steps flagged

Accuracy results are in-sample: neuron files and hyperparameters are built from the same datasets used for evaluation.

specific steps
  1. fitted input called prediction [§6.2 Datasets and Neuron-Files Selection; §6.5 Accuracy (Table 3)]
    "We generate task-specific neuron files offline using all datasets within a task, ensuring that the resulting neurons reflects consistent task-specific responses."

    The neuron rankings that determine which FFN neurons are loaded and computed for each task are produced by running the profiler on the same datasets later used as the accuracy benchmark. Section 6.2 lists SQuAD, MLQA, WMT16-de-en, CNN/DailyMail, and XSum as profiling datasets; Table 3 reports accuracy on SQuADv2, MLQA en-en, WMT16 de-en, XSUM, and CNN/DailyMail. The 'preserved task performance' numbers are therefore an in-sample fit of neuron selection to the evaluation distribution, not an independent prediction on unseen data. The accuracy result is, by construction, measuring how well the selection matches its own profiling set.

  2. fitted input called prediction [§6.4 Parameter Configuration; §6.5 Accuracy (Table 3)]
    "We tested values of δ from 25% to 35% in small increments (Fig 9) using the Llama3.2-3B model. The accuracy remains stable across this range, with δ=30% slightly outperforming the alternatives, making it a reasonable and robust choice."

    The reported operating point used for all Table 3 accuracy results (δ=30%, layers 4–26, γ=70%, φ=40%) was selected by empirical search on the same task families and benchmarks as the evaluation. The layer range was also chosen on Llama3.2-3B for 'best cross-task performance' using those tasks. Thus the accuracy-preservation claim is partly the product of hyperparameter tuning on the test tasks; the paper reports stability, which softens the effect, but the configuration is not a parameter-free prediction.

full rationale

SelectInfer's memory and decoding-speed measurements (Table 5, Fig 11) are hardware measurements and are not circular; the profiler cost and mechanism are also described independently. The circularity is confined to the accuracy claim. Task-specific neuron files are generated from all datasets within each task (§6.2), and the accuracy benchmark in Table 3 is drawn from those same datasets; hyperparameters are additionally tuned on the same task families (§6.4). Reporting accuracy on the profiling distribution therefore measures how well the neuron selection fits its own training data, not how well it preserves performance on unseen inputs. This is a fitted-input-called-prediction pattern rather than a self-citation chain: the references to CoreInfer are to prior external work and are not load-bearing self-citations. Non-circular correctness risks remain outside this score: §6.7 reports decoding tokens/s without prefill latency or end-to-end time, and Table 3 versus Table 4 shows large BLEU/EM drops relative to dense and 4-bit baselines, undermining the abstract's 'preserving task performance' claim. Those are measurement and correctness concerns, not derivation circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The method's contributions rest on several empirical premises about neuron importance, stability of task-specific neurons, the borrowed 40% compute rule, and FFN-only pruning. Each is asserted with partial or contradictory evidence; none is parameter-free. The selection thresholds γ, δ, ε, ϕ and the layer range are tunable numbers fit to the device and to the evaluation tasks.

free parameters (7)
  • gamma (overall loaded neuron ratio) = 0.7
    Chosen as the largest fraction that fits the Jetson Orin Nano memory; applied uniformly to Llama3.2-1B as well (§6.4).
  • delta (base/static neuron ratio) = 0.3
    Selected by scanning 25–35% and picking the value with slightly best accuracy on Llama3.2-3B (Fig 9).
  • epsilon (dynamic neuron ratio) = 0.1
    Residual such that δ+ε = ϕ = 0.4; no independent justification.
  • phi (computed neuron ratio) = 0.4
    Adopted from CoreInfer's prior analysis; paper confirms empirically without re-derivation (§6.4).
  • layer range (skip first 5, last 2) = layers 4–26 for Llama3.2-3B; 9 of 16 layers for Llama3.2-1B
    Chosen by evaluating first/last layer combinations (Fig 8) on a single model and device.
  • alpha (top neurons per token in dynamic selection) = undefined in text
    Algorithm in §5 requires α but no value or selection rule is given; needed for reproduction.
  • beta (top frequent neurons for dynamic set) = undefined in text
    Algorithm in §5 requires β but no value or selection rule is given; needed for reproduction.
axioms (4)
  • domain assumption Absolute activation magnitude at down_proj input is a valid importance score for FFN neurons.
    Eq (1) defines s_n = Σ|a_n(t)| and the method prunes low-score neurons, but no evidence shows this score predicts output-quality loss across layers and tasks.
  • domain assumption Neuron activation patterns are stable enough that offline scores transfer to runtime.
    §2 reports 58–72% overlap of top-40% sets, but Fig 2 and Fig 4 give contradictory cross-task overlap values, weakening this premise.
  • domain assumption Computing the top 40% of loaded neurons (ϕ=0.4) preserves quality.
    Adopted from CoreInfer (§6.4) rather than derived; the paper's own Table 3 vs Table 4 shows large BLEU drops from this choice.
  • domain assumption Only FFN layers need selective treatment; attention and embeddings remain fully loaded.
    The method is applied to FFN layers only (§4). No ablation demonstrates this split is optimal, and the layer-range choice comes from a single-device search.

pith-pipeline@v1.3.0-alltime-deepseek · 13257 in / 21097 out tokens · 587572 ms · 2026-08-01T16:10:01.465944+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across a range of Natural Language Processing (NLP) tasks, but their high computational and memory demands pose significant challenges for deployment on resource-constrained edge devices. Existing approaches to model compression and optimization often rely on coarse-grained pruning or quantization, which can compromise accuracy or require re-training and fine-tuning. In this work, we introduce SelectInfer, a neuron-level optimization framework that enables efficient LLM inference on edge devices through selective neuron loading and computation. By profiling and identifying both task-specific and general-purpose neurons using an offline LLM profiler, SelectInfer implements two key optimizations: selective loading, which reduces memory footprint by selectively loading a subset of neurons that were identified to be most important during the offline stage, and selective computation, which dynamically computes only the most relevant neurons at runtime. Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices

Figures

Figures reproduced from arXiv: 2607.18081 by Artem Burchanow, Eric Schniedermeyer, Huzaifa Shaaban Kabakibo, Lin Wang.

Figure 1
Figure 1. Figure 1: SelectInfer has two phases: offline (left) and online (right). In the offline phase, we run a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: High neurons over￾lap between different tasks, proving that a set of neurons always gets activated, regard￾less of the input task. PiQA WMT16 CNN/DM 0 20 40 60 80 100 Avg overlap % in top 40% neurons 58.5 65.1 69.6 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Process of selective computation. not loaded neurons secondary neurons neurons to load, γ task-specific neurons, δ model-specific neurons base neurons [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Dynamic neuron selection process [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Impact of different layer combinations on cross-task performance. The values are nor￾malized to the maximum value of each task [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Throughput comparison between baselines. inference, which evaluates all neurons. For the Llama3.2-1B model, SelectInfer and CoreInfer achieve comparable performance (∼19.2 tokens/s), as both similarly restrict computation to 40% of neurons. Meanwhile, the quantized model reaches only 10.71 tokens/s, approximately half the speed of the previous approaches. 6.8 Summary Our evaluation shows that SelectInfer … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 12 linked inside Pith

  1. [1]

    LLaMA: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023. URLhttps://arxiv.org/abs/2302.13971

    Pith/arXiv arXiv 2023

  2. [2]

    7 top large language model use cases and applications, 2024

    Daivi. 7 top large language model use cases and applications, 2024. URL https://www. projectpro.io/article/large-language-model-use-cases-and-applications/

    2024

  3. [3]

    Best applications of large language models, 2023

    Anastasiya Zharovskikh. Best applications of large language models, 2023. URL https: //indatalabs.com/blog/large-language-model-apps. Accessed: 2026-04-08

    2023

  4. [4]

    Real-world use cases for large language mod- els (llms), 2023

    CellStrat. Real-world use cases for large language mod- els (llms), 2023. URL https://cellstrat.medium.com/ real-world-use-cases-for-large-language-models-llms-d71c3a577bf2 . Ac- cessed: 2026-04-08

    2023

  5. [5]

    Clonecloud: elastic execution between mobile device and cloud

    Byung-Gon Chun, Sunghwan Ihm, Petros Maniatis, Mayur Naik, and Ashwin Patti. Clonecloud: elastic execution between mobile device and cloud. InProceedings of the sixth conference on Computer systems, pages 301–314, 2011

    2011

  6. [6]

    Powerinfer-2: Fast large language model inference on a smartphone, 2024

    Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. Powerinfer-2: Fast large language model inference on a smartphone, 2024. URL https://arxiv.org/abs/ 2406.06282

    Pith/arXiv arXiv 2024

  7. [7]

    PrivateGPT, May 2023

    Zylon by PrivateGPT. PrivateGPT, May 2023. URL https://github.com/zylon-ai/ private-gpt

    2023

  8. [8]

    Llm-rec: Personalized recommendation via prompting large language models, 2024

    Hanjia Lyu, Song Jiang, Hanqing Zeng, Yinglong Xia, Qifan Wang, Si Zhang, Ren Chen, Christopher Leung, Jiajie Tang, and Jiebo Luo. Llm-rec: Personalized recommendation via prompting large language models, 2024. URLhttps://arxiv.org/abs/2307.15780

    Pith/arXiv arXiv 2024

  9. [9]

    Jetson orin nano developer kit - 8gb

    NVIDIA Corporation. Jetson orin nano developer kit - 8gb. https://www. nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ nano-super-developer-kit/, 2025. Accessed: 2026-04-08

    2025

  10. [10]

    https://coral

    Google coral | Edge TPU / Coral Dev Board / Coral Accelerator Modules. https://coral. ai/, 2025. Accessed: 2026-04-08

    2025

  11. [11]

    Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation, 2024

    Qinsi Wang, Saeed Vahidian, Hancheng Ye, Jianyang Gu, Jianyi Zhang, and Yiran Chen. Coreinfer: Accelerating large language model inference with semantics-inspired adaptive sparse activation, 2024. URLhttps://arxiv.org/abs/2410.18311

    Pith/arXiv arXiv 2024

  12. [12]

    Llama 3.2 - 3b model

    Meta AI. Llama 3.2 - 3b model. https://huggingface.co/meta-llama/Llama-3.2-3B ,

  13. [13]

    Llama 3.2 - 1b model

    Meta AI. Llama 3.2 - 1b model. https://huggingface.co/meta-llama/Llama-3.2-1B ,

  14. [14]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https:// qwenlm.github.io/blog/qwen2.5/

    2024

  15. [15]

    Deja vu: contextual sparsity for efficient llms at inference time

    Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: contextual sparsity for efficient llms at inference time. InProceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  16. [16]

    Accessed: 2026-04-08

    2026

  17. [17]

    Relu strikes back: Exploiting activation sparsity in large language models, 2023

    Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models, 2023. URLhttps://arxiv.org/abs/2310.04564

    Pith/arXiv arXiv 2023

  18. [18]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. InThirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  19. [19]

    Llm in a flash: Efficient large language model inference with limited memory

    Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in a flash: Efficient large language model inference with limited memory. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12562–12584, 2024. 12

    2024

  20. [20]

    Teaching machines to read and comprehend

    Karl Moritz Hermann, Tomáš Koˇciský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. InProceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1693–1701, Cambridge, MA, USA, 2015. MIT Press

    2015

  21. [21]

    Findings of the 2016 conference on machine translation

    Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 201...

    2016

  22. [22]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1099. URL h...

    doi:10.18653/v1/p17-1099 2017

  23. [23]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URLhttps://arxiv.org/abs/1606.05250

    Pith/arXiv arXiv 2016

  24. [24]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv. org/abs/1705.03551

    Pith/arXiv arXiv 2017

  25. [25]

    Jetson chips

    NVIDIA Corporation. Jetson chips. https://www.nvidia.com/en-us/ autonomous-machines/embedded-systems/, 2025. Accessed: 2026-04-08

    2025

  26. [26]

    Findings of the 2014 workshop on statistical machine translation

    Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58,...

    2014

  27. [27]

    SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. InProceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China, November

  28. [28]

    Mlqa: Evaluating cross-lingual extractive question answering, 2020

    Patrick Lewis, Barlas O ˘guz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. Mlqa: Evaluating cross-lingual extractive question answering, 2020. URL https://arxiv.org/ abs/1910.07475

    Pith/arXiv arXiv 2020

  29. [29]

    Bitsandbytes quantization library, 2025

    Bitsandbytes. Bitsandbytes quantization library, 2025. URL https://github.com/ bitsandbytes-foundation/bitsandbytes

    2025

  30. [30]

    Working with quantized types

    NVIDIA Corporation. Working with quantized types. https://developer.nvidia. com/docs/drive/drive-os/7.0.3/public/drive-os-tensorrt-developer-guide/ work-quantized-types.html, 2025. Accessed: 2026-04-08. 13

    2025

  31. [31]

    Choosing the right quantization for self-hosted llms

    Jamie Dborin. Choosing the right quantization for self-hosted llms. https://www. doubleword.ai/resources/behind-the-stack-ep-7--, 2025. Accessed: 2026-04-08

    2025

  32. [32]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL https://arxiv.org/abs/1808.08745

    Pith/arXiv arXiv 2018

  33. [33]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of machine learning and systems, 6:87–100, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device llm compression and acceleration.Proceedings of machine learning and systems, 6:87–100, 2024

    2024

  34. [34]

    3-bit multilevel operation with accurate programming scheme in tio x/al2o3 memristor crossbar array for quantized neuromorphic system.Nanotechnology, 32(29):295201, 2021

    Tae-Hyeon Kim, Jaewoong Lee, Sungjoon Kim, Jinwoo Park, Byung-Gook Park, and Hyungjin Kim. 3-bit multilevel operation with accurate programming scheme in tio x/al2o3 memristor crossbar array for quantized neuromorphic system.Nanotechnology, 32(29):295201, 2021

    2021

  35. [35]

    A compre- hensive survey on model compression and acceleration.Artificial Intelligence Review, 53(7): 5113–5155, 2020

    Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A compre- hensive survey on model compression and acceleration.Artificial Intelligence Review, 53(7): 5113–5155, 2020

    2020

  36. [36]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023. URLhttps://arxiv.org/abs/2305.14314

    Pith/arXiv arXiv 2023

  37. [37]

    Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  38. [38]

    Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. InProceedings of the International Conference on High Performance Computing, Networking, Storag...

    2022

  39. [39]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads,

  40. [40]

    Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale, 2023

    Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale, 2023. URLhttps://arxiv.org/abs/2212.09095

    Pith/arXiv arXiv 2023

  41. [41]

    Powerinfer: Fast large language model serving with a consumer-grade gpu

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. Powerinfer: Fast large language model serving with a consumer-grade gpu. InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 590–606, 2024. 14

    2024

  42. [45]

    Shared memory-contention-aware concurrent dnn execution for diversely heterogeneous system-on-chips

    Ismet Dagli and Mehmet E Belviranli. Shared memory-contention-aware concurrent dnn execution for diversely heterogeneous system-on-chips. InProceedings of the 29th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, pages 243–256, 2024

    2024

  43. [2019]

    doi: 10.18653/v1/D19-5409

    Association for Computational Linguistics. doi: 10.18653/v1/D19-5409. URL https: //www.aclweb.org/anthology/D19-5409

    doi:10.18653/v1/d19-5409

  44. [2024]

    URLhttps://arxiv.org/abs/2401.10774

    Pith/arXiv arXiv