Pith. sign in

REVIEW 4 major objections 7 minor 73 references

HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single GPU can serve 10,000 tenant-specific language models at full inference throughput by splitting each model's knowledge into a precomputed lookup table and swappable adapters.

desk verdict A clever and well-engineered serving system whose headline 10,000-tenant accuracy claim is not backed by the experiments; worth refereeing, but needs revision and direct evidence that 10,000 distinct hPLMs were actually trained. read the letter →

arxiv 2504.17449 v1 pith:RURAYOQF submitted 2025-04-24 cs.LG cs.AIcs.CL

Jun Zhang , Jue Wang , Huan Li , Lidan Shou , Ke Chen , Gang Chen , Qin Xie , Guiming Xie
show 1 more author
Xuejian Gong
This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords multi-tenantinferencepretrainedlanguagemodelservinghierarchicalknowledgemanagementPLOTlookuptableadapterfine-tuningswappingpipelineprefetchingparameter-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that thousands of distinct tenant-customized pretrained language models can be served from one GPU at undiminished inference throughput, at the cost of roughly 1 percent average accuracy relative to fully fine-tuning every model. The move is to treat what a customized model knows as three separable kinds of knowledge: general knowledge from pretraining, domain knowledge from further pretraining, and task knowledge from fine-tuning. Domain knowledge is materialized once per domain in a lookup table of precomputed hidden states, task knowledge is packed into compact adapters, and the same pretrained upper transformer layers are shared by every tenant. HMI then manages these pieces through versioned domain tables, host-memory adapter swapping, and prefetching that hides CPU and I/O work behind GPU computation. If the claimed scaling holds, cloud providers could offer tens of thousands of bespoke models on commodity servers instead of dedicating a GPU to each.

What carries the argument

The load-bearing construction is the hierarchical PLM (hPLM), which splits a fine-tuned model into two physical data structures. PLOT (precomputed lookup table) stores the output of the lower transformer layers for text n-grams; inference retrieves and aggregates these vectors instead of computing those layers, so each domain needs only a table and each tenant needs no lower-layer weights. Adapters are bottleneck modules (down-project, ReLU, up-project, residual) injected after attention in the upper transformer layers and contain about 2-3% of the model's parameters; they hold the task-specific updates and can be swapped in from host memory per request. Two management mechanisms complete the design: a PLOT version tree with frequency-based entry updates keeps domain storage low, and fine-grained pipelining prefetches representations and adapter weights layer-wise so CPU lookup and I/O overlap with GPU transformer computation, while batched matrix multiplication runs several tenants' adapters in parallel.

What would settle it

Apply the hBERT pipeline to a suite of tasks that require deep lower-layer representation changes, such as heavily domain-idiomatic named entity recognition or tasks with adversarial word-order sensitivity, and compare the accuracy gap to fully fine-tuned BERT. If the average gap grows well beyond the reported 1%, or if serving 10,000 tenants only holds under artificially sparse request arrivals rather than the burst workload described, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that after further pretraining and fine-tuning, a PLM's weights separate by depth: the lower layers absorb domain-specific knowledge and the upper layers absorb task-specific knowledge. Because of that separation, a tenant-specific hPLM can be rebuilt as a PLOT (a key-value table mapping frequent text n-grams to their precomputed lower-layer representations) followed by the shared upper transformer layers of the original model with per-task adapters inserted. This changes the serving problem from swapping whole models to swapping only per-task adapters while looking up domain representations, so memory per tenant drops to a few megabytes. The paper reports that HMI serves up to 10,000 such hBERT and hGPT instances on one 16GB GPU with throughput that stays roughly constant as tenants grow, an average accuracy drop of about 1% versus full fine-tuning on ten tasks, and better generation quality than compressed baselines.

Load-bearing premise

The argument stands on the layer-wise separation of knowledge: further pretraining only meaningfully changes the lower half of the transformer, and fine-tuning only meaningfully changes the upper half, so freezing the lower half and replacing it with a lookup table costs almost no accuracy.

Editorial extensions

If this is right

  • One 16GB GPU can host more than 10,000 distinct tenant models rather than the 35 or 60 possible with full-model or distilled-model swapping.
  • Inference throughput stays nearly flat as the tenant count grows from 1 to 10,000, because all tenants share the same transformer backbone and only light per-request adapter loads vary.
  • The roughly 1% average accuracy gap to full fine-tuning is concentrated on domain-heavy tasks, while most classification and named-entity tasks match or nearly match full fine-tuning.
  • Decoder-only models can use the same hierarchy: the hGPT variant with an OPT backbone keeps high token throughput and outperforms distilled GPT baselines on summarization and rewriting quality at the same inference latency.
  • The frequency-based PLOT version tree limits additional domain storage to about 18.9GB for ten domains while retaining near-full accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the layer-separation observation generalizes to larger decoder models, any parameter-efficient adapter confined to upper layers could replace full model weights in the same architecture, turning multi-tenant serving of billion-parameter models into a memory-management problem rather than a compute-bound one.
  • The larger accuracy drops on the computer-science citation tasks suggest that tenants whose tasks depend on deep structural understanding pay a hidden tax, so a provider might route those tenants to full fine-tuning or richer domain-specific updates instead of the default hierarchy.
  • The finding that storing only the top 50% of frequent n-grams can outperform storing all of them indicates a tunable storage-accuracy frontier per domain rather than a single optimal lookup table.
  • A natural stress test is long-context generation, where PLOT's fixed n-gram window may lose information over long spans; combining the hierarchy with attention-sink or sparse-attention lower layers is a concrete next experiment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes HMI, a multi-tenant inference system for serving many tenant-specific pretrained language models (PLMs) on a single GPU. The key idea is to decompose each customized model into three knowledge levels: a shared root PLM for general knowledge, domain-specific PLOT lookup tables that replace the lower transformer layers, and compact task-specific adapters injected into the shared upper transformer layers. The system manages domain knowledge through frequency-based PLOT version trees and task knowledge through layer-wise adapter swapping, and it adds pipelined prefetching and batched adapter computation as system optimizations. The evaluation claims that HMI can serve up to 10,000 hPLMs on one 16GB GPU with roughly a 1% average accuracy drop relative to fully fine-tuned BERT, and reports throughput, accuracy, generative-model, and memory experiments.

Significance. If fully established, the capacity claim would be a meaningful advance for cost-effective multi-tenant PLM serving, since it combines model management with serving-system optimizations rather than treating models as black boxes. The paper compares against external baselines (BERT-dedicated, BERT-shared, BERT-compress) and shows substantial throughput advantages, and the hBERT-PLOT variant is competitive with BERT-dedicated on Table 3, which is a useful data point for lower-layer skipping. The pipeline ablation in Table 6 and Fig. 17 cleanly isolates the contribution of the prefetching optimizations. The main weakness is that the headline 10,000-tenant claim is supported by memory accounting and simulated tenant subsets, not by training and evaluating 10,000 distinct per-tenant hPLMs, so the central claim is not yet fully established.

major comments (4)
  1. [Section 7.1, Eqs. (4)-(5), Fig. 14, Table 2] The paper's central claim—that HMI can serve 10,000 distinct tenant-specific hPLMs on one GPU with only a negligible accuracy compromise—is not supported by any experiment in which 10,000 per-tenant hPLMs are trained and evaluated. Section 7.1 states that a hBERT instance is spawned 'for each task' for the accuracy evaluation, while the 10,000-tenant construction in Eqs. (4)-(5) and Fig. 14 is described as 'simulat[ing]' tenants by random subsetting; no training of per-tenant adapters is reported. Table 2's '>10,000' figure is a memory-accounting calculation, and the throughput measurements in Figs. 11-13 do not require trained adapters. The authors should either train and evaluate per-tenant adapters at a representative scale or explicitly restrict the accuracy claim to the ten task-level hBERTs and present the 10,000-tenant result as a capacity and throughput result.
  2. [Table 3 and Section 7.5] The 'negligible compromise in accuracy' statement is not uniform across tasks. hBERT averages 0.7929 versus 0.8034 for BERT-dedicated (about 1.05 points below), but ACL-ARC drops from 0.7266 to 0.6547 for hBERT, and hBERT-adapter is about 3.0 points below BERT-dedicated on average. Moreover, the per-tenant subsets defined by Eqs. (4)-(5) contain only about 20-30 samples per tenant, since n_t is proportional to |D_t|, so the full-data task adapter results cannot be assumed to transfer to per-tenant adapters. A per-tenant accuracy evaluation, or an explicit statement that the accuracy claim applies only at task granularity, is needed before the headline claim can be accepted.
  3. [Section 3.1 and Table 3] Observation O2, on which the entire design rests, is only approximately supported. Fig. 7 shows the layer-wise effect for a single FPT/FT pair, and Fig. 16 extends it to LLaMA-3.1-8B, but the ten-task evaluation in Table 3 is the direct test: freezing the lower half and training adapters only in the upper half costs about 3.0 points on average for hBERT-adapter, with larger gaps on ACL-ARC and Citation Intent. The paper should provide per-task evidence, or a systematic split-point ablation over the Table 3 tasks, that the lower-layer freeze is benign rather than relying on the single illustrative curve in Fig. 7.
  4. [Section 7.6, Table 4] The generative accuracy comparison is confounded: hGPT-PLOT6+6 uses OPT-125M as its backbone, while the GPT-compress baselines use DistilGPT2 because of 'the absence of compressed OPT models.' The ROUGE differences in Table 4 are small, but without a GPT-dedicated (full OPT-125M) baseline the reader cannot determine how much accuracy hGPT sacrifices relative to the model it is actually serving. If the generative extension is to support the 'similar results' claim, an OPT-dedicated baseline or a same-family compressed OPT baseline should be included.
minor comments (7)
  1. [Section 7.5, Table 3] The text says 'our approach achieves the best average accuracy except BERT-dedicated,' but hBERT-PLOT has the highest average in the table (0.8043 versus 0.8034 for BERT-dedicated); please clarify which variant 'our approach' refers to and avoid overinterpreting small differences without error bars.
  2. [Section 7.10] The sentence 'The effectiveness of PLOT R2. C1,C3n HMI is theoretically supported...' appears corrupted and should be rewritten.
  3. [Throughout Sections 3-7] There are numerous typos ('nstead,' 'otivated,' 'or most text fragments,' 'his section,' 'we acknowledgehe,' 'e conducteddditional'); a careful proofread is needed.
  4. [Section 8, References] References [58] and [59] are duplicate entries for Vaswani et al., 'Attention is all you need'; they should be merged.
  5. [Section 3.2] Section 3.2 says adapters use about 3% of PLM parameters, while Section 3.2.2 says about 2%; please make the estimate consistent.
  6. [Fig. 14] The caption says 'Downstream metrics' with the x-axis 'Num of tenants,' but the text says the points are obtained by 'randomly sampling a subset of each dataset to simulate' tenants; please clarify whether each point reflects trained per-tenant adapters or a shared task adapter evaluated on sampled data.
  7. [Section 7.7, Table 5] The alpha=100% entries report hundreds of GB (463GB, 339GB), while alpha=50% reports 1.42GB; please explain what is counted as storage (distinct trigrams versus occurrences) so the reader can interpret the 'top alpha% occurrences' criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the accuracy and throughput claims are measured against external baselines; PLOT/adapter components reuse prior work but are independently evaluated.

full rationale

The paper does not contain a circular derivation. The layer-wise knowledge assumptions O1/O2 are established by direct experiments in Fig. 7 (training/freezing lower vs. higher layers) rather than by assuming the hPLM construction; hPLM accuracy is then checked against fully fine-tuned BERT and compressed baselines in Table 3 and Table 7. PLOT and the tri-gram default are taken from the same group's SkipBERT [60] and the prototype is cited as [61], but those self-citations are not load-bearing: the PLOT-based hBERT variants are independently evaluated against external baselines, and the n-gram size, number of lower layers, and alpha threshold are ablated in Tables 5 and 8-11 rather than fixed by the claimed result. The headline 10,000-hPLM capacity is a memory-accounting statement (Table 2) combined with throughput measurements (Figs. 11-13) that do not presuppose the ~1% accuracy claim; that accuracy claim comes from the full-data task adapters in Table 3. Fig. 14 'simulates' tenants by random subsetting and does not show that 10,000 distinct per-tenant adapters were trained or evaluated, and Section 7.5's ~1% average drop is therefore not transferred to 10,000 trained tenants; this is an evidential gap or omitted demonstration, not an equivalence by construction. No equation in the paper reduces a predicted quantity to a fitted input or defines the target result in terms of itself.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

HMI introduces system-level abstractions (hPLM, PLOT version tree) rather than physical entities; the load-bearing assumptions are the layer-wise knowledge split, the fidelity of trigram lookups, and the ad hoc claim that adapter tuning matches full fine-tuning.

free parameters (3)
  • frequency threshold α = not stated as default; evaluated at 0%, 30%, 50%, 100%
    Controls which trigrams are stored per domain in the PLOT version tree (Section 5.1, Eq. 3); chosen on a storage/accuracy tradeoff in Section 7.7.
  • n-gram window size = 3 by default
    Selected from Tables 8 and 9 as a compromise between context coverage and storage; a system hyperparameter, not independently predicted.
  • lower-layer split point = 6 of 12 layers for BERT-base
    Split between materialized lower layers and shared higher layers; taken from SkipBERT and tuned in Tables 10 and 11.
assumptions (4)
  • domain assumption Domain-specific knowledge acquired by further pretraining is concentrated in the lower transformer layers (O1).
    Stated as observation O1 in Section 3.1, supported by Fig. 7 on BERT and Fig. 16 on LLaMA, but assumed to generalize across domains and model sizes.
  • domain assumption Task-specific knowledge acquired by fine-tuning is concentrated in the higher transformer layers (O2).
    Stated as observation O2 in Section 3.1; used to justify freezing lower layers and using adapters only in the upper half. Table 3 shows hBERT-adapter is about 3% below full fine-tuning, so this assumption does not hold exactly.
  • domain assumption Trigram representations from PLOT faithfully approximate the output of the skipped lower layers (Eq. 2).
    Borrowed from SkipBERT [60]; HMI relies on it to replace the lower six layers with lookup, but the approximation is lossy, as acknowledged in Section 3.2.1.
  • ad hoc to paper Adapter tuning achieves downstream accuracy similar to full fine-tuning (Section 3.2.2).
    The paper states this in Section 3.2.2 but its own Fig. 5 and Table 3 (hBERT-adapter vs BERT-dedicated) show a 3% gap, so the assumption is contradicted by the paper's data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models." pith.science (2026). https://pith.science/paper/RURAYOQF

@misc{pith2026250417449,
  author       = {Pith},
  title        = {Pith review of: HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RURAYOQF}},
  note         = {Machine review of arXiv:2504.17449}
}
read the original abstract

The significant computational demands of pretrained language models (PLMs), which often require dedicated hardware, present a substantial challenge in serving them efficiently, especially in multi-tenant environments. To address this, we introduce HMI, a Hierarchical knowledge management-based Multi-tenant Inference system, designed to manage tenants with distinct PLMs resource-efficiently. Our approach is three-fold: Firstly, we categorize PLM knowledge into general, domain-specific, and task-specific. Leveraging insights on knowledge acquisition across different model layers, we construct hierarchical PLMs (hPLMs) by extracting and storing knowledge at different levels, significantly reducing GPU memory usage per tenant. Secondly, we establish hierarchical knowledge management for hPLMs generated by various tenants in HMI. We manage domain-specific knowledge with acceptable storage increases by constructing and updating domain-specific knowledge trees based on frequency. We manage task-specific knowledge within limited GPU memory through parameter swapping. Finally, we propose system optimizations to enhance resource utilization and inference throughput. These include fine-grained pipelining via hierarchical knowledge prefetching to overlap CPU and I/O operations with GPU computations, and optimizing parallel implementations with batched matrix multiplications. Our experimental results demonstrate that the proposed HMI can efficiently serve up to 10,000 hPLMs (hBERTs and hGPTs) on a single GPU, with only a negligible compromise in accuracy.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 44 canonical work pages

  1. [1]

    In: 2020 ACM/IEEE 47th An- nual International Symposium on Computer Architecture (ISCA), pp

    Baek, E., Kwon, D., Kim, J.: A multi-neural network acceleration architecture. In: 2020 ACM/IEEE 47th An- nual International Symposium on Computer Architecture (ISCA), pp. 940–953. IEEE (2020)

    work page 2020

  2. [2]

    Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., King, I.: BinaryBERT: Pushing the limit of BERT quantization. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pp. 4334–4348. Associat...

    doi:10.18653/v1/2021.acl-long 2021

  3. [3]

    : Qwen technical report

    Bai, J., Bai, S., Chu, Y., et al. : Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)

    arXiv 2023

  4. [4]

    Beltagy, I., Lo, K., Cohan, A.: SciBERT: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3615–3620. Association for Computa- tional Linguistics, Hong Kong, China (2019)...

    doi:10.18653/v1/d19-1371 2019

  5. [5]

    In: Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pp

    Bergsma, S., Zeyl, T., Senderovich, A., Beck, J.C.: Gener- ating complex, realistic cloud workloads using recurrent neural networks. In: Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles, pp. 376–391 (2021)

    work page 2021

  6. [6]

    ArXiv preprint abs/2108.07258 (2021)

    Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportuni- ties and risks of foundation models. ArXiv preprint abs/2108.07258 (2021). URL https://arxiv.org/abs/ 2108.07258

    arXiv 2021

  7. [7]

    Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A....

    work page 2020

  8. [8]

    Experiments on Paraphrase Identification Using Quora Question Pairs Dataset

    Chandra, A., Stefanus, R.: Experiments on paraphrase identification using quora question pairs dataset (2020). URL https://arxiv.org/abs/2006.02648

    work page Pith review arXiv 2020

Show all 73 references
  1. [9]

    In: 2019 IEEE International Parallel and Distributed Pro- cessing Symposium (IPDPS), pp

    Chard, R., Li, Z., Chard, K., Ward, L., Babuji, Y., Woodard, A., Tuecke, S., Blaiszik, B., Franklin, M.J., Foster, I.: Dlhub: Model and data serving for science. In: 2019 IEEE International Parallel and Distributed Pro- cessing Symposium (IPDPS), pp. 283–292. IEEE (2019)

    2019

  2. [10]

    arXiv preprint arXiv:2310.18547 (2023)

    Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., Krishna- murthy, A.: Punica: Multi-tenant lora serving. arXiv preprint arXiv:2310.18547 (2023)

    2023 arXiv

  3. [11]

    CoRR abs/2410.16179 (2024)

    Chen, Z., Sadhukhan, R., Ye, Z., Zhou, Y., Zhang, J., Nolte, N., Tian, Y., Douze, M., Bottou, L., Jia, Z., Chen, B.: Magicpig: LSH sampling for efficient LLM genera- tion. CoRR abs/2410.16179 (2024). DOI 10.48550/ ARXIV.2410.16179. URL https://doi.org/10.48550/ arXiv.2410.16179

    2024 doi arXiv

  4. [12]

    arXiv preprint arXiv:2110.14168 (2021)

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training veri- fiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

    2021 arXiv

  5. [13]

    In: 14th USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 17), pp

    Crankshaw, D., Wang, X., Zhou, G., Franklin, M.J., Gon- zalez, J.E., Stoica, I.: Clipper: A {Low-Latency} online prediction serving system. In: 14th USENIX Sympo- sium on Networked Systems Design and Implementation (NSDI 17), pp. 613–627 (2017)

    2017

  6. [14]

    Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In: Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- no...

    2019

  7. [15]

    URL https: //arxiv.org/abs/2101.00027

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., Leahy, C.: The pile: An 800gb dataset of diverse text for language modeling (2020). URL https: //arxiv.org/abs/2101.00027

    2020 arXiv

  8. [16]

    In: International Con- ference on Learning Representations (2016)

    Han, S., Mao, H., Dally, W.J.: Deep compression: Com- pressing deep neural networks with pruning, trained quantization and huffman coding. In: International Con- ference on Learning Representations (2016)

    2016

  9. [17]

    In: International Conference on Learning Representations (2021)

    He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., Neubig, G.: Towards a unified view of parameter-efficient trans- fer learning. In: International Conference on Learning Representations (2021)

    2021

  10. [18]

    ArXiv preprint abs/1503.02531 (2015)

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. ArXiv preprint abs/1503.02531 (2015). URL https://arxiv.org/abs/ 1503.02531

    2015 arXiv

  11. [19]

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: K. Chaudhuri, R. Salakhutdinov (eds.) Proceedings of the 36th International Conference on Machine Learn- ing, ...

    2019

  12. [20]

    In: Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp

    Howard, J., Ruder, S.: Universal language model fine- tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 328–339. Association for Computational Linguistics, Melbourne, Austr...

    2018 doi

  13. [21]

    In: International Conference on Learning Representations (2021)

    Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)

    2021

  14. [22]

    ArXiv preprint abs/1904.05342 (2019)

    Huang, K., Altosaar, J., Ranganath, R.: Clinicalbert: Modeling clinical notes and predicting hospital readmis- sion. ArXiv preprint abs/1904.05342 (2019). URL https://arxiv.org/abs/1904.05342

    2019 arXiv

  15. [23]

    Iandola, F., Shaw, A., Krishna, R., Keutzer, K.: Squeeze- BERT: What can computer vision teach NLP about efficient neural networks? In: Proceedings of Sus- taiNLP: Workshop on Simple and Efficient Natural Lan- guage Processing, pp. 124–135. Association for Compu- tational Ling...

    2020 doi

  16. [24]

    3651–3657

    Jawahar, G., Sagot, B., Seddah, D.: What does BERT learn about the structure of language? In: Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 3651–3657. Association for Computational Linguistics, Florence, Italy (2019). DOI 10.186...

    2019 doi

  17. [26]

    In: Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language 24 Jun Zhang et al

    Jin, X., Zhang, D., Zhu, H., Xiao, W., Li, S.W., Wei, X., Arnold, A., Ren, X.: Lifelong pretraining: Continu- ally adapting language models to emerging corpora. In: Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language 24 Jun Z...

    2022 doi

  18. [27]

    Kim, S., Gholami, A., Yao, Z., Mahoney, M.W., Keutzer, K.: I-BERT: integer-only BERT quantization. In: M. Meila, T. Zhang (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Res...

    2021

  19. [28]

    Kumar, A., Boehm, M., Yang, J.: Data management in machine learning: Challenges, techniques, and systems. In: S. Salihoglu, W. Zhou, R. Chirkova, J. Yang, D. Su- ciu (eds.) Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD Confer- ence 2017, Ch...

    2017

  20. [29]

    In: 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: A lite BERT for self-supervised learning of language representations. In: 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view....

    2020

  21. [30]

    Bioinfor- matics 36(4), 1234–1240 (2020)

    Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language rep- resentation model for biomedical text mining. Bioinfor- matics 36(4), 1234–1240 (2020)

    2020

  22. [31]

    In: 13th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 18), pp

    Lee, Y., Scolari, A., Chun, B.G., Santambrogio, M.D., Weimer, M., Interlandi, M.: {PRETZEL}: Opening the black box of machine learning prediction serving systems. In: 13th USENIX Symposium on Operating Systems De- sign and Implementation (OSDI 18), pp. 611–626 (2018)

    2018

  23. [32]

    In: Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp

    Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. In: Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp. 3045–3059. Association for Computational Linguistics, Online and Punta Cana, Dom...

    2021

  24. [33]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pp

    Li, J., Liu, X., Zhao, H., Xu, R., Yang, M., Jin, Y.: BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance. In: Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing (EMNLP), pp. 3009–3018. As- sociation for C...

    2020 doi

  25. [34]

    Proceedings of the VLDB Endowment 14(12) (2021)

    Li, L., Nakandala, S., Kumar, A.: Intermittent human-in- the-loop model selection using cerebro: a demonstration. Proceedings of the VLDB Endowment 14(12) (2021)

    2021

  26. [35]

    Proceedings of the VLDB Endowment 14(11), 2327–2340 (2021)

    Li, S., Kumar, A.: Towards an optimized group by ab- straction for large-scale machine learning. Proceedings of the VLDB Endowment 14(11), 2327–2340 (2021)

    2021

  27. [36]

    Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the 59th An- nual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),...

    2021 doi

  28. [37]

    In: IEEE INFOCOM 2020-IEEE Con- ference on Computer Communications, pp

    Li, Y., Han, Z., Zhang, Q., Li, Z., Tan, H.: Automating cloud deployment for deep learning inference of real-time online services. In: IEEE INFOCOM 2020-IEEE Con- ference on Computer Communications, pp. 1668–1677. IEEE (2020)

    2020

  29. [38]

    arXiv preprint (2019)

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint (2019)

    2019

  30. [39]

    In: Proceedings of the 27th ACM International Con- ference on Architectural Support for Programming Lan- guages and Operating Systems, pp

    Liu, Z., Leng, J., Zhang, Z., Chen, Q., Li, C., Guo, M.: Veltair: towards high-performance multi-tenant deep learning services via adaptive compilation and schedul- ing. In: Proceedings of the 27th ACM International Con- ference on Architectural Support for Programming Lan- gu...

    2022

  31. [40]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp

    Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.: S2ORC: The semantic scholar open research cor- pus. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–

  32. [41]

    Pro- ceedings of the VLDB Endowment 13(12), 2159–2173 (2020)

    Nakandala, S., Zhang, Y., Kumar, A.: Cerebro: A data system for optimized deep learning model selection. Pro- ceedings of the VLDB Endowment 13(12), 2159–2173 (2020)

    2020

  33. [42]

    ArXiv abs/1808.08745 (2018)

    Narayan, S., Cohen, S.B., Lapata, M.: Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv abs/1808.08745 (2018)

    2018 arXiv

  34. [43]

    ArXiv preprint abs/1712.06139 (2017)

    Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., Soyke, J.: Tensorflow-serving: Flexible, high-performance ml serv- ing. ArXiv preprint abs/1712.06139 (2017). URL https://arxiv.org/abs/1712.06139

    2017 arXiv

  35. [44]

    URL https://arxiv.org/abs/2310

    Paster, K., Santos, M.D., Azerbayev, Z., Ba, J.: Open- webmath: An open dataset of high-quality mathemati- cal web text (2023). URL https://arxiv.org/abs/2310. 06786

    2023

  36. [45]

    In: International Conference on Learning Representations (2021)

    Qin, H., Ding, Y., Zhang, M., Qinghua, Y., Liu, A., Dang, Q., Liu, Z., Liu, X.: Bibert: Accurate fully binarized bert. In: International Conference on Learning Representations (2021)

    2021

  37. [46]

    Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre- training. Tech. rep., OpenAI (2018)

    2018

  38. [47]

    IEEE transac- tions on Neural Networks 4(5), 740–747 (1993)

    Reed, R.: Pruning algorithms-a survey. IEEE transac- tions on Neural Networks 4(5), 740–747 (1993)

    1993

  39. [48]

    Transactions of the Association for Computational Lin- guistics 8, 842–866 (2020)

    Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Lin- guistics 8, 842–866 (2020). DOI 10.1162/tacl a 00349. URL https://aclanthology.org/2020.tacl-1.54

    2020 doi

  40. [49]

    In: Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp

    R¨ uckl´ e, A., Geigle, G., Glockner, M., Beck, T., Pfeif- fer, J., Reimers, N., Gurevych, I.: AdapterDrop: On the efficiency of adapters in transformers. In: Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp. 7930–7946. Association...

    2021

  41. [50]

    ArXiv preprint abs/1910.01108 (2019)

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint abs/1910.01108 (2019). URL https://arxiv.org/abs/1910.01108 HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in...

    2019 arXiv

  42. [51]

    IEEE Data Engineering Bulletin (2015)

    Schelter, S., Biessmann, F., Januschowski, T., Salinas, D., Seufert, S., Szarvas, G.: On chal- lenges in machine learning model management. IEEE Data Engineering Bulletin (2015). URL https://www.amazon.science/publications/ on-challenges-in-machine-learning-model-management

    2015

  43. [52]

    arXiv preprint arXiv:2311.03285 (2023)

    Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., et al.: S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285 (2023)

    2023 arXiv

  44. [53]

    In: Interna- tional Conference on Learning Representations (2021)

    Stephenson, C., suchismita padhy, Ganesh, A., Hui, Y., Tang, H., Chung, S.: On the geometry of generalization and memorization in deep neural networks. In: Interna- tional Conference on Learning Representations (2021). URL https://openreview.net/forum?id=V8jrrnwGbuc

    2021

  45. [54]

    Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowl- edge distillation for BERT model compression. In: Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJ...

    2019 doi

  46. [55]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp

    Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., Zhou, D.: MobileBERT: a compact task-agnostic BERT for resource-limited devices. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158–2170. Association for Computa- tional Linguistic...

    2020 doi

  47. [56]

    Touvron, H., Martin, L., Stone, K., et al.: Llama 2: Open foundation and fine-tuned chat models (2023)

    2023

  48. [57]

    In: CIDR 2017, 8th Biennial Conference on Innovative Data Systems Research, Cham- inade, CA, USA, January 8-11, 2017, Online Proceed- ings

    Vartak, M.: MODELDB: A system for machine learn- ing model management. In: CIDR 2017, 8th Biennial Conference on Innovative Data Systems Research, Cham- inade, CA, USA, January 8-11, 2017, Online Proceed- ings. www.cidrdb.org (2017). URL http://cidrdb.org/ cidr2017/gongshow/ab...

    2017

  49. [58]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Atten- tion is all you need. In: I. Guyon, U. von Luxburg, S. Ben- gio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Pro- ...

    2017

  50. [59]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Atten- tion is all you need. In: I. Guyon, U. von Luxburg, S. Ben- gio, H.M. Wallach, R. Fergus, S.V.N. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Proce...

    2017

  51. [60]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Wang, J., Chen, K., Chen, G., Shou, L., McAuley, J.: SkipBERT: Efficient inference with shallow layer skip- ping. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7287–7301. Association for Computa- tional...

    2022

  52. [61]

    In: Proceedings of the 2023 International Conference on Management of Data (2023)

    Wang, J., Chen, K., Shou, L., Jiang, D., Chen, G.: Smile: A cost-effective system for serving massive pretrained language models in the cloud (demo). In: Proceedings of the 2023 International Conference on Management of Data (2023)

    2023

  53. [62]

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In: H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, H. Lin (eds.) Advances in Neural Information Processing Systems...

    2020

  54. [63]

    In: The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. In: The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net (2024). URL https://openreview.net/ fo...

    2024

  55. [64]

    In: 2016 46th Annual IEEE/IFIP International Conference on Depend- able Systems and Networks (DSN), pp

    Xue, J., Birke, R., Chen, L.Y., Smirni, E.: Managing data center tickets: Prediction and active sizing. In: 2016 46th Annual IEEE/IFIP International Conference on Depend- able Systems and Networks (DSN), pp. 335–346. IEEE (2016)

    2016

  56. [65]

    Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhut- dinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. In: H.M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alch´ e- Buc, E.B. Fox, R. Garnett (eds.) Advances in Neural Information...

    2019

  57. [66]

    In: 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp

    Zafrir, O., Boudoukh, G., Izsak, P., Wasserblat, M.: Q8bert: Quantized 8bit bert. In: 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pp. 36–39. IEEE (2019)

    2019

  58. [67]

    In: 2019 USENIX An- nual Technical Conference (USENIX ATC 19), pp

    Zhang, C., Yu, M., Wang, W., Yan, F.:{MArk}: Exploit- ing cloud services for{Cost-Effective},{SLO-Aware} ma- chine learning inference serving. In: 2019 USENIX An- nual Technical Conference (USENIX ATC 19), pp. 1049– 1062 (2019)

    2019

  59. [68]

    In: Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa- pers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (2024)

    Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., Mehrotra, S.: Draft& verify: Lossless large language model acceleration via self-speculative decoding. In: Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa-...

    2024

  60. [69]

    ArXiv preprint abs/2205.01068 (2022)

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: Open pre-trained transformer language mod- els. ArXiv preprint abs/2205.01068 (2022). URL https://arxiv.org/abs/2205.01068

    2022 arXiv

  61. [70]

    In: 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp

    Zhou, Z., Wei, X., Zhang, J., Sun, G.: {PetS}: A unified framework for {Parameter-Efficient} transformers serv- ing. In: 2022 USENIX Annual Technical Conference (USENIX ATC 22), pp. 489–504 (2022)

    2022

  62. [334]

    URL https://aclanthology.org/2021.acl-long. 334

    2021

  63. [1722]

    DOI 10.1145/3035918.3054775

    ACM (2017). DOI 10.1145/3035918.3054775. URL https://doi.org/10.1145/3035918.3054775

    2017

  64. [4186]

    DOI 10.18653/v1/N19-1423

    Association for Computational Linguistics, Min- neapolis, Minnesota (2019). DOI 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423

    2019 doi

  65. [4983]

    DOI 10.18653/v1/2020.acl-main.447

    Association for Computational Linguistics, On- line (2020). DOI 10.18653/v1/2020.acl-main.447. URL https://aclanthology.org/2020.acl-main.447

    2020 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.