{"id":"ce9c281e-7810-4333-9f9e-05d71d8819eb","arxiv_id":"2504.17449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HMI serves up to 10,000 tenant-specific BERT and GPT instances on one GPU by replacing lower-layer computation with trigram lookup tables and sharing upper layers with per-task adapters.","lead":"This paper describes HMI, a system that breaks each fine-tuned language model into shared backbone layers, a lookup table for domain knowledge, and small task-specific adapters, so one GPU can serve many customized tenant models. A generalist should read it because it claims to cut the hardware cost of hosting thousands of customized models on a single server.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '10,000 hPLMs on one GPU' claim is not backed by evidence that 10,000 distinct tenant-specific hPLMs were ever trained; accuracy is reported for ten full-data task adapters, so the headline conjoins an unverified distinctness claim with an accuracy claim.","rationale":"The reader's weakest_assumption (O2) is real but is not the single most load-bearing gap given the paper's own data: the final hBERT system in Table 3 is 1.05 accuracy points below BERT-dedicated on average, while per-task gaps are larger on ACL-ARC, so the accuracy compromise is measurable but bounded on the ten tasks actually tested. The central claim, however, is about 10,000 distinct hPLMs. Nothing in Section 7 shows that 10,000 tenant-specific models were trained: the dataset section (Eqs. 4-5) constructs tenant subsets, Table 3 reports ten full-data task adapters, and Fig. 14 'simulates' tenants by random sampling. If the throughput and capacity experiments used untrained or synthetic adapter weights, they demonstrate memory accounting and I/O pipelining but not serving of 10,000 hPLMs; if they did train 10K adapters, the paper omits the training setup, compute, and any per-tenant accuracy. This is a correctable but decisive evidentiary gap. It is independent of O1/O2: even granting layer-wise knowledge separation, the 10K distinctness claim needs direct evidence. I therefore keep the reader's conditional verdict but make the condition explicit: the 10K-hPLM claim should either be backed by a released artifact with 10K trained adapters and tenant-level evaluation, or the paper should be reframed as serving thousands of tenants with a much smaller set of shared task adapters.","tokens_in":32262,"tokens_out":8300,"duration_ms":84521,"concrete_test":"Obtain the released artifact (or request train/eval scripts and adapter manifest) and count the number of unique adapter checkpoint sets and the training subset per tenant for all 10K 'hBERTs'. Then run Fig. 14 correctly: fine-tune one hPLM per tenant on that tenant's D_s^t and evaluate on held-out tenant data; report mean, percentiles, and per-task breakdown. Also rerun the Figs. 11-13 throughput experiments using those 10K genuinely trained distinct adapters with requests uniformly sampled across tenants. If throughput stays flat and the tenant-level accuracy distribution is within the claimed ~1% of BERT-dedicated, the central claim stands; if the artifact contains only about ten task-level adapters or per-tenant accuracy is not reported, the 10,000-hPLM claim is unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest_claim couples two things: (i) one GPU can serve 10,000 distinct hPLMs at full throughput, and (ii) this costs only about 1% accuracy. Component (ii) is established in Table 3 for ten full-data, task-level hBERTs; the paper never shows that 10,000 per-tenant hPLMs were fine-tuned. Section 7.1 allocates 10K tenant subsets via Eqs. (4)-(5) but describes no training of per-tenant adapters, and Fig. 14 'simulates' 10/100/1K/10K tenants by random subsetting, which can be served by the same task adapter and does not require distinct trained hPLMs. Table 2's '>10,000' is a memory accounting result, not a trained-model result. Thus the distinctness of the 10,000 hPLMs is unverified: if only ten task adapters exist, tenants within a task share weights and the 'distinct hPLMs' claim is false; if 10,000 adapters were trained, their training subsets can be tiny (e.g., |D_t|/n_t near 4 examples for common tasks), so the ~1% accuracy claim does not transfer without per-tenant evaluation. The throughput experiments in Figs. 11-13 can be run with random or untrained adapter weights and are therefore independent evidence of neither distinctness nor accuracy. Separately, task-level accuracy gaps are not uniformly negligible (hBERT-adapter averages 3.0 points below BERT-dedicated; ACL-ARC is 0.6547 vs. 0.7266), but the decisive gap is the missing demonstration of 10,000 distinct trained hPLMs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HMI, a multi-tenant inference system for serving many tenant-specific pretrained language models (PLMs) on a single GPU. The key idea is to decompose each customized model into three knowledge levels: a shared root PLM for general knowledge, domain-specific PLOT lookup tables that replace the lower transformer layers, and compact task-specific adapters injected into the shared upper transformer layers. The system manages domain knowledge through frequency-based PLOT version trees and task knowledge through layer-wise adapter swapping, and it adds pipelined prefetching and batched adapter computation as system optimizations. The evaluation claims that HMI can serve up to 10,000 hPLMs on one 16GB GPU with roughly a 1% average accuracy drop relative to fully fine-tuned BERT, and reports throughput, accuracy, generative-model, and memory experiments.","tokens_in":32635,"tokens_out":10078,"duration_ms":95430,"significance":"If fully established, the capacity claim would be a meaningful advance for cost-effective multi-tenant PLM serving, since it combines model management with serving-system optimizations rather than treating models as black boxes. The paper compares against external baselines (BERT-dedicated, BERT-shared, BERT-compress) and shows substantial throughput advantages, and the hBERT-PLOT variant is competitive with BERT-dedicated on Table 3, which is a useful data point for lower-layer skipping. The pipeline ablation in Table 6 and Fig. 17 cleanly isolates the contribution of the prefetching optimizations. The main weakness is that the headline 10,000-tenant claim is supported by memory accounting and simulated tenant subsets, not by training and evaluating 10,000 distinct per-tenant hPLMs, so the central claim is not yet fully established.","major_comments":[{"comment":"The paper's central claim—that HMI can serve 10,000 distinct tenant-specific hPLMs on one GPU with only a negligible accuracy compromise—is not supported by any experiment in which 10,000 per-tenant hPLMs are trained and evaluated. Section 7.1 states that a hBERT instance is spawned 'for each task' for the accuracy evaluation, while the 10,000-tenant construction in Eqs. (4)-(5) and Fig. 14 is described as 'simulat[ing]' tenants by random subsetting; no training of per-tenant adapters is reported. Table 2's '>10,000' figure is a memory-accounting calculation, and the throughput measurements in Figs. 11-13 do not require trained adapters. The authors should either train and evaluate per-tenant adapters at a representative scale or explicitly restrict the accuracy claim to the ten task-level hBERTs and present the 10,000-tenant result as a capacity and throughput result.","section":"Section 7.1, Eqs. (4)-(5), Fig. 14, Table 2"},{"comment":"The 'negligible compromise in accuracy' statement is not uniform across tasks. hBERT averages 0.7929 versus 0.8034 for BERT-dedicated (about 1.05 points below), but ACL-ARC drops from 0.7266 to 0.6547 for hBERT, and hBERT-adapter is about 3.0 points below BERT-dedicated on average. Moreover, the per-tenant subsets defined by Eqs. (4)-(5) contain only about 20-30 samples per tenant, since n_t is proportional to |D_t|, so the full-data task adapter results cannot be assumed to transfer to per-tenant adapters. A per-tenant accuracy evaluation, or an explicit statement that the accuracy claim applies only at task granularity, is needed before the headline claim can be accepted.","section":"Table 3 and Section 7.5"},{"comment":"Observation O2, on which the entire design rests, is only approximately supported. Fig. 7 shows the layer-wise effect for a single FPT/FT pair, and Fig. 16 extends it to LLaMA-3.1-8B, but the ten-task evaluation in Table 3 is the direct test: freezing the lower half and training adapters only in the upper half costs about 3.0 points on average for hBERT-adapter, with larger gaps on ACL-ARC and Citation Intent. The paper should provide per-task evidence, or a systematic split-point ablation over the Table 3 tasks, that the lower-layer freeze is benign rather than relying on the single illustrative curve in Fig. 7.","section":"Section 3.1 and Table 3"},{"comment":"The generative accuracy comparison is confounded: hGPT-PLOT6+6 uses OPT-125M as its backbone, while the GPT-compress baselines use DistilGPT2 because of 'the absence of compressed OPT models.' The ROUGE differences in Table 4 are small, but without a GPT-dedicated (full OPT-125M) baseline the reader cannot determine how much accuracy hGPT sacrifices relative to the model it is actually serving. If the generative extension is to support the 'similar results' claim, an OPT-dedicated baseline or a same-family compressed OPT baseline should be included.","section":"Section 7.6, Table 4"}],"minor_comments":[{"comment":"The text says 'our approach achieves the best average accuracy except BERT-dedicated,' but hBERT-PLOT has the highest average in the table (0.8043 versus 0.8034 for BERT-dedicated); please clarify which variant 'our approach' refers to and avoid overinterpreting small differences without error bars.","section":"Section 7.5, Table 3"},{"comment":"The sentence 'The effectiveness of PLOT R2. C1,C3n HMI is theoretically supported...' appears corrupted and should be rewritten.","section":"Section 7.10"},{"comment":"There are numerous typos ('nstead,' 'otivated,' 'or most text fragments,' 'his section,' 'we acknowledgehe,' 'e conducteddditional'); a careful proofread is needed.","section":"Throughout Sections 3-7"},{"comment":"References [58] and [59] are duplicate entries for Vaswani et al., 'Attention is all you need'; they should be merged.","section":"Section 8, References"},{"comment":"Section 3.2 says adapters use about 3% of PLM parameters, while Section 3.2.2 says about 2%; please make the estimate consistent.","section":"Section 3.2"},{"comment":"The caption says 'Downstream metrics' with the x-axis 'Num of tenants,' but the text says the points are obtained by 'randomly sampling a subset of each dataset to simulate' tenants; please clarify whether each point reflects trained per-tenant adapters or a shared task adapter evaluated on sampled data.","section":"Fig. 14"},{"comment":"The alpha=100% entries report hundreds of GB (463GB, 339GB), while alpha=50% reports 1.42GB; please explain what is counted as storage (distinct trigrams versus occurrences) so the reader can interpret the 'top alpha% occurrences' criterion.","section":"Section 7.7, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's PLOT root is built from the authors' own SkipBERT checkpoint [60] and the prototype was presented in their SIGMOD demo [61]; this is legitimate prior work, but the novelty relative to those two papers should be crisply stated. Also, S-LoRA and PetS are cited as closely related multi-tenant serving systems but are not included in the experiments; the revision should either add a comparison or explain why it is omitted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a real systems paper, but the marquee claim is not actually demonstrated. The abstract's \"serve up to 10,000 hPLMs on one GPU with only a negligible compromise in accuracy\" conjoins two claims. The accuracy part is established only for ten full-data task adapters in Table 3. The 10,000 part is memory accounting plus throughput measurements that do not require 10,000 trained adapters. The two are never shown together. That gap is the load-bearing issue.\n\nWhat is genuinely new: combining PLOT-style lower-layer skipping with task adapters, managing domain PLOTs through a frequency-updated version tree, and hiding CPU and I/O work behind GPU computation with layer-wise prefetching. The O1/O2 observations (domain knowledge lives in lower layers, task knowledge in higher layers) are tested directly in Figure 7 and extended to LLaMA in Figure 16, which is real evidence, not hand-waving. The frequency-based PLOT pruning with a modest threshold is a nice practical contribution, and the pipeline optimizations are ablated carefully.\n\nWhere the soft spots are, in proportion:\n\n- The 10,000-tenant accuracy claim is not supported. Section 7.1 defines tenant subsets via Eqs. (4)-(5), and Figure 14 \"simulates\" tenants by random sampling, but the paper never states how many distinct adapters were trained. If only ten task adapters exist, tenants within a task share weights and \"distinct hPLMs\" is not meaningfully true. If 10,000 adapters were trained, each per-tenant subset would be about 23 examples, so the full-data Table 3 accuracy numbers cannot transfer without per-tenant evaluation.\n- \"Negligible\" overstates even the full-data results. hBERT-adapter averages 3.0 points below BERT-dedicated, and ACL-ARC drops 7.2 points (0.6475 vs. 0.7266). hBERT-PLOT closes some gaps, but task-level variance remains.\n- Direct competitors for adapter-based multi-tenant serving—S-LoRA, PetS, Punica—appear only in related work. They are not benchmarked. For a systems paper claiming to advance the state of the art, that omission matters.\n- There are no error bars or seed variance reported, which is especially relevant for the random subset simulations.\n\nThe reliance on the authors' own SkipBERT for PLOT is not a real problem; they build on it and evaluate against external baselines. The paper is clearly written and the engineering is plausible. It deserves a serious referee, but only with a request to either train and evaluate 10,000 distinct adapters or substantially soften the claim. Send to peer review, major revision.","headline":"A clever and well-engineered serving system whose headline 10,000-tenant accuracy claim is not backed by the experiments; worth refereeing, but needs revision and direct evidence that 10,000 distinct hPLMs were actually trained.","tokens_in":33201,"tokens_out":3626,"would_cite":true,"duration_ms":38770,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single GPU can serve 10,000 tenant-specific language models at full inference throughput by splitting each model's knowledge into a precomputed lookup table and swappable adapters.","keywords":["multi-tenant inference","pretrained language model serving","hierarchical knowledge management","PLOT lookup table","adapter fine-tuning","model swapping","pipeline prefetching","parameter-efficient fine-tuning"],"falsifier":"Apply the hBERT pipeline to a suite of tasks that require deep lower-layer representation changes, such as heavily domain-idiomatic named entity recognition or tasks with adversarial word-order sensitivity, and compare the accuracy gap to fully fine-tuned BERT. If the average gap grows well beyond the reported 1%, or if serving 10,000 tenants only holds under artificially sparse request arrivals rather than the burst workload described, the central claim fails.","tokens_in":32039,"feed_emoji":"🖥️","tokens_out":9394,"duration_ms":80013,"temperature":0.7,"pith_summary":"The paper claims that thousands of distinct tenant-customized pretrained language models can be served from one GPU at undiminished inference throughput, at the cost of roughly 1 percent average accuracy relative to fully fine-tuning every model. The move is to treat what a customized model knows as three separable kinds of knowledge: general knowledge from pretraining, domain knowledge from further pretraining, and task knowledge from fine-tuning. Domain knowledge is materialized once per domain in a lookup table of precomputed hidden states, task knowledge is packed into compact adapters, and the same pretrained upper transformer layers are shared by every tenant. HMI then manages these pieces through versioned domain tables, host-memory adapter swapping, and prefetching that hides CPU and I/O work behind GPU computation. If the claimed scaling holds, cloud providers could offer tens of thousands of bespoke models on commodity servers instead of dedicating a GPU to each.","feed_headline":"One GPU serves 10,000 custom language models","feed_subtitle":"Separating domain and task knowledge into tables and adapters keeps throughput with ~1% accuracy loss.","key_machinery":"The load-bearing construction is the hierarchical PLM (hPLM), which splits a fine-tuned model into two physical data structures. PLOT (precomputed lookup table) stores the output of the lower transformer layers for text n-grams; inference retrieves and aggregates these vectors instead of computing those layers, so each domain needs only a table and each tenant needs no lower-layer weights. Adapters are bottleneck modules (down-project, ReLU, up-project, residual) injected after attention in the upper transformer layers and contain about 2-3% of the model's parameters; they hold the task-specific updates and can be swapped in from host memory per request. Two management mechanisms complete the design: a PLOT version tree with frequency-based entry updates keeps domain storage low, and fine-grained pipelining prefetches representations and adapter weights layer-wise so CPU lookup and I/O overlap with GPU transformer computation, while batched matrix multiplication runs several tenants' adapters in parallel.","core_discovery":"The central claim is that after further pretraining and fine-tuning, a PLM's weights separate by depth: the lower layers absorb domain-specific knowledge and the upper layers absorb task-specific knowledge. Because of that separation, a tenant-specific hPLM can be rebuilt as a PLOT (a key-value table mapping frequent text n-grams to their precomputed lower-layer representations) followed by the shared upper transformer layers of the original model with per-task adapters inserted. This changes the serving problem from swapping whole models to swapping only per-task adapters while looking up domain representations, so memory per tenant drops to a few megabytes. The paper reports that HMI serves up to 10,000 such hBERT and hGPT instances on one 16GB GPU with throughput that stays roughly constant as tenants grow, an average accuracy drop of about 1% versus full fine-tuning on ten tasks, and better generation quality than compressed baselines.","pith_inferences":["If the layer-separation observation generalizes to larger decoder models, any parameter-efficient adapter confined to upper layers could replace full model weights in the same architecture, turning multi-tenant serving of billion-parameter models into a memory-management problem rather than a compute-bound one.","The larger accuracy drops on the computer-science citation tasks suggest that tenants whose tasks depend on deep structural understanding pay a hidden tax, so a provider might route those tenants to full fine-tuning or richer domain-specific updates instead of the default hierarchy.","The finding that storing only the top 50% of frequent n-grams can outperform storing all of them indicates a tunable storage-accuracy frontier per domain rather than a single optimal lookup table.","A natural stress test is long-context generation, where PLOT's fixed n-gram window may lose information over long spans; combining the hierarchy with attention-sink or sparse-attention lower layers is a concrete next experiment."],"forward_implications":["One 16GB GPU can host more than 10,000 distinct tenant models rather than the 35 or 60 possible with full-model or distilled-model swapping.","Inference throughput stays nearly flat as the tenant count grows from 1 to 10,000, because all tenants share the same transformer backbone and only light per-request adapter loads vary.","The roughly 1% average accuracy gap to full fine-tuning is concentrated on domain-heavy tasks, while most classification and named-entity tasks match or nearly match full fine-tuning.","Decoder-only models can use the same hierarchy: the hGPT variant with an OPT backbone keeps high token throughput and outperforms distilled GPT baselines on summarization and rewriting quality at the same inference latency.","The frequency-based PLOT version tree limits additional domain storage to about 18.9GB for ten domains while retaining near-full accuracy."],"supporting_citations":[{"why":"It supplies the precomputed lookup table approach used for PLOT: lower-layer hidden states for n-grams are computed in advance and retrieved at inference, skipping shallow-layer computation.","marker":"[60]"},{"why":"It provides the adapter module that HMI uses to capture task-specific knowledge with a small fraction of the model's parameters.","marker":"[19]"},{"why":"It defines the BERT architecture, pretraining objective, and fine-tuning protocol that all hBERT variants build on and compare against.","marker":"[14]"},{"why":"It supplies one of the distillation-based compressed baselines used for accuracy, latency, and throughput comparisons.","marker":"[50]"},{"why":"It supplies the layer-wise distilled checkpoints used to show that compression still leaves many full model copies in memory.","marker":"[25]"},{"why":"It provides the parallel adapter execution technique that HMI adapts to compute different tenants' adapters concurrently with batched matrix multiplication.","marker":"[49]"},{"why":"It motivates the further-pretraining stage that HMI treats as the source of domain-specific knowledge in lower layers.","marker":"[26]"},{"why":"It is the prototype demonstration this paper extends with hierarchical knowledge management and systematic evaluation.","marker":"[61]"}],"fun_headline_variants":["10k bespoke LLMs on one GPU via hierarchical knowledge","Hierarchical knowledge trees fit 10,000 models on one GPU","One GPU hosts 10k custom models via layer-wise knowledge","Serving 10k LLMs per GPU with hierarchical PLM management","Memory per tenant shrinks to MBs for 10k models on one GPU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on the layer-wise separation of knowledge: further pretraining only meaningfully changes the lower half of the transformer, and fine-tuning only meaningfully changes the upper half, so freezing the lower half and replacing it with a lookup table costs almost no accuracy.","fun_headline_variants_meta":{"raw":{"variants":["10k bespoke LLMs on one GPU via hierarchical knowledge","Hierarchical knowledge trees fit 10,000 models on one GPU","One GPU hosts 10k custom models via layer-wise knowledge","Serving 10k LLMs per GPU with hierarchical PLM management","Memory per tenant shrinks to MBs for 10k models on one GPU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000758,"raw_usage":{"total_tokens":3383,"prompt_tokens":976,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2314}},"tokens_in":592,"tokens_out":2407,"duration_ms":15516,"temperature":1.0,"reasoning_tokens":2314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:39:56.052490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the hBERT pipeline to a suite of tasks that require deep lower-layer representation changes, such as heavily domain-idiomatic named entity recognition or tasks with adversarial word-order sensitivity, and compare the accuracy gap to fully fine-tuned BERT. If the average gap grows well beyond the reported 1%, or if serving 10,000 tenants only holds under artificially sparse request arrivals rather than the burst workload described, the central claim fails.","supporting_citations":[{"cited_title":"In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp","cited_arxiv_id":null,"evidence_quote":"It supplies the precomputed lookup table approach used for PLOT: lower-layer hidden states for n-grams are computed in advance and retrieved at inference, skipping shallow-layer computation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the adapter module that HMI uses to capture task-specific knowledge with a small fraction of the model's parameters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the BERT architecture, pretraining objective, and fine-tuning protocol that all hBERT variants build on and compare against."},{"cited_title":"In: Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pp","cited_arxiv_id":null,"evidence_quote":"It provides the parallel adapter execution technique that HMI adapts to compute different tenants' adapters concurrently with batched matrix multiplication."},{"cited_title":"In: Proceedings of the 2023 International Conference on Management of Data (2023)","cited_arxiv_id":null,"evidence_quote":"It is the prototype demonstration this paper extends with hierarchical knowledge management and systematic evaluation."}],"review_version":1}