{"id":"c2b7f1c6-2659-43cc-b87d-e33520ea8573","arxiv_id":"2505.06481","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A single-GPU serving system for multiple fine-tuned MoE LLMs achieves near-single-model throughput by sharing similar experts and reconfiguring non-expert layers at runtime.","lead":"This paper presents a serving system that lets one GPU run several fine-tuned Mixture-of-Experts language models at once by sharing similar experts across models and swapping non-expert layers for each request. It reports throughput close to single-model serving and much lower turnaround time than NVIDIA's MIG, which matters for groups that need to host multiple custom models on limited hardware.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Expert-substitution quality is shown only for favorable cases; TTFT overhead is under-characterized.","rationale":"The reader's weakest assumption is that experts at the same (layer, expert) position across fine-tuned variants are interchangeable. I agree this is the load-bearing assumption for the quality claim, and I also agree the evidence for it is thin: only two model families, no per-position analysis, and no stress test for the case where the most similar expert is not the one selected. My independent review confirms the paper's own numbers support the throughput/turnaround claims (85% turnaround reduction, ridge point at lambda=0.060 matching single-model serving), but the TTFT claim is overstated in the abstract: the paper itself reports ~0.5 s average increase but ~1.2 s actual swap latency, and the evaluation uses short prompts (20 input tokens, 25 output tokens) with no variance or adversarial arrival pattern. The quality claim is plausible but conditional: the benchmark results (Table 3: MT-Bench 8.16 vs instruct 8.13; perplexity 3.85 vs base 3.81) are surprisingly good given that roughly half the experts are foreign, which suggests the substitution works, but the paper does not explain why it works or when it might fail. The Switch experiments are also asymmetric: Model A is the task model and the proposed approach keeps ROUGE-1 high, but the evaluation does not include the reverse direction (serving Model B, C, or D on their own tasks), so the resilience claim is only demonstrated for one of the four models. These gaps do not refute the central claim, but they make it conditional on further validation.","tokens_in":13959,"tokens_out":3294,"duration_ms":24568,"concrete_test":"Run a per-position ablation on Mixtral: for each (layer, expert) coordinate, load the target model's own expert and compare output quality (perplexity on WikiText and MT-Bench score) against the consolidated layout, and against a layout that deliberately swaps in the least-similar expert at each position. If many positions show large quality drops when the foreign expert is used, the consolidation's quality guarantee needs a per-position caveat. Additionally, run the QoS experiment with requests alternating between the two models with zero think time between requests, and report TTFT variance and tail latency; if the average TTFT increase exceeds the measured swap latency or the tail grows disproportionately, the 'negligible TTFT increase' claim needs revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the system achieves competitive output quality while maintaining throughput comparable to serving a single model. The load-bearing assumption is that foreign experts loaded via similarity-based consolidation can substitute for a target model's own experts with acceptable quality loss. The paper supports this with Figure 2 (same-position expert L2 distances 3.6–4.9 versus 150–250) and benchmark results in Table 3. However, the evidence is incomplete. For Mixtral, only two variants are used (base and instruct), and the expert map is built by round-robin assignment over positions ranked by similarity. This means for the base model's quality evaluation (perplexity on WikiText, C4, PTB), the non-expert layers are base while roughly half the expert positions are instruct experts; for the instruct model's quality evaluation (MT-Bench etc.), non-experts are instruct with roughly half base experts. The reported quality is competitive, but there is no analysis of which positions tolerate substitution or what happens if the most similar expert is not the one selected. In the Switch experiments, the evaluation is asymmetric: Model A (the task model) contributes all non-expert layers and a round-robin share of experts, and Table 4 shows the proposed approach keeps ROUGE-1 at 0.46–0.49 even when adding models C and D which have near-zero standalone SAMSum ROUGE. This suggests the expert substitution absorbs interference well, but it also means the test is not probing hard cases, such as serving Model B or C on their own tasks. The main weakness is that cross-model expert interchangeability is validated only on two model families and a narrow workload, with no analysis of per-position sensitivity or of cases where the most similar expert is not the one loaded. This leaves the central quality claim under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a single-GPU serving system for multiple fine-tuned Mixture-of-Expert (MoE) LLMs. To reduce memory, it consolidates experts across models by ranking (layer, expert) positions by L2 distance between the corresponding expert weights from different models, then loading experts in round-robin order up to a capacity C. To preserve output quality, it swaps only the non-expert layers at runtime when a request targets a different model, while using the preloaded shared experts (including foreign experts) whenever possible and fetching the target model's own expert on a miss. The authors evaluate QoS with Poisson arrivals on an A100 for two Mixtral-8x7B variants, reporting throughput comparable to serving a single model, an average TTFT increase from 0.89 s to 1.41 s, and an 85% average turnaround-time reduction versus NVIDIA MIG. Output quality is assessed with perplexity, MT-Bench, MMLU, HellaSwag, and TruthfulQA for Mixtral, and with SAMSum ROUGE scores for up to four Switch Transformer Base-8 variants, comparing against weight averaging.","tokens_in":14235,"tokens_out":8096,"duration_ms":88916,"significance":"If the claims hold, the paper addresses a real deployment problem: hosting several fine-tuned MoE LLMs on one GPU without time-slicing or MIG-style partitioning. The strengths are concrete: the system is implemented and measured, the code is provided, quality is evaluated on standard external benchmarks rather than fitted to the objective, and the comparison includes both a single-model throughput baseline and a hardware-partitioning baseline. The four-model Switch experiment is a useful stress test of scalability. The main risks are that the 'negligible' TTFT claim is contradicted by the paper's own Table 2, that the central memory-saving contribution is never quantified, and that the quality evaluation does not isolate the effect of the L2-similarity ranking. These are fixable with additional measurements and revised wording, and the core idea of runtime non-expert reconfiguration with shared experts is a reasonable and potentially useful systems contribution.","major_comments":[{"comment":"The claim that the increase in time-to-first-token is 'negligible' is not supported by the reported numbers. Table 2 shows average TTFT rising from 0.89 s for single-model serving to 1.41 s for the proposed system, an increase of 0.52 s (roughly 58% of the baseline), and Section 5.1 states that the non-expert swap itself takes approximately 1.2 s. Since TTFT is one of the two QoS metrics in the title and the abstract explicitly contrasts with 'serving a single model', this wording should be revised to a quantitative statement, e.g., the absolute increase in seconds and its relation to the MIG baseline, rather than 'negligible'.","section":"Abstract; Section 5.1, Table 2"},{"comment":"The paper's central memory-sharing contribution is not quantified. The loader takes an expert capacity C as input, but the evaluation never reports C, the resulting GPU memory footprint of the consolidated layout, or the measured expert hit/miss distribution during the QoS experiments. Table 1 gives per-layer latencies for 0, 1, and 2 expert hits, but without knowing how often each case occurs in the Mixtral and Switch workloads, the end-to-end throughput results cannot be interpreted. Please report C, the GPU memory used by the proposed layout versus single-model offloading and MIG, and measured hit rates for the evaluated workloads.","section":"Section 3.1, Algorithm 1; Section 5.1, Table 1"},{"comment":"The evaluation does not isolate the effect of the L2-similarity ranking on output quality. The proposed map is compared only against weight averaging; without a control in which experts are assigned to positions randomly (or by a deliberately poor similarity measure), the observed quality retention cannot be attributed to similarity-based consolidation rather than to general expert redundancy. In addition, Table 3 reports point estimates only, with no variance or number of MT-Bench judging runs, and the Mixtral evidence uses only two variants. A per-layer or per-position analysis of which substitutions are tolerated would directly test the load-bearing assumption and would strengthen the scalability claims.","section":"Sections 5.2 and 5.3, Tables 3-4"}],"minor_comments":[{"comment":"The caption says the values are normalized to their maximum values, but the table reports values in seconds; the caption and the text in Section 5.1 should be made consistent.","section":"Table 2"},{"comment":"The abstract claims an '85% average reduction in turnaround time' compared to MIG, but Table 2 implies 8.78 s versus 49.67 s, i.e., about 82.3%; please clarify how the average is computed and which arrival rates are included.","section":"Abstract; Section 5.1"},{"comment":"The throughput curves are described as averages over five runs, but no error bars or variance are shown; adding them would help assess whether the proposed system and single-model serving differ at the ridge points.","section":"Figure 4"},{"comment":"There is a typo: 'we asses the performance' should be 'we assess the performance'.","section":"Section 4.2"},{"comment":"The comment 'Load targetModel non-experts on GPU' should say 'non-expert layers' to avoid confusion with the expert offloading described elsewhere.","section":"Algorithm 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a systems/ML conference and the core idea is sound enough to warrant revision. The main blocking issues are quantitative: the TTFT claim needs to be made accurate, the memory savings need to be measured and reported, and the quality evaluation needs a control that isolates the similarity-based selection. If the authors can provide those, the paper would be a solid systems contribution; without them, the central 'QoS-efficient' and 'memory-saving' claims remain under-supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real systems contribution—sharing same-position experts across fine-tuned MoE variants and swapping only non-expert layers per request—and the Mixtral experiments back the main throughput claim. But the abstract calls a 0.5 s TTFT increase “negligible,” and the quality evaluation is thinner than the claim, especially for the Switch experiments.\n\nThe new thing is the specific combination. Expert pruning and model merging exist, but using L2 distance to build a round-robin expert map across models, then reconfiguring non-expert layers at request time, is not something I’ve seen. The authors also ship code (github.com/hamid-74/Multi-MoE) and report real measurements on a single A100: throughput roughly matching single-model serving, and 85% average turnaround reduction over MIG. That’s a concrete operational win for small-GPU shops.\n\nWhere it’s soft. First, the TTFT framing: Table 2 shows 0.89 s to 1.41 s. That’s a 0.5 s increase, roughly 58% relative, on a 20-token prompt. Not “negligible”; the paper itself later says the swap costs about 1.2 s, so the abstract should be toned down. Second, memory savings are never quantified. The whole pitch is fitting multiple MoEs on one GPU, but there’s no table of GPU footprints, expert capacity C, or hit-rate distribution. That’s a fixable omission, not a flaw. Third, the quality evidence is narrower than the claim. For Mixtral, two variants and a round-robin half/half expert split do give competitive perplexity and MT-Bench vs the base/instruct models—that’s genuinely encouraging. But for Switch, the evaluation only tests Model A’s SAMSum task, and models C and D score near zero on that task standalone, so adding them isn’t a hard stress test. The stress-test note’s point about no per-position sensitivity analysis is fair; it would strengthen the paper to show which (layer, expert) positions tolerate substitution and what happens when the most similar expert isn’t the one loaded. Fourth, no variance/error bars anywhere, and experiments are single-batch only, so the QoS numbers are a controlled baseline rather than production throughput.\n\nNone of this sinks the central argument. The L2-distance picture in Figure 2 is compelling, the design is sensible, and the Mixtral quality results would be surprising if the consolidation were badly broken. It’s a paper with an honest core and some overclaiming at the edges.\n\nBottom line: send it to review. A good referee will ask for the missing memory/hit-rate table, a corrected TTFT statement, and an additional Switch scenario where the merged model is asked to do each served model’s own task. That’s revision material, not rejection material.","headline":"Real systems idea with a useful consolidation trick; the paper oversells TTFT and the Switch quality test is too easy, but the core is solid and it deserves peer review.","tokens_in":14809,"tokens_out":3571,"would_cite":true,"duration_ms":37135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single GPU can serve multiple fine-tuned mixture-of-experts LLMs at near-single-model throughput by sharing similar experts and swapping only non-expert layers per request.","keywords":["mixture-of-experts","LLM inference","expert consolidation","model merging","runtime reconfiguration","multi-tenant GPU","quality of service","GPU memory"],"falsifier":"Run the same serving scheme on a pair of fine-tuned MoE variants whose same-position expert distances are artificially inflated (for example, by fine-tuning on disjoint domains or rotating expert vectors) while keeping benchmark performance intact, and measure whether perplexity and instruction-following scores collapse; if quality drops sharply even when the loaded expert is the closest one available, the consolidation premise fails.","tokens_in":13795,"feed_emoji":"⚡","tokens_out":6105,"duration_ms":52835,"temperature":0.7,"pith_summary":"This paper claims it can serve several fine-tuned mixture-of-experts LLMs from a single GPU without the usual performance collapse. Its proposal is to load one consolidated set of expert parameters chosen by similarity across the models, and to re-load only the non-expert transformer layers whenever the request switches models. On an A100 with two Mixtral-8x7B variants, the approach sustains throughput comparable to serving one model, adds roughly half a second of time-to-first-token, and cuts turnaround time by 85% compared with NVIDIA MIG. Quality benchmarks stay close to the individually fine-tuned models, and experiments with four Switch Transformer variants show the quality advantage over simple weight averaging grows with the number of merged models.","feed_headline":"One GPU serves multiple fine-tuned LLMs with 85% less turnaround time","feed_subtitle":"Similar experts are shared across models; non-expert layers swap per request, keeping quality competitive.","key_machinery":"The system has two mechanisms: similarity-based expert consolidation and runtime partial reconfiguration. Consolidation computes pairwise L2 distances between flattened expert weights at every (layer, expert) coordinate across all model pairs, ranks coordinates by total distance, and assigns each coordinate to one model in round-robin order until the GPU expert capacity is filled, so the loaded experts are equally representative of all served models. Reconfiguration swaps only the non-expert layers (attention, normalization, embedding, head) when the requested model changes, because those parameters are always active and contribute about 14 percent of the output; expert parameters are fetched from host memory only on a miss, preventing further quality loss.","core_discovery":"The central discovery is that fine-tuned MoE variants share enough structure at the expert level to be served from a common pool: experts at the same (layer, expert) position have L2 distances between 3.6 and 4.9, while cross-position distances range from 150 to 250, so a round-robin selection of the most similar experts yields a unified layout that represents all models. At inference time the system swaps only the non-expert parameters (about 3 GB, taking roughly one second over PCIe) when the target model changes, and otherwise routes each request's selected experts through the preloaded pool, fetching a true target-model expert only on a miss. The paper reports that this preserves output quality: perplexity on WikiText, C4, and PTB stays near the base model, MT-Bench average reaches 8.16 versus 8.13 for the instruct model, and ROUGE-1 on SAMSum degrades only from 0.49 to 0.46 when four models are merged, while the averaging baseline drops to 0.25.","pith_inferences":["If the same-position expert similarity pattern holds across other MoE families, the same consolidated layout could serve arbitrary collections of fine-tuned variants without retraining, making it a practical alternative to per-tenant GPU partitioning.","The round-robin assignment of expert slots by similarity rank is a heuristic; a policy that routes each request to the most similar loaded expert, or that caches the most conflicting experts, could push quality even closer to the dedicated models.","Because the non-expert swap cost is fixed per model switch, batching requests by target model would amortize the reload latency and push throughput beyond the reported single-model-equivalent figure.","The quality measurements do not isolate which expert positions tolerate substitution; a per-position sensitivity map derived from the distance data could predict when sharing becomes unsafe without running benchmarks."],"forward_implications":["A single 80GB GPU can serve at least two Mixtral-8x7B variants with throughput equal to serving one model at the same aggregate request rate, and with 85 percent lower average turnaround time than a two-instance NVIDIA MIG partition.","Switching the served model adds about 1.2 seconds of non-expert reload latency, but because this happens only when consecutive requests target different models, the average time-to-first-token increase is roughly 0.5 seconds.","The expert-sharing scheme keeps quality close to the dedicated models: perplexity stays within about 0.04 of the base model and MT-Bench scores within 0.03 of the instruct model in the two-variant Mixtral experiments.","When the number of served variants grows from two to four on Switch Transformer, ROUGE-1 on the SAMSum task falls from 0.49 to 0.46, whereas weight averaging falls from 0.42 to 0.25, showing the method's advantage widens with more models."],"supporting_citations":[{"why":"Supplies the single-model expert-offloading baseline and the PCIe transfer overhead numbers that motivate the multi-model consolidation problem.","marker":"Eliseev & Mazur, 2023"},{"why":"Describes NVIDIA MIG, the space-sharing baseline the proposed system is compared against for quality of service.","marker":"Choquette et al., 2021"},{"why":"Defines weight averaging, the model-merging baseline whose quality degrades as more models are merged.","marker":"Izmailov et al., 2018"},{"why":"Introduces the Switch Transformer architecture used for the four-variant scalability experiments.","marker":"Fedus et al., 2022"},{"why":"Introduces the Mixtral-8x7B models that are the primary evaluation family in the paper.","marker":"Jiang et al., 2024"},{"why":"Supports the observation that expert weights diverge more in deeper layers, which motivates measuring expert-to-expert distances for consolidation.","marker":"Shen et al., 2024"}],"fun_headline_variants":["One GPU serves multiple MoE LLMs with 85% less turnaround","Expert sharing and partial reconfiguration serve many LLMs on one GPU","Single-GPU serving of multiple fine-tuned MoE LLMs via expert reuse","85% faster multi-LLM serving by reusing similar experts","MoE LLMs share experts to fit one GPU, 85% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that experts at the same (layer, expert) position in different fine-tuned variants are interchangeable enough that substituting a foreign expert while keeping the target model's non-expert layers preserves task quality.","fun_headline_variants_meta":{"raw":{"variants":["One GPU serves multiple MoE LLMs with 85% less turnaround","Expert sharing and partial reconfiguration serve many LLMs on one GPU","Single-GPU serving of multiple fine-tuned MoE LLMs via expert reuse","85% faster multi-LLM serving by reusing similar experts","MoE LLMs share experts to fit one GPU, 85% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3663,"prompt_tokens":1003,"completion_tokens":2660,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2563}},"tokens_in":619,"tokens_out":2660,"duration_ms":16079,"temperature":1.0,"reasoning_tokens":2563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:41:11.834394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same serving scheme on a pair of fine-tuned MoE variants whose same-position expert distances are artificially inflated (for example, by fine-tuning on disjoint domains or rotating expert vectors) while keeping benchmark performance intact, and measure whether perplexity and instruction-following scores collapse; if quality drops sharply even when the loaded expert is the closest one available, the consolidation premise fails.","supporting_citations":[],"review_version":1}