{"id":"0821f917-19ad-419b-bcb9-3619b3bfde67","arxiv_id":"2501.13111","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"iServe uses small proxy models, called fingerprints, to estimate latency and memory for many LLM deployment configurations and then selects the configuration that best satisfies a user's intent.","lead":"The paper presents iServe, a serving system that picks LLM deployment settings such as parallelism and quantization automatically from a user's stated goal, like low latency or low cost. It profiles a miniature 'fingerprint' version of each model to predict how the full model behaves, cutting profiling cost by about 6x according to the authors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline latency/throughput claims are measured with batching disabled for all systems; they may not hold against continuous-batching production serving (e.g., vLLM), leaving the central performance claim unestablished.","rationale":"I read the paper in good faith. The fingerprint-based profiling is a genuine engineering contribution, and the linearity assumption between hidden-layer count and latency/memory is tested on six decoder-only models with reported errors of 4.91% (latency) and 6.92% (memory). That assumption is plausible for homogeneous decoder layers, though not derived from first principles. The more load-bearing gap for the central claim is external validity: the headline latency, SLO, and throughput numbers are produced with batching disabled for every system. Modern LLM serving frameworks achieve most of their throughput through continuous batching, so comparing only batch-size-1 runs does not establish superiority over state-of-the-art serving systems. The paper even states that batching is excluded, but it does not quantify the impact of that exclusion. The reader's weakest_assumption focused on the linear scaling, and the reader's rationale did flag batching as a main weakness, so my read partially agrees with the reader. I keep the verdict UNCHANGED because the conditional accept is still appropriate: the concern is testable and would not invalidate the fingerprint-profiling idea, but it must be addressed before the comparative performance claims can be accepted at face value.","tokens_in":29889,"tokens_out":6796,"duration_ms":64242,"concrete_test":"Enable TensorRT-LLM's in-flight batching (and, if possible, vLLM with continuous batching) for iServe and all baselines on the same 8-GPU testbed and Azure traces (code and conversation, rate-scale 0.1–0.4); re-measure median/p95 latency, SLO attainment, and GPU throughput while keeping iServe's fingerprint-based configuration selection and placement policy. If iServe's relative gains shrink by more than a few percent or reverse, the headline claims are conditional on batch-size-1 serving; if the gains persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims iServe 'best meets user intents' with 77.62% lower latency, 7.09x fewer SLO violations, and 4.72x higher GPU throughput versus state-of-the-art systems. However, Sections 5.3 and 6 state that batching is disabled for all systems, and the evaluation compares only against TensorRT-LLM PP/TP, Accelerate, and AlpaServe, all at batch size 1. Production LLM serving systems use continuous batching (vLLM's PagedAttention, TensorRT-LLM's in-flight batching) precisely because it raises GPU throughput by scheduling multiple sequences per iteration; at batch size 1, GPUs are underutilized and throughput numbers are not comparable to the state of the art. Thus the headline gains may be an artifact of comparing configuration/placement choices in an unrepresentative serving regime, rather than evidence that iServe's fingerprint-based selection provides the claimed improvement over modern systems. The paper acknowledges this limitation but provides no experiment quantifying its effect. This is load-bearing because the central claim is explicitly comparative: the claimed reductions are relative to 'state-of-the-art' systems running under a constraint that no production system would adopt.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"iServe is an LLM serving system that automates deployment configuration selection across parallelism, quantization, and pruning. The central idea is an 'LLM fingerprint': a reconstruction of a model with 1-2 hidden layers that is profiled on a few configurations, after which latency and memory are linearly extrapolated to the full model. A deployment controller ranks configurations by user intent (minimize latency, cost, memory, GPU-hours, or meet SLO targets) and a load-aware placement policy assigns models to GPUs. The evaluation, on an 8-GPU A6000 cluster with six LLMs and Azure traces, reports a 77.62% latency reduction, 7.09x fewer SLO violations, 4.72x higher GPU throughput, and 6.05x lower profiling cost relative to TensorRT-LLM static-parallelism baselines, Accelerate, and AlpaServe.","tokens_in":30142,"tokens_out":9079,"duration_ms":80957,"significance":"If the claims hold, iServe addresses a real pain point: the cost of profiling deployment configurations for LLMs. The fingerprint idea is elegant, and the paper ships a substantial empirical study: design-space exploration of profiling methods (L1/L2, M1/M2/M3), placement policies, multiple intents, two Azure traces at four load levels, and profiling accuracy measured against ground-truth full-LLM measurements. The profiling error is low (4.91% latency, 6.92% memory), and the evaluation is not circular: fingerprint predictions are checked against independent full-LLM measurements. However, the contribution's central comparative claims are currently restricted to an unrepresentative batch-size-1 regime, and the linear extrapolation assumption is only validated on dense decoder-only models. With additional experiments or narrower claims, this could be a strong systems paper.","major_comments":[{"comment":"The evaluation disables batching for all systems (stated in §5.3 and §6: 'we disable batching in this work' and 'we exclude runtime preemption, swapping, and batching') and compares only against TensorRT-LLM PP/TP static-configuration baselines, Accelerate, and AlpaServe. None of these baselines uses continuous batching or paged KV-cache, which are standard techniques in production LLM serving systems such as vLLM and TensorRT-LLM's in-flight batching. The headline claims in the abstract (77.62% latency reduction, 7.09x fewer SLO violations, 4.72x GPU throughput improvement) are therefore measured in a regime where GPUs are underutilized and throughput is not comparable to the state of the art. The manuscript acknowledges this limitation but does not quantify its effect; this is load-bearing because the central claim is explicitly comparative.","section":"§5.3, §6, §8"},{"comment":"The fingerprint-based extrapolation assumes exact linear scaling of memory footprint and inference latency with the number of hidden layers. This assumption is validated on six dense decoder-only LLMs, but the Introduction and Section 2 present Mixtral MoE as a target architecture. MoE layers contain routing and multiple experts, and encoder-decoder or multimodal models contain heterogeneous layer types; the linear model is empirical and not derived from architectural invariants. The low estimation errors for the evaluated dense models (Section 8.4) are reassuring, but they do not establish the general scaling law. The paper should either validate the linearity on at least one non-dense or heterogeneous architecture, or explicitly restrict the scope and adjust the abstract's 'various LLMs' claim.","section":"§5, Figure 7"},{"comment":"The paper's claim in §6 that the profiler 'can easily predict LLM latency and memory needs for different batch sizes using simple modeling (e.g., linear regressors)' is unsupported; no batching-enabled experiment is presented, and Equation (1) does not include batching or queueing effects. Since the evaluation's SLO attainment results (§8.3) are derived in the same batch-size-1 setting, the reported 7.09x SLO-violation improvement may not carry over to production workloads. Please either add a batching-enabled experiment or remove or substantially temper this claim.","section":"§5.3, §6"}],"minor_comments":[{"comment":"The 'GPU throughput' metric used throughout Figures 11-16 is reported in req/s, but the denominator (per GPU, per server, or cluster-wide) and the time averaging are not defined. Please state the definition once in the Metrics paragraph.","section":"§8, Metrics"},{"comment":"The hybrid placement policy depends on a load threshold θ, but the paper does not state how θ is set or whether it is tuned per trace. Please provide the value or the selection procedure to make the evaluation reproducible.","section":"§6, Hybrid placement policy"},{"comment":"The main text reports results for the 'cost' intent without specifying which of the three cost formulations from §4.1 is used. Please state the formulation used in Figure 13 and confirm whether the other two formulations yield similar conclusions.","section":"§8.2"},{"comment":"The sentence 'iServe average latency and memory estimation error is 4.91% (< 50ms error) and 6.92% (1.74GB)' is unclear: are these mean or median absolute errors, and across which configurations and models? Please clarify.","section":"§8.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid systems contribution in terms of engineering and design-space study. The main concern is the mismatch between the broad comparative claims and the batch-1, no-continuous-batching evaluation. I would recommend that the authors add a vLLM or TensorRT-LLM in-flight batching baseline (or at least a batch-size > 1 experiment) and re-evaluate the headline numbers; if the gains persist, the paper is publishable. Also, the MoE/Mixtral mention in the introduction raises expectations that the evaluation does not meet; either add a MoE validation or soften the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. The fingerprint idea is genuinely new: profile a 1-2 hidden-layer proxy of an LLM, linearly extrapolate latency and memory to the full model, and use those estimates to navigate a large deployment configuration space (parallelism, quantization, pruning). Second, the headline numbers—77.62% lower latency, 4.72x higher GPU throughput, 7.09x fewer SLO violations—should be read with a heavy caveat: the evaluation disables batching for all systems and runs at batch size 1, with no continuous-batching baseline like vLLM. That caveat is not hidden; the paper states it in Section 5.3 and mentions it again in Section 6, but it is never quantified, and it directly affects the throughput comparison.\n\nWhat the paper does well: the fingerprint method is a real reduction in profiling cost. The design exploration of profiling strategies (M1-M3 for memory, L1-L2 for latency) is thorough and honest, with average latency estimation error around 4.91% and memory error around 6.92%, measured against ground truth. The evaluation covers six LLMs, two Azure traces, four intents, and several baselines (TensorRT-LLM PP/TP, Accelerate, AlpaServe, AlpaServe*). The placement policy evaluation (least-loaded vs packing vs hybrid) is sensible, and the modular architecture is a plus.\n\nWhere the soft spots are: the batching issue is the biggest one. At batch size 1, GPU throughput is dominated by launch overheads and idle time, and production LLM serving systems use continuous batching precisely to keep GPUs busy. iServe uses FCFS scheduling with no batching, so claiming a 4.72x throughput improvement over TensorRT-LLM under those conditions is not convincing evidence that iServe would beat a continuous-batching system in a realistic deployment. The paper says disabled batching is 'similar to previous systems,' but AlpaServe also predates the continuous-batching era. A serious revision should add a vLLM or TensorRT-LLM in-flight batching baseline, or restrict the throughput claim to single-query latency-focused serving.\n\nThe linear scaling assumption is validated on six decoder-only LLMs; it could break for MoE or cross-attention models, but the error is measured and the method is empirical, so I would call that a minor concern rather than a fatal one. The preprint does not include the open-sourced code or data, which limits reproducibility.\n\nBottom line: the fingerprint profiling contribution is real and deserves referee time. The paper needs another evaluation round in a more production-realistic regime before the headline comparative claims are credible. I would send it to peer review with a request to fix the batching baseline issue and release the code.","headline":"Genuinely novel fingerprint-based profiling for LLM serving, but the headline throughput/latency claims rest on a batch-size-1 evaluation that needs a continuous-batching baseline before they are credible.","tokens_in":30644,"tokens_out":3797,"would_cite":true,"duration_ms":33771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"iServe claims LLM serving can be automated by profiling a lightweight fingerprint instead of the full model, then automatically choosing the deployment configuration that best matches the developer's stated intent.","keywords":["intent-based serving","LLM inference","model fingerprinting","deployment configuration search","profiling cost","load-aware placement","parallelism","quantization"],"falsifier":"Run a full model with 4, 8, 16, 32, and 64 layers under a fixed configuration and compare measured latency and memory to the linear extrapolation from the one- and two-layer fingerprints; if the residual grows nonlinearly or changes sign across layer counts, the extrapolation is unsound. A quicker check is a mixture-of-experts model whose per-layer behavior differs, since the linearity claim assumes homogeneous layers.","tokens_in":29716,"feed_emoji":"🤖","tokens_out":4849,"duration_ms":46496,"temperature":0.7,"pith_summary":"iServe aims to remove the manual, expert step of choosing how an LLM is deployed. The paper's core claim is that the latency and memory of a decoder-only LLM can be estimated across hundreds of deployment configurations by profiling a \"fingerprint\" — the same architecture with at most two hidden layers — because both metrics scale linearly with layer count. If true, a serving platform could automatically pick the parallelism, quantization, and pruning combination that best matches a developer's intent (lower latency, lower cost, or a target for either) without expensive full-model profiling. The paper reports that this reduces latency by 77.62%, SLO violations by 7.09x, and profiling GPU-hours by 6.05x compared to prior systems.","feed_headline":"Tiny fingerprints pick the best LLM serving config","feed_subtitle":"Profile a 2-layer stand-in, extrapolate to every deployment, and hit latency or cost goals","key_machinery":"The fingerprint is a reconstructed model that keeps the embedding layer, hidden layers, normalization layers, linear layers, and softmax of the original but with at most two hidden layers, so it is much lighter: Llama-2-70B's fingerprint is 41.59x smaller and 9.75x faster. The argument is carried by two decompositions. Memory splits into weights ($W$), activations ($A$), and pipeline activations ($A_P$) whose replication under tensor and pipeline parallelism is known, and latency splits into time-to-first-token and time-per-output-token, each decomposed into per-hidden-layer and \"other\" components. Observing the fingerprint under three parallelism configurations supplies enough equations to solve for the memory components, and observing two output lengths on two fingerprint sizes supplies enough equations to solve for the latency components; linear scaling then predicts every configuration of the full LLM.","core_discovery":"On the paper's own terms, the discovery is a fingerprint-based serving pipeline: a system that profiles a one- or two-layer stand-in of an LLM under a few configurations, solves small systems of equations to recover per-layer memory and latency components, extrapolates to the full model, and then greedily ranks configurations by the user's intent and maps the chosen deployment onto available GPUs. The mechanism rests on the empirical observation that memory and latency grow linearly with the number of hidden layers, which the paper validates on six decoder-only LLMs. With this, iServe reports that it best meets user intent across varied workloads, cutting latency by 77.62%, cost by 86.70%, SLO violations by 7.09x, and profiling cost by 6.05x compared to state-of-the-art baselines.","pith_inferences":["If the linear scaling with layer count generalizes beyond the six decoder-only LLMs tested, the fingerprint idea could extend to mixture-of-experts or vision-language models, though those architectures would need extra profiling for their non-identical layers.","The intent interface suggests a natural next step: adding accuracy as a first-class intent, which the paper explicitly leaves as future work.","A testable consequence of the method is that estimation error should stay roughly constant in absolute terms as models grow, since errors for smaller models are a larger fraction of their footprint; the paper's error analysis already points in that direction."],"forward_implications":["Developers could declare intent (minimize latency or cost, or meet a target for either) and get a deployment choice automatically, instead of hand-searching hundreds of configurations.","Profiling for a new LLM could run on spare GPUs in the serving cluster rather than dedicated hardware, because fingerprints are small and fast enough to avoid disrupting inference.","The system can adapt placement to fluctuating load, packing LLMs onto fewer GPUs at low load and spreading them at high load to reduce latency.","SLO attainment improves substantially when latency SLOs are given, and cost, memory, and GPU-hours intents can all be optimized from the same profile data.","The same estimated metrics support multiple intents simultaneously, so one profiling run yields the configuration map for latency, cost, memory, and GPU-hour objectives."],"supporting_citations":[{"why":"Provides the primary state-of-the-art serving baseline that auto-selects parallelism, against which iServe compares serving performance, SLO attainment, and profiling cost.","marker":"[61]"},{"why":"Supplies the real-world Azure LLM inference traces used to evaluate iServe under varying query arrival patterns and loads.","marker":"[84]"},{"why":"The inference serving framework iServe is built on, and the source of two static baseline configurations.","marker":"[80]"},{"why":"Provides the fully sharded data parallel baseline that replicates a non-expert deployment approach.","marker":"[103]"},{"why":"Motivates the parallelism and latency decomposition used in the paper and provides context for the configuration search space.","marker":"[74]"}],"fun_headline_variants":["Fingerprint profiling picks optimal LLM serving","Small stand-in model guides LLM serving config","iServe slashes latency 77% and SLO violations 7x","Profiling a tiny LLM tunes the full model's serving","Intent-based serving: no costly exhaustive profiling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim collapses if latency and memory do not actually scale linearly with the number of identical hidden layers, so that a one- or two-layer fingerprint mispredicts the full LLM's behavior.","fun_headline_variants_meta":{"raw":{"variants":["Fingerprint profiling picks optimal LLM serving","Small stand-in model guides LLM serving config","iServe slashes latency 77% and SLO violations 7x","Profiling a tiny LLM tunes the full model's serving","Intent-based serving: no costly exhaustive profiling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3264,"prompt_tokens":971,"completion_tokens":2293,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2214}},"tokens_in":587,"tokens_out":2293,"duration_ms":17738,"temperature":1.0,"reasoning_tokens":2214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:31:42.447845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a full model with 4, 8, 16, 32, and 64 layers under a fixed configuration and compare measured latency and memory to the linear extrapolation from the one- and two-layer fingerprints; if the residual grows nonlinearly or changes sign across layer counts, the extrapolation is unsound. A quicker check is a mixture-of-experts model whose per-layer behavior differs, since the linearity claim assumes homogeneous layers.","supporting_citations":[{"cited_title":"Splitwise: Efficient gen- erative LLM inference using phase splitting","cited_arxiv_id":null,"evidence_quote":"Supplies the real-world Azure LLM inference traces used to evaluate iServe under varying query arrival patterns and loads."},{"cited_title":"TensorRT-LLM","cited_arxiv_id":null,"evidence_quote":"The inference serving framework iServe is built on, and the source of two static baseline configurations."},{"cited_title":"Accelerate: Training and inference at scale made simple, efficient and adaptable","cited_arxiv_id":null,"evidence_quote":"Provides the fully sharded data parallel baseline that replicates a non-expert deployment approach."},{"cited_title":"SpotServe: Serving Generative Large Lan- guage Models on Preemptible Instances","cited_arxiv_id":null,"evidence_quote":"Motivates the parallelism and latency decomposition used in the paper and provides context for the configuration search space."}],"review_version":1}