{"id":"b239782e-7984-488d-b8dd-e8d4aee87b18","arxiv_id":"2608.01891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.","lead":"This paper presents AFlex, a serving framework that runs LLM Attention and FFN layers on separate GPU pools with independent frequency scaling. It reports up to 49% lower energy per token than disaggregated baselines while meeting latency targets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline energy savings are demonstrated only under P90 SLOs; if the SLO tail percentile is tightened to P99, the claimed up-to-49% reduction may not hold.","rationale":"The reader's weakest_assumption identifies the same load-bearing choice: P90 SLOs in §VI-A. I find this to be the most consequential threat to the central claim because the energy-saving mechanism is precisely a frequency-reduction policy that uses latency slack. All of AFlex's frequency choices—the ILP's baseline frequencies and the local DVFS controller's online selection—are gated by whether predicted latency fits the SLO. Moving the tail percentile from P90 to P99 directly shrinks the available slack and therefore the energy savings. The reported P90 margins (25% for TTFT, 19% for TPOT) are not large enough to assume P99 would also pass, and the paper presents no P99 data. Other concerns—missing source code, lack of run-to-run variance, absence of throttLL'eM/GreenLLM baselines—are real but less central: they affect reproducibility and completeness, not the internal logic of the energy-saving claim. The paper is internally coherent under the stated P90 definition; the issue is external validity under stronger SLOs. A conditional verdict remains appropriate: the system's value is plausible, but the headline should be scoped to P90 unless the P99 check passes.","tokens_in":19471,"tokens_out":7061,"duration_ms":88301,"concrete_test":"Rerun the end-to-end Conversation and Coding evaluations at the operating points that produce the largest claimed savings (e.g., 16 RPS on two nodes) with SLO compliance defined at P99 instead of P90: record P99 TTFT, P99 TPOT, and energy per token for AFlex and for the best baseline (DistServe/DynamoLLM). If P99 latencies exceed 400/120 ms, or if the energy-per-token reduction over the baseline falls below the paper's stated range, the headline claim is SLO-percentile-dependent and should be reworded. If per-request latencies were already logged, this can be computed offline without new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—AFlex reduces energy per token by up to 49% while satisfying TTFT and TPOT SLOs—rests on defining SLO compliance at the P90 percentile (§VI-A: 400 ms TTFT, 120 ms TPOT). This is load-bearing because AFlex's energy savings come from exploiting latency slack to lower operator frequencies: the Local DVFS Controller (Alg. 1) selects the lowest-energy feasible frequency pair, and feasibility is judged against the same P90 target. P90 leaves substantially more slack than the P99/P999 tail SLOs commonly used for interactive LLM serving. The reported margins are small: P90 TTFT is ≤302 ms against a 400 ms SLO (~25% headroom), and P90 TPOT is ≤97 ms against 120 ms (~19% headroom). If the SLO percentile were shifted to P99, the feasible frequency set F_tau would shrink, pushing the controller to higher frequencies and increasing energy per token. The paper does not report P99 latency or energy-per-token under P99 SLOs, so the headline savings are unverified under a stricter—and arguably more realistic—tail guarantee. This is not an internal inconsistency: the system does satisfy the stated P90 SLO. But the abstract's 'stringent SLOs' and the general claim of SLO-preserving energy reduction depend on this percentile choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AFlex, a framework for energy-efficient LLM serving that disaggregates attention (A) and feed-forward network (F) execution into separate GPU pools and independently scales GPU frequencies for the two operators across prefill and decode. A global resource controller solves an ILP (Eq. 1) to allocate A/F pairs, TP degrees, and baseline frequencies; a local DVFS controller (Alg. 1) adjusts frequencies at runtime; an interleaved A/F pipeline with dynamic microbatch depth and adaptive batching reduces bubbles. The system is implemented in SGLang and evaluated on A800 GPUs with Qwen3-32B and Mixtral-8×7B under Azure Conversation/Coding traces and controlled workloads. The reported results show up to 49% energy-per-token reduction over DistServe and 48% over DynamoLLM while meeting P90 TTFT (400 ms) and TPOT (120 ms) SLOs.","tokens_in":19768,"tokens_out":6638,"duration_ms":72379,"significance":"If the quantitative claims hold, AFlex is a meaningful advance over phase-level DVFS for LLM serving: it exposes operator-level frequency sensitivity and demonstrates measured, end-to-end energy savings on real hardware. The strengths of the paper include the full system implementation, the clear ILP formulation, the ablation isolating global scheduling and local DVFS, the low measured overheads, and the high predictor accuracy (R²≥0.97). The main caveats are that SLO compliance is evaluated at P90 rather than a stricter tail percentile, that two recent frequency-scaling baselines are omitted, and that all measurements are single runs without variance. These issues are fixable and do not invalidate the design, but they currently limit the strength of the headline claims.","major_comments":[{"comment":"SLO compliance is defined at the P90 percentile (400 ms TTFT, 120 ms TPOT), and the local DVFS controller uses exactly these targets to select the lowest-energy feasible frequency pair: the prefill budget is s = min_r(TTFTSLO(r) - Queue(r)) and Alg. 1 constructs F_τ = {f' : t̂(D,M,f') ≤ τ}. The reported P90 headroom is about 25% for TTFT (≤302 vs 400 ms) and 19% for TPOT (≤97 vs 120 ms). Since the energy savings come from exploiting this slack, tightening the SLO percentile to P99 or P999 will shrink F_τ and likely reduce the savings. The paper does not report P99/P999 latencies or energy-per-token under stricter SLOs. This is load-bearing: the abstract's 'stringent SLOs' and the SLO-preserving claim depend on the percentile choice. Please add a tail analysis, e.g., report P99/P999 latencies and re-run under P99 SLOs, or explicitly justify P90 for the target workloads.","section":"§VI-A, §IV-D"},{"comment":"The baseline set omits throttLL'eM [14] and GreenLLM [16], both of which are SLO-aware DVFS systems for LLM serving. The abstract and conclusion claim up to 48% reduction 'over frequency-scaling systems,' but the evaluation only includes DynamoLLM and BiScale. This overstates the state of the art. Please add these baselines or narrow the claim to 'over the evaluated baselines.'","section":"§VI-A"},{"comment":"All quantitative results are single point estimates; no error bars, confidence intervals, or repetition counts are reported. NVML energy measurements are subject to run-to-run variability (DVFS transitions, temperature, system noise). To support the quantitative margins (e.g., 19.6%–46.8% over the best baseline), the authors should run each configuration multiple times and report mean±std (or box plots). Without this, the headline energy savings may not be statistically robust.","section":"§VI"},{"comment":"The design introduces several tunable parameters that are not studied: capacity margin ε (Eq. 1f), batch-size change threshold δ_b and window constants κ/W_min (Alg. 1), and the workload-monitor thresholds (§IV-B). The decode DVFS controller's switching frequency is directly governed by δ_b and κ, and these values are not justified or ablated. A sensitivity analysis over these parameters is needed to show the results are not tuned to a single set of constants.","section":"§IV-D, §VI"}],"minor_comments":[{"comment":"The overlap model assumes communication is perfectly amortized as t_comm/M for M≥2. Please provide a derivation or measurement of overlap efficiency, particularly for cross-node A/F transfers.","section":"§IV-C, Eq. (2)"},{"comment":"Several panels need better captions. Fig. 5 combines two different units (s and ms) in one panel; Fig. 12's red dashed line is not explained in the caption; Fig. 4's relative-range color scales are not labeled.","section":"Figures 4, 5, 12"},{"comment":"The sentence 'AFlex reduces energy per token by 19.6%–46.8% relative to the best-performing baseline' should be tied to the specific workloads/RPS and reconciled with the 49%/48% numbers reported for individual baselines.","section":"§VI-B"},{"comment":"The 'first' claim for combining operator-disaggregated execution with independent frequency scaling would benefit from a brief distinction from operator-disaggregation systems [34], [40] that do not control frequency.","section":"§IV"},{"comment":"Reference [3] is a generic GitHub URL; please cite a specific version or dataset identifier so the traces are reproducible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the venue and is likely to be a solid systems contribution after revision. The most consequential risk is the P90 SLO definition: the energy savings are, by construction, a function of the latency slack at the chosen percentile. I would ask the authors to add a tail-latency analysis and, if possible, re-run under P99 SLOs. The missing throttLL'eM/GreenLLM baselines are also important for the 'state-of-the-art' claim. I do not see a fatal flaw; the central approach is sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine systems contribution that deserves a real referee, but the headline energy savings are conditional on the P90 SLO definition, and that caveat needs to be front and center.\n\nWhat's new: AFlex is the first framework that combines operator-level disaggregation of Attention and FFN with independent per-operator DVFS. The observation that these two operators have different energy-optimal frequencies (average gap 429 MHz) is well supported by the measurements in Section III, and the design that follows—global ILP for provisioning, local window-based DVFS for slack recovery, and an interleaved pipeline with dynamic microbatch depth—is coherent and carefully motivated. The evaluation is honest: energy is measured with NVML, not predicted, across two models and production traces, with ablations and overhead tables. That's more than many systems papers do.\n\nSoft spots: The biggest one is the SLO percentile. All compliance is reported at P90 (TTFT 400ms, TPOT 120ms). At P90 the headroom is 25% on TTFT and 19% on TPOT; at P99 that margin would shrink and the DVFS controller would be forced to higher frequencies, meaning the up-to-49% savings would likely drop. The paper doesn't report P99 latency or energy under a P99 SLO, so the general claim of 'stringent SLOs' is overstated. This is not a fatal flaw, but it is load-bearing. Also missing: repeated runs or error bars, and a comparison against throttLL'eM and GreenLLM, both of which are cited in related work. The in-sample profiling of the same A800 hardware is a standard practice in systems research, not a circularity problem, but it does mean the numbers are testbed-specific.\n\nThe bottom line: this is a well-executed systems paper with real novelty in a niche that matters. The conditional verdict is right. I'd want to see P99 sensitivity and at least a brief discussion of why P90 is the right tail metric before accepting. With those additions, it would be a solid contribution.","headline":"Real systems contribution with genuine novelty, but the headline energy savings are measured only against P90 SLOs and need P99 sensitivity before I'd sign off.","tokens_in":20330,"tokens_out":2554,"would_cite":true,"duration_ms":25808,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AFlex shows that assigning Attention and FFN separate GPU pools with independent clock speeds cuts LLM serving energy per token by up to 49% while keeping P90 TTFT and TPOT within SLOs.","keywords":["LLM serving","energy efficiency","DVFS","Attention-FFN disaggregation","GPU frequency scaling","SLO-aware scheduling","interleaved pipeline","dynamic microbatch depth"],"falsifier":"Re-run the same workloads and baselines on the same hardware with P99 TTFT and TPOT SLOs (instead of P90). If AFlex's energy-per-token advantage over the disaggregated and frequency-scaling baselines drops below roughly 10% (or to zero), the central claim that operator-level DVFS yields large SLO-safe savings is falsified.","tokens_in":19319,"feed_emoji":"⚡","tokens_out":7351,"duration_ms":66986,"temperature":0.7,"pith_summary":"This paper argues that running every GPU operator at a single shared frequency—or even one frequency per inference phase—wastes energy, because Attention and feed-forward network (FFN) operators respond very differently to clock speed. To prove the point, the authors build AFlex, a serving framework that splits Attention and FFN onto separate GPU pools, gives each pool its own dynamic frequency, and schedules resources with a global optimizer. Across production workloads on dense and mixture-of-experts models, AFlex reduces energy per token by up to 49% relative to state-of-the-art disaggregated serving and 48% relative to frequency-scaling systems, while keeping P90 time-to-first-token and per-output-token latencies within their SLOs. The practical stake is that datacenter LLM serving could cut a large fraction of its GPU energy bill without asking users to accept slower responses.","feed_headline":"Two clock speeds per layer cut LLM serving energy up to 49%","feed_subtitle":"Giving Attention and FFN separate clocks trades slack for power, cutting energy per token up to 49% at the same SLOs.","key_machinery":"The load-bearing mechanism is a two-level control plane plus a pipelined data plane. The global scheduler uses an offline latency/energy predictor and an ILP to allocate GPU counts, tensor-parallel degrees, and baseline frequencies to the four operator-phase pools; the local DVFS controller refines frequencies at runtime, using window-based amortization for decode frequency switches. The data plane runs an interleaved Attention–FFN pipeline that overlaps hidden-state transfers with computation and tunes microbatch depth and request batch size to reduce bubbles. Together these let AFlex lower the frequency of the less latency-sensitive operator (typically Attention) without violating per-requ","core_discovery":"The central discovery is that Attention and FFN operators have distinct, workload-dependent frequency sensitivities: over the profiled configurations, Attention reaches its energy-optimal frequency on average 429 MHz below FFN, and a single shared frequency cannot match both operators' optima. AFlex exploits this by formalizing four operator-phase pools (prefill Attention, prefill FFN, decode Attention, decode FFN), each with its own GPU allocation, tensor-parallel degree, and frequency. A global scheduler solves an integer linear program to select these configurations under TTFT and TPOT constraints, while a local controller adapts frequencies at runtime, and an interleaved A/F pipeline wit","pith_inferences":["The reported savings are anchored to P90 latency SLOs; if operators require P99 or stricter tail guarantees, the available frequency slack shrinks and the 49% and 48% numbers are likely optimistic.","The operator-split idea may extend to finer-grained execution units (e.g., expert layers in MoE models) wherever frequency sensitivity differs, potentially yielding additional savings on models like Mixtral.","The 5-minute global replanning window means workloads that shift faster than that would rely on the local DVFS controller alone; a learned or online planner could close that gap.","The quantitative results are measured on A800 GPUs; on hardware with a narrower DVFS range or a different compute/memory balance, the magnitude of savings would change, though the qualitative principle—match frequency to the bottleneck operator—should carry over."],"forward_implications":["Energy per token can drop by up to 49% relative to state-of-the-art disaggregated serving and 48% relative to frequency-scaling systems while keeping P90 TTFT and TPOT within SLOs.","Operator-level disaggregation and DVFS are complementary: the global scheduler alone (without local DVFS) accounts for a large share of savings, and local DVFS adds up to a further 20–47% depending on workload.","The interleaved A/F pipeline with dynamic microbatch depth and adaptive batching reduces pipeline bubbles by 8–65% and per-layer latency by up to roughly 38%, which is what makes the frequency savings SLO-safe.","The approach generalizes across dense and mixture-of-experts models and up to 32 GPUs, with energy savings over the best baseline remaining at least about 20%.","The system adds negligible planning overhead (ILP solves in under 0.6 seconds, less than 0.2% of the scheduling window) and reduces TP reconfiguration time by 52–72% via incremental weight resharding."],"fun_headline_variants":["Attention and FFN get distinct clocks, cutting LLM energy up to 49%","Separate clock speeds for attention and FFN save up to 49% energy per token","Operator-level DVFS: attention and FFN frequencies differ, saving up to 49%","AFlex: Separate clocks for attention and FFN reduce energy up to 49%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation defines SLO compliance at the P90 percentile of TTFT and TPOT; if the SLO must be met at a stricter tail such as P99, the latency slack that AFlex trades for energy savings narrows, so the claimed energy reductions may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Attention and FFN get distinct clocks, cutting LLM energy up to 49%","Separate clock speeds for attention and FFN save up to 49% energy per token","Operator-level DVFS: attention and FFN frequencies differ, saving up to 49%","AFlex: Separate clocks for attention and FFN reduce energy up to 49%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000741,"raw_usage":{"total_tokens":3168,"prompt_tokens":795,"completion_tokens":2373,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2280}},"tokens_in":539,"tokens_out":2373,"duration_ms":19178,"temperature":1.0,"reasoning_tokens":2280,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:44:30.939666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same workloads and baselines on the same hardware with P99 TTFT and TPOT SLOs (instead of P90). If AFlex's energy-per-token advantage over the disaggregated and frequency-scaling baselines drops below roughly 10% (or to zero), the central claim that operator-level DVFS yields large SLO-safe savings is falsified.","supporting_citations":[],"review_version":1}