{"id":"9b5e6660-21a3-425e-ba05-acfd0b6a7ea3","arxiv_id":"2411.11560","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A topology-aware preemption scheduler for co-located LLM workloads raises topology affinity hit rate from 44.5% to 100% in simulation, but the 55% performance improvement is inferred, not directly measured.","lead":"This paper presents a scheduler that considers server hardware topology when choosing which low priority jobs to evict for high priority LLM services. In simulation, it raises the rate at which freed resources match a service's NUMA and socket preferences from 44.5% to 100%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 55% performance improvement is asserted, not measured: Section 5's sole metric is the same topology-affinity score that Eq. 1 optimizes, so the 100%-vs-45% result mainly shows self-consistency.","rationale":"The reader identified calibration of the topology scoring function as the weakest assumption. I see a more fundamental issue underneath that calibration concern: the evaluation metric is the scheduler's own objective. Section 2.2 defines scheduled performance as a performance gain affected by resource scheduling, but Section 5 measures only a binary hit rate computed from the same NUMA/socket preference that Eq. 1 maximizes. Therefore the headline 55% improvement is unsupported regardless of how much NUMA affinity actually matters. The proposed concrete test would settle the issue by measuring real LLM serving performance across the three placement classes and recomputing the aggregate gain. This is not an attack on the system: FlexTopo may well increase topology-affinity hit rate or reduce TopologyAffinityError failures, and the paper could be accepted with that narrower claim. I retain the reader's CONDITIONAL verdict, with the condition being an end-to-end performance measurement or a substantially narrowed claim.","tokens_in":13669,"tokens_out":4435,"duration_ms":45363,"concrete_test":"Run the Section 5 simulation on the same 100-node 4090 topology, but after each of the 5000 preemption decisions, measure actual LLM serving performance (e.g., p99 time-to-first-token or requests/sec with vLLM or TensorRT-LLM) for the chosen placement class: same-NUMA, same-socket-different-NUMA, and cross-socket. Aggregate the weighted performance gain of FlexTopo over Godel's standard preemption. If the weighted gain is not close to 55%, the headline claim should be narrowed to hit-rate or scheduling-failure-rate reduction.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim ('improves overall scheduled performance for LLM workloads by 55%') rests entirely on the 'topology affinity hit rate' reported in Section 5. That hit rate is defined by whether freed resources land in the same NUMA node or socket as the preemptor, which is precisely the objective encoded in the scoring function T(C_flextopo) in Section 3.4, Eq. 1. The scheduler maximizes S(C) = alpha * 1/sum(priority) + (1-alpha) * T(C_flextopo), and then the evaluation counts placements satisfying that same preference as 'hits'. A scheduler optimized for a score naturally achieves that score, so 100% versus 45% demonstrates only that FlexTopo optimizes its own objective, not that LLM scheduled performance improves by 55%. No end-to-end LLM latency, throughput, or goodput measurement appears anywhere in the paper; the '55% improvement in Scheduled Performance' in Section 5 directly conflates hit-rate gain with performance gain. Additionally, the numerical values assigned to high/medium/low in T are unspecified, and alpha is not reported, so even the hit-rate comparison is under-determined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FlexTopo, a graph-based representation of per-node hardware topology and allocation state, and integrates it into the Gödel scheduler as a preemption plugin. The scheduler uses a weighted scoring function combining victim priority and topology affinity, plus an incremental minimal preemption algorithm to limit search cost. The evaluation, conducted in a 100-node KWOK simulation and a near-production cluster, reports an increase in topology-affinity hit rate from 44.5% to 100% and claims a 55% improvement in scheduled performance for LLM workloads.","tokens_in":13911,"tokens_out":3977,"duration_ms":38448,"significance":"If the claimed performance improvement were supported by end-to-end measurements, the work would be a useful practical addition to topology-aware scheduling: victim selection during preemption is a genuine gap in existing Kubernetes/Gödel schedulers, and the open-source implementation is a concrete artifact. The proposed unified graph representation and the incremental search heuristic are reasonable engineering contributions. However, the current evidence does not substantiate the headline performance claim, because the evaluation metric is essentially the same topology score that the scheduler optimizes.","major_comments":[{"comment":"The only quantitative success metric, topology-affinity hit rate, is defined by same-NUMA/same-socket placement, which is exactly the objective T(C_flextopo) in Eq. (1). A scheduler that maximizes this score will naturally achieve a high hit rate, so the 100%-vs-44.5% result demonstrates self-consistency rather than an improvement in LLM scheduled performance. The paper contains no measurement of end-to-end latency, throughput, TTFT, or goodput for any LLM workload, and the phrase \"55% improvement in Scheduled Performance\" in Section 5 is therefore unsupported. Re-run the comparison with actual serving metrics (e.g., p50/p99 token-generation latency, throughput, or the number of placement failures observed by the kubelet) or restrict the claims to hit-rate improvement.","section":"Section 5, Table 4"},{"comment":"The scoring function uses α, whose value is never reported, and a piecewise T(C_flextopo) whose high/medium/low numeric values are never specified. This underdetermines the reported comparison: depending on α and the numeric mapping, the tie-breaking between priority and topology can change the selected victims, so the 100% hit rate cannot be reproduced by a reader. Please report the exact values, justify them, and include a sensitivity analysis over α and the topology-score values.","section":"Section 3.4, Eq. (1)"},{"comment":"The simulation uses one fixed workload mix, one GPU server type (8×RTX 4090), and one saturation state, and no replication or confidence interval is reported for the 5000-preemption hit-rate numbers. The \"100%\" result especially needs to be tested across load levels, workload mixes, and hardware topologies (e.g., A100-style 2-socket/2-NUMA servers) before claiming that cross-socket allocations are eliminated. At minimum, report run-to-run variation or worst-case outcomes over multiple seeds and configurations.","section":"Section 5, Simulation Configuration and Table 3"}],"minor_comments":[{"comment":"There are typos in the prose: \"Readers are refereed\" should be \"Readers are referred,\" and \"as detailed in the bellow\" should be \"as detailed below.\"","section":"Section 2.2 and Section 3.4"},{"comment":"The authors acknowledge that the before/after production snapshots may not be strictly comparable; please label this figure as illustrative, since it is not a controlled comparison and should not be cited as quantitative evidence.","section":"Section 5, Figure 8"},{"comment":"The sentence claiming that the scheduling failure rate is \"reduced by 55%\" conflates a 55.5-percentage-point absolute increase in hit rate with a relative reduction or a performance improvement; please state the arithmetic precisely.","section":"Section 5, text after Table 4"},{"comment":"The title and abstract emphasize LLM workloads, but no LLM inference engine is exercised in the evaluation; clarify that the results currently concern topology-based victim selection in a simulated Kubernetes cluster, with LLM relevance argued from hardware topology rather than measured inference performance.","section":"Abstract and Section 5"}],"recommendation":"major_revision","confidential_remarks":"The gap between the headline claim and the measured evidence is substantial, but it is fixable within the manuscript's scope: the authors have the scheduler plugin and could run real inference workloads on a small testbed to obtain latency or throughput comparisons. The paper would also be stronger if the simulation compared against Gödel's native NUMA-aware preemption rather than only against standard preemption. The open-source artifacts are a positive signal, but the current evaluation does not yet support the abstract's 55% performance claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a real engineering paper with a plausible system, but the headline 55% improvement is not supported by the evidence. The evaluation measures topology affinity hit rate, which is exactly the objective the scheduler optimizes, so the 100%-vs-45% result mostly shows self-consistency. That does not mean the work is worthless—just that the claim needs re-scoping.\n\nWhat is genuinely new and good: FlexTopo is a reasonable unified graph representation for server topology, and IMP is a practical incremental algorithm for minimal victim selection that avoids the exponential blow-up in most cases. The implementation on the Gödel scheduler is concrete, the code is available, and the overhead analysis is honest—IMP really does cut candidate sourcing cost, and the authors even show that Gödel's standard preemption is still faster, which is the kind of thing you don't always see. They also cite Gödel's existing NUMA-aware preemption and position their contribution as an extension, not a replacement. That is fair.\n\nWhere it gets soft: the simulation's only success metric is topology affinity hit rate, defined by whether freed resources land on the same NUMA node or socket. That is the same preference encoded in the scoring function of Eq. 1. So 100% versus 45% tells you the scheduler does what it was built to do, not that LLM workloads actually run better. The '55% improvement in Scheduled Performance' in the abstract and Section 5 is a direct conflation of hit-rate gain with performance gain. There is no end-to-end latency, throughput, or goodput measurement anywhere. Also, the high/medium/low values in the topological score are never specified, alpha is not reported, and the simulation uses a single workload configuration with no variance. These omissions make even the hit-rate comparison under-determined.\n\nAll that said, the central idea is sound and the system is described well enough to be reproduced. The paper belongs at a systems conference, not in the bin. It just needs either real performance measurements or a claim limited to topology affinity.\n\nWho should read it: people building Kubernetes or cluster schedulers for GPU/LLM workloads. It is a useful data point on how to do preemption with topology awareness, and a cautionary example of evaluation overreach. I would not cite it in my own next year of work, but I would bring it to a reading group to discuss what counts as evidence in scheduler papers.\n\nRecommendation: send it to peer review. A serious referee can push for a revised evaluation, but the system is concrete enough to deserve that engagement.","headline":"A sensible, well-built scheduling system that overclaims its 55% performance number, since the evaluation only measures the same topology score the scheduler already optimizes.","tokens_in":14414,"tokens_out":1499,"would_cite":false,"duration_ms":17254,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that preempting co-located LLM jobs with the victim's hardware topology in view — freeing GPUs near the same NUMA node or socket the high-priority service needs — raises topology-affinity hit rate from 44.5% to 100% in a…","keywords":["LLM serving","co-location","topology-aware scheduling","preemption","NUMA affinity","GPU cluster scheduling","Kubernetes scheduler"],"falsifier":"A controlled latency experiment on one of the paper's own server types (for example, an 8-GPU 4090 or A100 node): place the same LLM inference workload in three conditions — same-NUMA, same-socket different-NUMA, and cross-socket — and measure end-to-end throughput and tail latency. If the cross-socket placement performs within noise of the same-NUMA placement for the workloads the scheduler targets, the scoring function's ordering is miscalibrated and the claimed 55% scheduled-performance improvement would not appear; conversely, a direct before/after comparison of tail latency for a scaled-up service, with and without FlexTopo preemption, would settle whether the hit-rate gain translates into the claimed performance gain.","tokens_in":13481,"feed_emoji":"🎯","tokens_out":11412,"duration_ms":97068,"temperature":0.7,"pith_summary":"Clusters that co-locate LLM workloads fill their GPUs with a mix of latency-sensitive services and preemptible batch jobs, but when demand spikes, the scheduler evicts low-priority jobs without checking whether the hardware they release actually suits the high-priority service that needs to scale up. The paper argues this is the root cause of failed or degraded placements of online services, and that preemption should be topology-aware: the resources freed by victims should match the preemptor's NUMA and socket affinity. To that end it introduces FlexTopo, a graph-based representation of each server's real-time hardware topology, and a victim-selection policy that scores candidate preemptions by topology fitness while still preferring minimal, low-priority victim sets. The central empirical claim is a jump in topology-affinity hit rate from 44.5% to 100% across 5000 simulated preemptions, with the authors translating this into a 55% reduction in scheduling failures for guaranteed-QoS services and a 55% improvement in scheduled performance for best-effort ones. If true, co-location becomes much more attractive: clusters can stay fully packed and still reclaim topology-suitable resources on demand.","feed_headline":"Scheduler that frees NUMA-friendly GPUs hits 100% preemption affinity","feed_subtitle":"Choosing preempted victims by socket and NUMA proximity lifts LLM scheduled performance by 55 percent.","key_machinery":"The load-bearing objects are three. (1) FlexTopo, a graph-based, server-generic representation of hardware topology and live allocation: sockets, NUMA nodes, CPU core groups, and GPU devices are nodes; host, contain, localized, and nearby edges encode hierarchy and proximity; status and used-by attributes link components to running instances. (2) A scoring function for preemption candidates, $S(C) = \\alpha \\times \\frac{1}{\\text{sum of priority}(C)} + (1-\\alpha)\\, T(C_{\\mathrm{flextopo}})$, where $T$ is a piecewise affinity score: high for same-NUMA victim resources, medium for same-socket but different-NUMA, low for cross-socket; $\\alpha \\in [0,1]$ trades victim priority against topology fitness. (3) Incremental Minimal Preemption (IMP), a greedy search that tests victim subsets from size 1 upward and stops at the first feasible size, keeping the average-case cost close to polynomial while preserving minimal victim sets. An in-cluster FlexTopo agent keeps the representation fresh with periodic hardware checks and event-driven allocation updates, and a two-stage pipeline applies guaranteed filtering and best-effort sorting according to the preemptor's QoS.","core_discovery":"On the paper's own terms, the discovery is that preemption quality is a topology problem, not just a priority problem. A scheduler that knows, at victim-selection time, which sockets, NUMA nodes, CPU core groups, and GPU devices each candidate victim occupies can free resources that line up with the preemptor's requirements, and it can do so without an exponential search. The paper demonstrates this with FlexTopo — a unified graph whose nodes are sockets, NUMA nodes, CPU core groups, and GPUs, with edges for hosting, containment, and proximity — and a piecewise scoring of candidates that rates same-NUMA allocation highest, same-socket allocation medium, and cross-socket allocation lowest, balanced against victim priority by a weight $\\alpha$. The reported outcome is that in 100 simulation cycles with 50 preemptions each on a 100-node simulated 4090 cluster, the standard scheduler met topology requirements in 2225 of 5000 preemptions (44.5%), while FlexTopo-based preemption met them in all 5000 (100%); the paper reads the gap as a 55% reduction in scheduling failures for guaranteed QoS and a 55% improvement in scheduled performance for best-effort QoS. A greedy Incremental Minimal Preemption procedure keeps the victim search near-polynomial in the average case, cutting candidate-sourcing latency by 7.3% to 76.5% relative to exhaustive evaluation.","pith_inferences":["A direct calibration test is missing from the paper: the score assumes same-NUMA beats same-socket beats cross-socket, but the paper itself reports communication penalties of 1.2× within a socket and 3.2× across sockets on 4090 servers (2× on A100 servers); plugging those measured ratios into the topology score would be a natural, testable tightening.","The 55% scheduled-performance gain is inferred from the affinity-hit-rate gap rather than measured end-to-end; a direct before/after measurement of tail inference latency for the scaled-up service would confirm whether the hit-rate improvement actually buys the latency win the paper claims.","The FlexTopo representation and scoring could serve normal placement and defragmentation, not only preemption; the paper leaves those applications implicit.","The benefit is bounded by how topology-sensitive the workload actually is: for inference engines or quantization schemes that tolerate memory-bandwidth imbalance, the same scoring may over-rank or under-rank candidates, so the win would vary by engine."],"forward_implications":["Preemption can be made topology-reliable: in the paper's simulation, all 5000 preemptions met the preemptor's affinity requirements, versus 2225 (44.5%) for the standard scheduler.","The Kubernetes TopologyAffinityError failure mode is reduced by 55% for guaranteed-QoS preemptors, and scheduled performance improves by 55% for best-effort preemptors.","Saturation allocation becomes practical: clusters can run fully packed with mixed LLM workloads and still reclaim resources that satisfy latency-sensitive services when auto-scaling fires.","The overhead of topology-aware victim selection is manageable: IMP cuts candidate-sourcing latency by 7.3% to 76.5% versus exhaustive FlexTopo evaluation, with the smallest overhead for small GPU requests.","Because FlexTopo abstracts away server-specific layouts, the same policy transfers across hardware generations (for example, 4090 and A100 servers) without per-server adaptation."],"supporting_citations":[{"why":"Defines the standard Kubernetes priority-based preemption that the paper extends with topology awareness and treats as the conceptual baseline.","marker":"[16]"},{"why":"Gödel, the production scheduler whose standard preemption policy is the direct comparison baseline and into which the FlexTopo plugin is integrated.","marker":"[4]"},{"why":"Kubernetes topology manager; its admission-phase enforcement motivates the paper's claim that topology decisions must move into the scheduler rather than the kubelet.","marker":"[15]"},{"why":"Supplies the graph-based topology representation approach for multi-GPU scheduling that FlexTopo generalizes to mainstream servers such as the 4090 and A100.","marker":"[5]"},{"why":"KWOK, the lightweight simulated-Kubernetes framework used to run the 100-node, 5000-preemption evaluation.","marker":"[19]"},{"why":"Names the TopologyAffinityError failure mode whose reduction (55%) is the paper's headline quantitative claim for guaranteed QoS.","marker":"[20]"}],"fun_headline_variants":["Topology-aware preemption achieves 100% LLM affinity","NUMA-aware preemption boosts LLM performance by 55%","FlexTopo preemption: 100% topology match, 55% gain","Preempt by topology, not just priority: 55% LLM gain","Scheduler's topology-smart preemption hits 100% affinity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a hand-assigned scoring rule — same NUMA node is best, same socket is second, cross-socket is worst — captures how much LLM inference performance actually depends on hardware topology, so that optimizing this score genuinely improves real scheduled performance.","fun_headline_variants_meta":{"raw":{"variants":["Topology-aware preemption achieves 100% LLM affinity","NUMA-aware preemption boosts LLM performance by 55%","FlexTopo preemption: 100% topology match, 55% gain","Preempt by topology, not just priority: 55% LLM gain","Scheduler's topology-smart preemption hits 100% affinity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000531,"raw_usage":{"total_tokens":2601,"prompt_tokens":1034,"completion_tokens":1567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1472}},"tokens_in":650,"tokens_out":1567,"duration_ms":12820,"temperature":1.0,"reasoning_tokens":1472,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:23:35.814727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled latency experiment on one of the paper's own server types (for example, an 8-GPU 4090 or A100 node): place the same LLM inference workload in three conditions — same-NUMA, same-socket different-NUMA, and cross-socket — and measure end-to-end throughput and tail latency. If the cross-socket placement performs within noise of the same-NUMA placement for the workloads the scheduler targets, the scoring function's ordering is miscalibrated and the claimed 55% scheduled-performance improvement would not appear; conversely, a direct before/after comparison of tail latency for a scaled-up service, with and without FlexTopo preemption, would settle whether the hit-rate gain translates into the claimed performance gain.","supporting_citations":[{"cited_title":"Pod priority and preemption","cited_arxiv_id":null,"evidence_quote":"Defines the standard Kubernetes priority-based preemption that the paper extends with topology awareness and treats as the conceptual baseline."},{"cited_title":"Gödel: Unified large-scale resource management and scheduling at bytedance","cited_arxiv_id":null,"evidence_quote":"Gödel, the production scheduler whose standard preemption policy is the direct comparison baseline and into which the FlexTopo plugin is integrated."},{"cited_title":"Kubernetes topology manager moves to beta","cited_arxiv_id":null,"evidence_quote":"Kubernetes topology manager; its admission-phase enforcement motivates the paper's claim that topology decisions must move into the scheduler rather than the kubelet."},{"cited_title":"Topology-aware gpu scheduling for learning workloads in cloud environments","cited_arxiv_id":null,"evidence_quote":"Supplies the graph-based topology representation approach for multi-GPU scheduling that FlexTopo generalizes to mainstream servers such as the 4090 and A100."},{"cited_title":"Kubernetes without kubelet","cited_arxiv_id":null,"evidence_quote":"KWOK, the lightweight simulated-Kubernetes framework used to run the 100-node, 5000-preemption evaluation."},{"cited_title":"Control topology management policies on a node","cited_arxiv_id":null,"evidence_quote":"Names the TopologyAffinityError failure mode whose reduction (55%) is the paper's headline quantitative claim for guaranteed QoS."}],"review_version":1}