{"id":"2d2b3ca4-1bef-429b-9a6a-51d84ef49b9c","arxiv_id":"2412.14479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frenzy uses a memory formula and best-fit bin packing to automatically allocate heterogeneous GPUs to LLM training jobs, claiming 12-18% faster average job completion time versus Sia.","lead":"Frenzy is a scheduling system that automatically decides how many and which kinds of GPUs an LLM training job needs, using a memory-prediction formula. It aims to let users submit training jobs without manually configuring hardware, and claims 12-18% faster average job completion than existing heterogeneous schedulers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never specifies the priority function that orders MARP's resource plans, so HAS's 'optimal plan' and the claimed 12-18% JCT improvement rest on an undefined decision rule.","rationale":"The reader's weakest assumption concerns the accuracy of the memory formulas, which is a legitimate empirical risk, but the paper at least reports measured accuracy on two models and cites established formulas. A more fundamental problem is that the algorithm that converts feasible memory plans into a selected resource allocation is never defined. Section IV-A only establishes which (d,t) plans fit in memory; the text says MARP priority-ranks plans and HAS takes the first satisfiable one, but no priority rule is given. This is not a parameter uncertainty or an evaluation gap: it is an omitted algorithmic component. Without the ranking, the system is underspecified and the reported JCT improvements cannot be independently reproduced or attributed to the described scheduler. This does not change the overall verdict from CONDITIONAL because the issue is addressable by adding the missing specification and rerunning the evaluation, but it should be a primary condition for acceptance rather than a secondary concern.","tokens_in":8568,"tokens_out":7407,"duration_ms":71431,"concrete_test":"Ask the authors to release the exact plan-ranking function or code used for Figures 4 and 5. Independently implement HAS with three plausible rankings, minimum GPU count, maximum memory utilization, and estimated throughput per GPU type, then rerun the Helio/Philly simulations. If the average JCT improvement over Sia varies by more than a few percent across rankings or disappears for any ranking, the central claim is not robust and the missing priority is material.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Frenzy's serverless claim requires automatically choosing GPU types and counts, but the choice rule is absent. Section IV-A gives only a feasibility inequality (20W/t + activation term < capacity); it says MARP 'adopts a priority ranking' for plans from different (d,t) settings, yet no objective, formula, or algorithm for that ranking is presented anywhere. HAS (Algorithm 1) then simply returns the first satisfiable plan in this unspecified order. Consequently, the paper's central quantitative claims, apart from memory prediction accuracy, depend on a hidden ordering. If the order was chosen to favor plans that happen to improve JCT, the 12-18% improvement over Sia is not attributable to the described design. The JCT reduction might come from the omitted ranking rather than from HAS, and any independent implementation would have to guess how plans are prioritized.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Frenzy, a serverless-style scheduling system for LLM training on heterogeneous GPU clusters. Frenzy consists of MARP, which predicts peak GPU memory usage from closed-form formulas for model states and activations under data and tensor parallelism and generates candidate resource-allocation plans, and HAS, which selects the first satisfiable plan and allocates GPUs using a best-fit/greedy node-placement strategy. The evaluation uses a small real cluster and the PAI simulator with Philly and Helio traces, comparing against Sia and opportunistic scheduling. The paper claims 92-98% memory prediction accuracy, a 10x reduction in scheduling overhead, and 12-18% average job completion time (JCT) reduction.","tokens_in":8673,"tokens_out":3882,"duration_ms":35248,"significance":"If the claims hold, the paper addresses a real usability problem: automating the choice of GPU types and counts for LLM training on heterogeneous clusters. The memory formulas are taken from prior published work and are validated against measured peak memory rather than fitted, which is a strength. The use of real workload traces (Philly, Helio) and comparison to Sia are also appropriate. However, the central JCT and scheduling claims currently rest on an undefined plan-ranking rule, an uncontrolled comparison that does not account for total GPU resources consumed, and simulation results without released code or error bars. These issues must be resolved before the contribution can be assessed reliably.","major_comments":[{"comment":"The priority ranking that orders MARP's resource plans is never specified. Section IV-A states that MARP 'adopts a priority ranking' and Figure 1 refers to plans 'with priorities', but no objective function, formula, or algorithm for this ranking is given anywhere in the paper. HAS (Algorithm 1) simply returns the first plan in this undefined order, so the 'optimal' plan that drives the JCT results depends entirely on a hidden decision rule. The authors must define the ranking (e.g., minimize estimated training time, maximize predicted throughput, or minimize GPU-hours) and evaluate its sensitivity, because otherwise the 12-18% JCT improvement cannot be attributed to the described design.","section":"Section IV-A and Algorithm 1"},{"comment":"The JCT comparison to Sia and opportunistic scheduling does not control for the total amount of GPU resources consumed. Since MARP determines both the type and the number of GPUs, Frenzy may achieve lower JCT simply by allocating more or larger GPUs than the Sia baseline, which is given user-specified GPU counts. The paper reports no GPU-hours, utilization, or resource-normalized throughput. Without such a control, the claimed 12-18% JCT reduction is not evidence of scheduling efficiency. The authors should report total resource consumption, per-job GPU-hours, or an equivalent-resource comparison, and should add error bars or confidence intervals, since no variance or significance information is reported.","section":"Section V-B, Figures 4 and 5"},{"comment":"The memory predictor is validated only on GPT2-350M and GPT2-7B with a limited set of parallelism configurations, yet MARP's output drives all downstream claims. The formula in Section IV-A deliberately considers only data and tensor parallelism and omits pipeline parallelism, activation recomputation, communication buffers, memory fragmentation, and framework-specific overhead. The paper should either demonstrate that the formula generalizes to larger models and other parallelism configurations (e.g., GPT-3-scale or models with pipeline parallelism), or state explicitly the configurations for which the resource plans are guaranteed to be safe. As written, an OOM or under-allocation outside the tested range would invalidate the serverless guarantee.","section":"Section IV-A and Section V-C"},{"comment":"The HAS pseudocode appears buggy. Line 14 computes fitSz as the minimum GPU size satisfying reqSz, but line 15 filters nodes with 'gpu.size > fitSz', excluding nodes whose size is exactly fitSz, which contradicts the best-fit text. If no node has size strictly greater than fitSz but some have size equal to fitSz, N_Lst can be empty and the while loop cannot make progress. The pseudocode also lacks a termination condition for the case where no feasible node exists. The authors should correct the inequality, add an explicit tie-breaking rule, and prove or argue that the allocation loop always terminates on a satisfiable plan.","section":"Algorithm 1, lines 14-34"}],"minor_comments":[{"comment":"In the example, J ob(2, 32) is described as requiring 'at least 36GB of memory'; the numbers 32 and 36 are inconsistent and should be aligned.","section":"Section IV-B"},{"comment":"The scheduling-overhead figure lacks labeled axes and numerical units, and the 10x claim is not supported by any reported numeric overhead values in the text.","section":"Figure 5(a)"},{"comment":"The 'memory prediction accuracy' metric is never defined. Please specify whether it is (predicted - measured)/measured, a ratio, or a percentage error, and report per-configuration values.","section":"Section V-C"},{"comment":"The workload name is written inconsistently as both 'Helio' and 'Helios'; the same name should be used throughout.","section":"Section V-B"},{"comment":"The baseline called 'Opportunistic Scheduling' is cited to Lyra, but the description in Section V-A sounds like a generic FCFS policy; please clarify the relationship or cite the original opportunistic scheduling work.","section":"Reference [23]"},{"comment":"There are typographical errors in the pseudocode identifiers (e.g., 'J obRespourceP lans', 'singleN odeM eetRequirement'), which should be corrected for readability.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the missing priority function is accurate and central: no independent implementation could reproduce the reported JCT numbers without guessing how MARP orders plans. The resource-consumption control is also a real gap. If the authors can specify the ranking and rerun the evaluation with resource-normalized metrics and error bars, the paper could become a solid systems contribution. The other issues in the reader's report (circularity, fitted constants) do not appear to be problems; the memory formulas are indeed from prior work and are validated against measurements. The paper is within the scope of the journal/series for distributed computing systems, but the evaluation section needs substantial strengthening."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Frenzy does something real: it lets a user submit an LLM training job without specifying GPU types or counts, predicts peak memory with standard closed-form estimates, and schedules with a fast best-fit heuristic. The serverless framing is not new, but the packaging for heterogeneous clusters is a legitimate contribution. Credit where due: MARP uses published formulas (20W/t and the activation expression) rather than fitting to the experiments, and the 92–98% accuracy on two real models is evidence those formulas work for those configurations. The scheduling overhead reduction is believable because bin packing is cheap compared to Sia's ILP.\n\nThe soft spots are real and one is load-bearing. The priority ranking across MARP's resource plans is never defined. Section IV-A says MARP 'adopts a priority ranking' and Fig. 1 says plans come out 'with priorities,' but no objective, formula, or algorithm is given. HAS then returns the first satisfiable plan. The text says the plans at the forefront indicate higher training efficiency, but the mapping is absent. That ordering determines the resource choice, and the resource choice determines the JCT. So the claimed 12–18% improvement over Sia could be an artifact of a hidden ranking rather than a property of HAS. That unfalsifiability is not minor. The evaluation also has no error bars, no released code or traces, and no control for total GPU resources consumed, so over-provisioning could explain the gains. Memory prediction is validated on only two GPT-2 variants, and the paper honestly limits the model to data and tensor parallelism, leaving out pipeline parallelism, fragmentation, and framework overhead. These are addressable, but they need real work.\n\nWho is this for: systems researchers working on GPU cluster scheduling and serverless ML platforms. It deserves a serious referee, not a desk reject: the design is plausible, the Sia comparison is the right target, and the gaps are fixable. My recommendation: send to review, and require the authors to specify the ranking, release code and traces, and rerun with error bars and a resource-equivalent baseline.","headline":"Frenzy's idea is sensible, but the undefined resource-plan priority ordering makes the headline JCT gains unfalsifiable until it is specified.","tokens_in":9209,"tokens_out":3186,"would_cite":false,"duration_ms":25590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frenzy predicts GPU memory from model hyperparameters and schedules jobs on heterogeneous clusters, so users can submit LLM training without specifying GPU types or counts.","keywords":["serverless computing","heterogeneous GPU clusters","LLM training","GPU memory prediction","resource scheduling","data parallelism","tensor parallelism","job completion time"],"falsifier":"Train a large decoder-only transformer with pipeline parallelism enabled on a heterogeneous cluster, measure the actual peak GPU memory, and compare it to the formula's prediction. If the measured peak exceeds the prediction by more than the claimed error margin, or if training aborts with out-of-memory, the central memory-prediction claim is falsified.","tokens_in":8355,"feed_emoji":"⚡","tokens_out":9192,"duration_ms":64663,"temperature":0.7,"pith_summary":"Frenzy claims that LLM training on a heterogeneous GPU cluster can be made serverless: the user submits a model and training configuration, and the system decides how many GPUs of which type to allocate. The feasibility rests on a closed-form estimate of peak GPU memory, which lets the system enumerate feasible combinations of data and tensor parallelism. The paper reports that this memory predictor is 92-98% accurate, that the scheduler overhead is about ten times lower than an existing scheduling method, and that average job completion time drops by 12-18% on real cluster workload traces.","feed_headline":"Serverless LLM training picks its own GPUs","feed_subtitle":"Memory prediction plus heterogeneity-aware scheduling cut job completion time 12-18% and scheduling overhead 10x.","key_machinery":"The memory-aware resource predictor (MARP) is the load-bearing component. It uses the weight formula $W = Vh + l(12h^2+13h)$, static memory $20W/t$, and activation memory $sBhl(10/d + 24/(dt) + 5as/(dht))$ to estimate peak GPU memory as a function of model shape, batch size, data parallelism $d$, and tensor parallelism $t$. The constraint $20W/t + sBhl(10/d + 24/(dt) + 5as/(dht)) < C$, with $C$ the GPU memory capacity, turns memory feasibility into a search over $(d,t)$ pairs. The heterogeneity-aware scheduler (HAS) then chooses among the ranked plans using best-fit node packing, preferring the fewest nodes with the tightest-fit GPUs.","core_discovery":"Frenzy's central claim is that peak GPU memory during LLM training can be decomposed into a static part (model weights plus Adam optimizer states in mixed precision, $20W/t$ under tensor parallelism $t$) and a dynamic part (activations, $sBhl(10/d + 24/(dt) + 5as/(dht))$ with data parallelism $d$), where the weight count is $W = Vh + l(12h^2 + 13h)$. Requiring the sum to be below a GPU's memory capacity yields feasible combinations of data and tensor parallelism, hence the number and type of GPUs needed. The paper then claims that scheduling these plans with a best-fit node-packing strategy that prefers single-node placements reduces average job completion time compared with existing heterogeneity-aware schedulers, because the resource plans match actual memory use and the scheduler keeps communication within nodes.","pith_inferences":["The same memory-estimation inequality could be extended to include pipeline parallelism, which the paper deliberately excludes; adding a term for layer-partitioned parameters would let the predictor handle the very largest models.","The closed-form weight formula assumes a standard decoder-only transformer; testing on mixture-of-experts or encoder-decoder architectures would show whether the 92-98% accuracy claim generalizes beyond the tested transformer family.","Resource plans that map model configuration directly to GPU type and count could also drive cost-aware or energy-aware placement, since each plan implies a specific hardware mix and runtime.","The tenfold scheduling-overhead reduction is demonstrated against one specific prior method; comparing against simpler heuristics like first-fit would isolate how much of the gain comes from the prioritized plan list versus the bin-packing itself."],"forward_implications":["A developer can submit a model without specifying GPU types or counts; Frenzy derives the hardware plan from the model's shape, batch size, and the cluster's available memory.","The prioritized list of resource plans lets the scheduler pick the most training-efficient plan that the current cluster can satisfy, avoiding out-of-memory failures and over-provisioning.","Because HAS uses best-fit node packing instead of solving an integer linear program, scheduling overhead stays low even as the number of concurrent tasks grows.","On the evaluated workloads, average job completion time falls by 12-18% compared to existing heterogeneity-aware scheduling, and per-job throughput rises by roughly 27-29%."],"supporting_citations":[{"why":"supplies the activation memory formula used for the dynamic part of the peak memory estimate.","marker":"[19]"},{"why":"source of the 20W static-memory figure for mixed-precision training with the Adam optimizer.","marker":"[24]"},{"why":"defines tensor parallelism, which MARP uses to divide static parameters across GPUs.","marker":"[15]"},{"why":"provides the data-parallel memory model and optimizer-state breakdown that underlies the static-memory constant.","marker":"[14]"},{"why":"the heterogeneity-aware scheduler baseline against which Frenzy measures scheduling overhead and job completion time.","marker":"[8]"},{"why":"provides a real-world multi-tenant cluster workload trace used in the evaluation.","marker":"[5]"},{"why":"provides a second real-world workload trace with longer-running tasks used in the evaluation.","marker":"[20]"},{"why":"provides the cluster simulator used for large-scale scheduling validation.","marker":"[25]"}],"fun_headline_variants":["LLM training auto-selects GPUs via memory prediction","Frenzy: Serverless LLM training that right-sizes GPU clusters","Memory-aware scheduler cuts LLM training time by 18%","Auto-pick GPU types for serverless LLM training","Frenzy predicts memory to pick GPUs, slashes job time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The memory formula assumes only data and tensor parallelism and omits pipeline parallelism, communication buffers, memory fragmentation, and framework overhead; if these are substantial for a given model, the predicted GPU counts and types will be wrong and the job-completion-time gains will not hold.","fun_headline_variants_meta":{"raw":{"variants":["LLM training auto-selects GPUs via memory prediction","Frenzy: Serverless LLM training that right-sizes GPU clusters","Memory-aware scheduler cuts LLM training time by 18%","Auto-pick GPU types for serverless LLM training","Frenzy predicts memory to pick GPUs, slashes job time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1335,"prompt_tokens":891,"completion_tokens":444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":507,"tokens_out":444,"duration_ms":3941,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:11:15.473352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a large decoder-only transformer with pipeline parallelism enabled on a heterogeneous cluster, measure the actual peak GPU memory, and compare it to the formula's prediction. If the measured peak exceeds the prediction by more than the claimed error margin, or if training aborts with out-of-memory, the central memory-prediction claim is falsified.","supporting_citations":[{"cited_title":"Reducing activation recomputation in large transformer models","cited_arxiv_id":null,"evidence_quote":"supplies the activation memory formula used for the dynamic part of the peak memory estimate."},{"cited_title":"Zero: Memory optimizations toward training trillion parameter models","cited_arxiv_id":null,"evidence_quote":"provides the data-parallel memory model and optimizer-state breakdown that underlies the static-memory constant."},{"cited_title":"Sia: Heterogeneity-aware, goodput- optimized ML-cluster scheduling","cited_arxiv_id":null,"evidence_quote":"the heterogeneity-aware scheduler baseline against which Frenzy measures scheduling overhead and job completion time."},{"cited_title":"Analysis of Large-ScaleMulti-TenantGPU clus- ters for DNN training workloads","cited_arxiv_id":null,"evidence_quote":"provides a real-world multi-tenant cluster workload trace used in the evaluation."},{"cited_title":"Characterization and prediction of deep learning workloads in large-scale gpu datacenters","cited_arxiv_id":null,"evidence_quote":"provides a second real-world workload trace with longer-running tasks used in the evaluation."},{"cited_title":"Alibaba clusterdata","cited_arxiv_id":null,"evidence_quote":"provides the cluster simulator used for large-scale scheduling validation."}],"review_version":1}