{"id":"91d423b4-af79-4def-9870-590c8d9ac889","arxiv_id":"2508.18556","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A GPU-sharing framework that dynamically resizes Nvidia MIG partitions and predicts memory growth achieves up to 6.2x throughput and 5.9x energy improvements on batches of scientific and ML jobs.","lead":"This paper presents a scheduling system that divides one Nvidia GPU into smaller hardware slices, resizes those slices on the fly, and places jobs on the tightest slice that fits their memory needs. The result is up to 6.2x higher job throughput and 5.9x lower energy use on tested workloads.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM gains hinge on the Section 3.2.3 linear-growth assumption and an unspecified CONVERGE check; neither is validated against workloads with nonlinear memory growth, and the reported 1.43x/1.11x LLM improvements rest on only three homogeneous single-job mixes.","rationale":"The Pith reader's weakest_assumption identified the same load-bearing concern: the Section 3.2.3 linear growth model plus the unspecified CONVERGE check. I agree, and I would keep the verdict at CONDITIONAL rather than moving to REJECT or ACCEPT. The paper's central claim is not yet refuted; it is under-supported at the exact point where it is most novel. The Rodinia and ML results (6.20x and 1.59x) depend on compile-time analysis and DNNMem estimation, respectively, which are established techniques and are evaluated on real workloads. The LLM results (1.43x/1.11x) depend on the time-series predictor, which is the paper's main claimed contribution, and that component is validated only by a 14.98% average error at 10% of iterations across four workloads, with no statement of the convergence rule, no error bars, and no adversarial nonlinear-growth cases. This is a correctness risk in the sense that the claimed general capability (handling dynamically growing LLM memory) may not transfer beyond the specific roughly-linear workloads tested. A conditional accept is appropriate because the concern is concrete and testable, not a demonstration of internal inconsistency. The paper's own text flags part of this: Section 3.2.3 states the linear model is 'often sufficient' and is chosen partly for stability with few data points, which is an honest admission that the model is a heuristic rather than a grounded characterization of LLM memory behavior. That admission, combined with the missing definition of CONVERGE, supports the conditional verdict. I do not agree with any stronger move: there is no evidence of fraud or a broken system, only a novelty-dependent component that needs sharper validation. The concrete test I propose would settle whether the concern lands: if a stepwise-growth workload triggers late or missed resizes, the LLM claims in their current generality should be weakened; if the predictor still catches the peak early enough, the concern is retired.","tokens_in":16847,"tokens_out":2723,"duration_ms":22067,"concrete_test":"Construct a controlled experiment on the A100 with an LLM whose memory growth is intentionally nonlinear: for example, a decoder-only model with a KV-cache that grows in block-sized chunks rather than continuously, or a workload that doubles its sequence length at a known iteration. Run MIGM with the Section 3.2.3 predictor and record, for each workload, the iteration at which CONVERGE returns, the predicted peak, and whether the subsequent resize happens before the actual OOM. Then compare against a variant in which growth is approximately linear (e.g., the reported Qwen2 scenario). If, for the stepwise-growth workload, CONVERGE either never fires before the true peak or fires with a predicted peak below the true peak by more than the 99% CI margin, the early-restart mechanism fails exactly in the regime the paper claims to handle.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest_claim is that MIGM achieves up to 1.43x throughput and 1.11x energy savings for LLM workloads, credited to the time-series memory predictor in Section 3.2.3. The predictor assumes requested memory grows linearly (m_t = a*t + b) with normal residuals, then predicts peak memory at a future iteration as a*t + b + z*sigma. This is fragile for exactly the workloads the paper targets: LLM inference with growing context windows often exhibits stepwise or superlinear memory growth when new layers, KV-cache blocks, or allocator segments are allocated, and training with sequence packing or gradient checkpointing can have similar discontinuities. A linear fit with a 99% CI can systematically under-predict the peak if growth is convex or stepwise, causing the scheduler to restart too late or not at all; conversely, over-prediction triggers premature restarts and wastes energy, which directly erodes the claimed energy savings. Making the concern concrete: the paper reports Qwen2 predictions of 11.41GB peak versus 12.23GB actual (Section 5.2.2), a 6.7% under-prediction that still lands inside the 10GB partition, but this example does not demonstrate robustness at the partition boundary. The most serious gap is that the convergence criterion in Algorithm 1 is never specified: CONVERGE(mem_pred) decides when the early prediction is stable, but the paper gives no definition of it, no sensitivity analysis, and no discussion of how many iterations are required on each workload. If convergence requires the peak estimate to stop changing across, say, k consecutive iterations, then workloads with slow, nonlinear growth will not converge early enough to enable the early-restart optimization that the paper says drives LLM gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MIGM, a scheduler and partition manager for NVIDIA Multi-Instance GPU (MIG) on A100, aiming to maximize throughput and energy efficiency by dynamically right-sizing MIG partitions. Memory requirements are obtained from three sources: compiler analysis (CASE) for scientific workloads, the DNNMem estimator for static ML models, and a runtime time-series predictor for dynamic ML/LLM workloads. The time-series predictor fits a linear trend to per-iteration requested memory and an inverse reuse ratio, adds a 99% confidence interval based on normal residuals, and triggers an early restart when the predicted peak exceeds the current partition. The scheduler offers a reordering policy (scheme A) and an in-order policy (scheme B), with a partition manager that uses a precomputed reachability metric to choose placements preserving future configurations. Evaluation on an A100 40GB reports up to 6.20x throughput and 5.93x energy savings on Rodinia mixes, 1.59x/1.12x for ML training workloads, and 1.43x/1.11x for LLM workloads, against a sequential non-partitioned baseline.","tokens_in":17135,"tokens_out":6561,"duration_ms":56311,"significance":"If the results hold, MIGM addresses a practical problem: modern GPUs provide MIG features but dynamic reconfiguration is hard, and accurate memory estimation enables tighter packing. The paper integrates existing techniques (compiler analysis, DNNMem) with a new component (time-series peak prediction) and a principled partition-placement policy based on future-configuration reachability. The evaluation is broad, covering scientific, CNN/transformer training, and LLM inference, and the reported internal consistency (e.g., throughput and energy trends moving together) is a plus. The main caveat is that the central new mechanism, the time-series predictor, is evaluated on a small set of workloads under a linear-growth assumption, and the convergence criterion is not specified. With additional validation, this would be a useful contribution to the MIG scheduling literature.","major_comments":[{"comment":"The convergence check CONVERGE(mem_pred) that gates the early prediction is never defined. Since the early-restart mechanism (Section 5.2.2) is the credited source of LLM gains, the paper must specify what makes a prediction stable (e.g., a threshold on the change in predicted peak over consecutive iterations, a minimum number of fitted points, or a criterion on the coefficient of determination), and should report the number of iterations required for convergence on each workload. Without this, the reader cannot judge whether the predictor risks premature restarts (wasting energy) or late restarts (wasting iterations), which directly affects the claimed 1.43x/1.11x LLM improvements.","section":"Section 3.2.3 / Algorithm 1"},{"comment":"The model m_t = a*t + b with normally distributed residuals is assumed without validation for workloads with nonlinear memory growth. LLM inference with growing context windows can exhibit stepwise or superlinear allocation (e.g., KV-cache blocks, allocator segment growth), and the reported Qwen2 example (11.41GB predicted vs 12.23GB actual, Section 5.2.2) shows a 6.7% under-prediction that is not tested at a partition boundary. The paper needs to evaluate the predictor on workloads with nonlinear or stepwise growth and report whether the early restart still fires before the actual OOM point. This is load-bearing because the LLM results rest on this predictor.","section":"Section 3.2.3, Eq. (1)"},{"comment":"The LLM experiments are listed as 'homogeneous mixes' but each has batch size 1, meaning the throughput/energy comparison is for a single job with early restart versus a baseline. The paper should clarify the baseline behavior for these jobs (does the baseline run to OOM and then restart, or is it a clean run to completion on a large partition?), and should present per-job turnaround time and energy per job rather than 'throughput' of a one-job batch. As written, the 1.43x throughput claim for LLMs is not a batch-concurrency throughput claim and may overstate the result.","section":"Section 5, Table 2"},{"comment":"All results are from single runs with no error bars or repeated-run statistics, and energy is derived from nvidia-smi polling at 0.1s. For short jobs such as the myocyte kernel in Table 3 (GPU kernel runtime ~2.6ms), 0.1s polling cannot attribute energy accurately, so the energy savings numbers (up to 5.93x) are not robust. The paper should report variance across at least several runs and use a finer-grained energy measurement (e.g., NVML counters or instrumented CUDA events) for short jobs.","section":"Section 5.1 / 5.2"},{"comment":"The baseline is a non-partitioned GPU executing the batch sequentially. This is a weak comparator: the authors' own preliminary experiment (Section 2.2) shows that tight partitions alone give 20.6% throughput improvement over the next-largest partition, and a static MIG configuration (e.g., seven 5GB slices) would already capture part of the reported gains for homogeneous small-job mixes. The paper should compare against at least one static MIG configuration to isolate the benefit of dynamic reconfiguration, partition fusion/fission, and prediction. This is central to the contribution claim.","section":"Section 5 baseline"},{"comment":"The paper asserts that 'PyTorch reserved memory' does not directly cause OOM, so that only allocated memory plus CUDA context must be predicted. However, from the CUDA driver's perspective, memory that PyTorch has reserved and cached is still allocated to the process and counts against the MIG slice's memory limit; a subsequent CUDA malloc can fail when the total reserved size already consumes the partition. Since the predictor's target is defined by this exclusion, the paper should empirically validate the claim (e.g., by showing a workload whose reserved memory exceeds the partition but whose active tensors do not, and confirming no OOM occurs). If the claim is wrong, the prediction target undercounts real memory pressure and early restarts will be systematically late.","section":"Section 3.2.1"}],"minor_comments":[{"comment":"The related work section refers to 'MIGPRO' where the framework is elsewhere called 'MIGM' (e.g., 'MIGPRO focuses first on the hard problem of memory estimation... MIGM's scheduler uses this information'). This naming inconsistency should be fixed.","section":"Section 6"},{"comment":"There are several typos: 'iste ad of' should be 'instead of', 'Similarily' should be 'Similarly', and 'we can predicts' should be 'we can predict'. A copyedit pass is needed.","section":"Section 5.2.2"},{"comment":"The paper states the A100 is 'the state of the art MIG used in industry'; given the paper's own abstract mentions Hopper series (H100, H200), this phrasing should be softened to reflect that A100 is a representative MIG-capable GPU.","section":"Section 4.1"},{"comment":"The text says 'A lower reuse ratio indicates more reuse', which is confusing because a reuse ratio, as the name implies, should be higher when more reuse occurs. Please define reuse_ratio precisely (e.g., fraction of a new request served from the cached pool) and then align the inverse-ratio formulation with that definition.","section":"Section 3.2.3"},{"comment":"The ML mix table lists 'Heterogeneous' for Ml1-Ml3 but the column header says 'Type'; the caption for Figure 4 does not state the normalization direction for energy plots (whether higher is better). Please add a note to the figure captions and table headers.","section":"Table 2 / Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for a systems venue. The main missing pieces are the specification and validation of the convergence criterion and the linear-growth assumption, plus a stronger baseline for the dynamic-reconfiguration claims. These are fixable with additional experiments and analysis, so I do not recommend rejection on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real system with real experiments, and the Rodinia results are convincing. The weak spots are the linear memory predictor and a baseline choice that flatters the LLM numbers. I'd send it to review, but it needs revision.\n\nThe new piece is the integration: compiler analysis from CASE, DNNMem-style model-size estimation, a linear time-series predictor for dynamic workloads, and an FSM-based partition manager that picks the MIG layout with maximum future reachability. The partition manager is clearly described and is a nice contribution in itself. The Rodinia evaluation is credible: up to 6.2x throughput for homogeneous small jobs is exactly what you'd expect from 7 slices, and the heterogeneous mix results look reasonable. The paper is honest that PCIe contention limits some gains.\n\nSoft spots, in order of importance.\n\nFirst, the baseline is sequential execution on a non-partitioned GPU. That's a weak baseline. A static MIG configuration with a good packing policy would be the natural comparison, and without it you can't tell how much of the gain comes from dynamic reconfiguration versus just using MIG at all. The 20-30% gains on heterogeneous Rodinia mixes are the more interesting numbers, and those are against the same weak baseline.\n\nSecond, the time series predictor in Section 3.2.3 assumes memory grows linearly with normal residuals, and Algorithm 1 uses a CONVERGE function that is never defined. No sensitivity analysis, no stress test against convex or stepwise growth. LLM inference with growing KV caches is exactly the case where you'd expect nonlinear jumps. The 15% average prediction error at 10% of iterations is reported, but not how that error propagates to scheduling decisions.\n\nThird, the LLM evaluation is thin. Qwen2 and Llama-3 mixes have batch size 1. With a single job, throughput is just the inverse of completion time, and the paper never explains how starting on a 10GB slice and restarting after prediction beats running the same job on the full 40GB GPU. That needs a direct comparison, not just the OOM-avoidance anecdote.\n\nFourth, no error bars or repeated runs, and energy is polled from nvidia-smi every 0.1s while some Rodinia kernels run for microseconds. Energy numbers for short jobs are likely noisy.\n\nAlso, there's a naming inconsistency: MIGPRO in Section 6, MIGM everywhere else. Minor.\n\nBottom line: the integrated system is a genuine extension of prior work, the Rodinia results are solid, and the FSM partition manager is worth building on. The LLM claims and the predictor are under-supported as written. A serious referee would ask for a stronger baseline, the CONVERGE specification, and a better explanation of the single-job LLM results. I'd take it.","headline":"A real MIG scheduler with a strong Rodinia evaluation, but the linear memory predictor and single-job LLM baseline leave the headline gains under-supported.","tokens_in":17746,"tokens_out":3958,"would_cite":false,"duration_ms":35932,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single A100 GPU's MIG partitions can be dynamically right-sized with memory prediction, yielding up to 6.20x throughput and 5.93x energy savings compared with sequential execution.","keywords":["multi-instance GPU","GPU partition scheduling","memory prediction","time series","out-of-memory recovery","LLM inference","energy efficiency","GPU utilization"],"falsifier":"Run a model whose memory footprint is flat for many iterations and then jumps abruptly, for example from 4GB to 9GB at iteration 50, while the predictor is active. If the linear 99% interval computed at iteration 5 projects a peak below the partition size and the job then OOMs at iteration 50, the predictor's central assumption fails for that workload.","tokens_in":16608,"feed_emoji":"⚡","tokens_out":9609,"duration_ms":91140,"temperature":0.7,"pith_summary":"This paper tries to establish that the concurrency offered by Multi-Instance GPU (MIG) hardware can be systematically exploited by treating partition sizing as a scheduling problem. It combines three memory-estimation routes: compiler analysis for scientific programs, offline model-size estimation for deep networks, and a runtime time-series forecast for workloads whose memory grows during execution, such as LLMs with expanding context windows, and it uses those estimates to place each job on the smallest legal partition. A partition manager models all valid MIG layouts as a finite state machine and chooses placements that preserve the most future configurations, with fusion and fission used to create tight partitions on demand. The measured payoff on an A100 40GB, against a baseline that runs the same batch sequentially on the unpartitioned GPU, is up to 6.20x throughput and 5.93x energy savings for general workloads, 1.59x and 1.12x for ML workloads, and 1.43x and 1.11x for LLM workloads.","feed_headline":"GPU partition scheduler reports 6.2x throughput on one A100","feed_subtitle":"Time-series memory forecasts let a single GPU run many jobs on tight partitions and restart before OOM.","key_machinery":"The load-bearing object is the time-series peak-memory predictor: per-iteration instrumentation of the framework's caching allocator, a linear regression $\\hat{m}_t = a t + b$ for requested memory, a normal residual model giving a 99% confidence interval, and a separately fitted inverse-reuse-ratio model that converts requested memory into physical memory demand. Its job is to make the early-restart optimization possible. The second mechanism is the partition manager, a finite-state machine over all valid MIG partition states whose offline-precomputed reachability score $\\mathrm{fcr}(s)$ counts how many fully configured layouts remain reachable; online allocation picks the placement with the largest score, and fusion/fission operations create tight slices on demand. These two mechanisms together let the scheduler pack jobs tightly without stranding the GPU in a fragmented configuration.","core_discovery":"On its own terms, the paper's discovery is that dynamic memory growth, not static memory size, is what prevents MIG partitions from being packed tightly, and that this growth can be forecast early enough to act on it. The system instruments the ML framework's caching allocator to record requested tensor memory and a reuse ratio at every iteration, fits a linear trend $\\hat{m}_t = a t + b$ to each, adds a 99% confidence interval from the residuals to catch peaks, and projects the final peak before the job reaches it. When the projection exceeds the current slice, the scheduler restarts the job on a larger partition instead of waiting for an out-of-memory crash; in the paper's LLM example the predictor flags the overflow at iteration 6 versus the actual crash at iteration 94. Combined with compiler-based memory estimates for scientific jobs and offline model-size estimates for deep networks, the same scheduler handles all three workload classes, and the evaluation reports the throughput and energy gains listed in the abstract.","pith_inferences":["Editorial inference: the linear-growth assumption is the part most likely to give way; a variant that detects stepwise or piecewise growth, for example a new layer or a context-window expansion, would preserve the early-restart benefit for workloads the current 99% interval can miss.","Editorial inference: the future-configuration reachability rule is a greedy choice; for a known batch one could precompute an optimal global layout sequence offline and compare it against the greedy online choices, which would show how much flexibility preservation costs.","Editorial inference: the reported multipliers compare against sequential execution on an unpartitioned GPU, so they are upper bounds on the concurrency benefit; against a static MIG layout with good packing, the gains would come mainly from dynamic resizing and early restart, not from packing itself."],"forward_implications":["A single GPU can serve mixed batches of scientific, DNN-training, and LLM jobs concurrently, with job throughput and energy per job improving over running the same batch one job at a time.","LLM inference with growing context windows becomes schedulable on small MIG slices: the predictor can flag an imminent out-of-memory condition during the first few percent of iterations, so the job is moved before the wasted work accumulates.","Because the reachability scores are precomputed offline, the online partition manager stays cheap regardless of how many layouts the GPU supports, making the scheme portable to other MIG-capable GPUs.","Scheduling policy matters: sorting jobs by memory size and scheduling in uniform slices beats FIFO with on-demand reconfiguration for heterogeneous batches, at the cost of fairness within a batch."],"supporting_citations":[{"why":"Supplies the Rodinia benchmark suite whose mixes produce the general-workload throughput and energy results.","marker":"[2]"},{"why":"Supplies the compiler-assisted memory and warp analysis used to right-size partitions for scientific workloads.","marker":"[4]"},{"why":"Supplies the offline DNN memory-size estimates used as the starting slice size for deep network workloads.","marker":"[7]"},{"why":"FLAN-T5 is one of the homogeneous LLM workloads used to test the time-series predictor.","marker":"[5]"},{"why":"Llama 3 is one of the homogeneous LLM workloads used to test the time-series predictor.","marker":"[6]"},{"why":"Defines the valid MIG partition states and transition rules that the partition manager's finite-state machine encodes.","marker":"[14]"},{"why":"Documents PCIe bandwidth sharing across MIG instances, which explains why data-transfer-heavy workloads fall short of the theoretical 7x throughput ceiling.","marker":"[24]"},{"why":"Qwen2 is the LLM example in which the predictor flags an overflow at iteration 6 versus the actual crash at iteration 94.","marker":"[29]"}],"fun_headline_variants":["Forecasting GPU memory peaks hikes throughput 6x","Predict OOM to pack GPUs tighter, get 6x speed","Scheduler restarts jobs before OOM, 6x throughput","Peak memory projection packs A100, 6x faster","GPU scheduler sees memory spikes, avoids crashes, 6x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The early-restart optimization rests on the assumption that a dynamic ML job's memory grows at a roughly steady, predictable rate, so a straight-line forecast with a 99% safety margin can see the peak before the job actually runs out of memory.","fun_headline_variants_meta":{"raw":{"variants":["Forecasting GPU memory peaks hikes throughput 6x","Predict OOM to pack GPUs tighter, get 6x speed","Scheduler restarts jobs before OOM, 6x throughput","Peak memory projection packs A100, 6x faster","GPU scheduler sees memory spikes, avoids crashes, 6x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1516,"prompt_tokens":936,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":552,"tokens_out":580,"duration_ms":6085,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:55:51.327835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a model whose memory footprint is flat for many iterations and then jumps abruptly, for example from 4GB to 9GB at iteration 50, while the predictor is active. If the linear 99% interval computed at iteration 5 projects a peak below the partition size and the job then OOMs at iteration 50, the predictor's central assumption fails for that workload.","supporting_citations":[{"cited_title":"Sheaffer, Sang-Ha Lee, and Kevin Skadron","cited_arxiv_id":null,"evidence_quote":"Supplies the Rodinia benchmark suite whose mixes produce the general-workload throughput and energy results."},{"cited_title":"CASE: a compiler-assisted scheduling framework for multi-gpu systems","cited_arxiv_id":null,"evidence_quote":"Supplies the compiler-assisted memory and warp analysis used to right-size partitions for scientific workloads."},{"cited_title":"Estimating GPU memory consumption of deep learning models","cited_arxiv_id":null,"evidence_quote":"Supplies the offline DNN memory-size estimates used as the starting slice size for deep network workloads."},{"cited_title":"Zhao, Yanping Huang, Andrew M","cited_arxiv_id":null,"evidence_quote":"FLAN-T5 is one of the homogeneous LLM workloads used to test the time-series predictor."},{"cited_title":"Mig user guide","cited_arxiv_id":null,"evidence_quote":"Defines the valid MIG partition states and transition rules that the partition manager's finite-state machine encodes."},{"cited_title":"Pcie bandwidth-aware scheduling for multi-instance gpus","cited_arxiv_id":null,"evidence_quote":"Documents PCIe bandwidth sharing across MIG instances, which explains why data-transfer-heavy workloads fall short of the theoretical 7x throughput ceiling."}],"review_version":2}