{"id":"7984221e-f8cf-4f51-89b9-10c05371d1a3","arxiv_id":"2506.02025","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ReAct-style LLM schedulers can balance multiple HPC scheduling objectives on 10-100 job workloads, though cloud API latency makes them unsuitable for real-time deployment.","lead":"A team tests whether chatbots that can reason step by step can schedule supercomputer jobs while balancing speed, fairness, and efficiency. On small synthetic workloads and one real cluster trace, the LLM schedulers match or beat simple rules and sometimes OR-Tools on several metrics, but they are far too slow for real-time use.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Wait-time metric is internally inconsistent: under §3.3 all jobs arrive at t=0 and wait equals start time, so FCFS cannot have zero average wait in contended scenarios such as Long Job Dominant, yet §3.5 drops wait time because it is 0.0; this invalidates the headline wait-time/fairness comparisons.","rationale":"The reader identified the unspecified OR-Tools objective as the weakest assumption, and that is a real problem: if OR-Tools was configured to maximize utilization only, its fairness degradation is an artifact of the chosen objective and not evidence that LLMs balance objectives better. However, the more load-bearing issue is the reported zero average wait time for all methods including FCFS. Under the paper's own definitions, FCFS must experience positive average wait time in scenarios with large 128-256 node jobs, so the claim in Section 3.5 is either a metric-computation error, a selective observation meant only for low-contention scenarios, or evidence that the described resource constraints were not actually enforced. Since the fairness metric is derived from wait times, the fairness comparisons share this defect. The public code and the Polaris real-trace experiment provide some independent support, and the overhead analysis is informative, so a full rejection would be too strong; the paper should be accepted only after the wait-time metric, the dynamic-arrival description, and the OR-Tools objective are clarified or corrected. This is consistent with the reader's CONDITIONAL verdict, so no verdict change is recommended.","tokens_in":14378,"tokens_out":8533,"duration_ms":79022,"concrete_test":"Inspect the released repository (ReasonedScheduler/llm_scheduler) and rerun the Long Job Dominant scenario with 60 jobs under the Section 3.3 configuration, logging each job's submission time s_j and start time x_j for FCFS. Compute average wait time as mean(x_j - s_j). If the result is nonzero, the paper's claim that FCFS wait time is 0.0 is false, and all wait-time and fairness normalizations in Figure 3 must be recomputed; if the result is truly zero, verify whether the resource demands, durations, or capacity settings are degenerate, since 128-node jobs cannot all start at time 0 on a 256-node cluster.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs 'effectively balance multiple objectives' and 'reduce wait times' rests on the wait-time and fairness comparisons in Figures 3, 4, and 7. Those comparisons are not interpretable as reported. Section 3.3 fixes all submission times at s_j=0, and Section 3.2 defines average wait time as w_j = x_j (the start time, since s_j=0). Under that definition, FCFS wait time is identically zero only if every job starts at time 0, which is impossible when jobs require 128-256 nodes (Long Job Dominant, High Parallelism, Adversarial). Nevertheless, Section 3.5 states that 'the average wait time for all methods, including FCFS, was 0.0' and omits wait time from normalization. If that note is meant only for low-contention scenarios such as Homogeneous Short and Resource Sparse, it does not justify dropping wait time for the whole figure; if it is meant globally, it contradicts the stated resource constraints. The dynamic-arrival description in Sections 3.1 and 4 deepens the inconsistency: if jobs arrive via Poisson processes, the formula w_j = x_j is wrong because submission times are not all zero. Either way, the claimed wait-time and fairness advantages of the LLM schedulers are not supported by the reported synthetic experiments, independent of the also-unaddressed question of what objective OR-Tools was configured to optimize.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based HPC job scheduler using a ReAct-style reasoning loop with a persistent scratchpad memory and a separate constraint-enforcement module. The authors evaluate OpenAI's O4-Mini and Anthropic's Claude 3.7 against FCFS, SJF, and Google OR-Tools on seven synthetic workload scenarios with 10 to 100 jobs, on a five-run robustness study, and on a 100-job segment of the Polaris workload trace. The central claims are that the LLM schedulers balance makespan, wait time, utilization, throughput, and fairness without domain-specific training, that they provide transparent natural-language reasoning traces, and that their main limitation is computational overhead.","tokens_in":14692,"tokens_out":4999,"duration_ms":45773,"significance":"If the central claims were fully supported, this would be a useful early empirical study of reasoning-capable LLMs for HPC scheduling. The paper has notable strengths: the authors release code, prompts, and configurations publicly, they provide representative reasoning traces that make the decision process transparent, they evaluate on a real workload trace, and they openly discuss the overhead barrier to real-time deployment. However, the quantitative support for the headline 'balanced multiobjective scheduling' claim is currently compromised by an internally inconsistent definition of wait time, an impossible zero-wait-time report, and an underspecified OR-Tools baseline. These issues affect Figures 3, 4, 7, and 8 and the abstract's claims, so the paper needs substantial revision before its empirical conclusions can be accepted.","major_comments":[{"comment":"Section 3.1 states that jobs arrive dynamically over time following Poisson processes with scenario-specific arrival rates, whereas Section 3.2 defines average wait time as w_j = x_j 'since all s_j = 0' and Section 3.3 states that all n jobs are submitted simultaneously at time s_j = 0. These are incompatible descriptions of the same experiments. If arrivals are dynamic, the wait-time formula is incorrect; if arrivals are simultaneous, the event-driven arrival process described in Section 3.1 was not used for the reported wait-time results. Please disambiguate the arrival model and recompute wait time and Jain's fairness indices under the correct definition.","section":"3.1, 3.2, 3.3"},{"comment":"The statement that 'the average wait time for all methods, including FCFS, was 0.0' is impossible under the resource constraints of Section 3.3 for contended scenarios. In Long Job Dominant and Adversarial, jobs requesting 128 nodes cannot all start at time 0, so FCFS wait times must be positive for at least some jobs, and the average wait time cannot be zero. Omitting wait time from the normalized comparison because of this value invalidates the wait-time and fairness conclusions drawn in Section 3.5, including the claims of reduced wait time for LLM schedulers and OR-Tools' fairness degradation.","section":"3.5"},{"comment":"The OR-Tools baseline is described only as 'an optimization-based scheduling solution' that computes globally optimal or near-optimal schedules; the objective function, constraints, and solver configuration are not stated anywhere in the paper. The recurring claim that OR-Tools maximizes utilization at the expense of fairness is not interpretable without knowing the scalar objective it was configured to optimize. Please specify the OR-Tools model (objective, constraints, variable definitions) or point to the exact code in the public repository.","section":"3.3"},{"comment":"The overhead analysis is internally inconsistent: Section 3.7.1 excludes delay calls from the measurements, Section 3.7.3 says scheduling times range from 1-2 hours for 100 jobs, and Figure 6's caption reports 4,000 s (O4-Mini) and 700 s (Claude 3.7) at 100 jobs, with a 6.9k s spike at 80 jobs. Deployment conclusions should be based on total wall-clock time including all LLM queries, and the stated ranges should match the reported data.","section":"3.7"}],"minor_comments":[{"comment":"The abstract writes 'FJFS' where FCFS is meant; please correct the typo.","section":"Abstract"},{"comment":"The caption contains the typo 'perofrmance'; please fix it.","section":"Figure 7"},{"comment":"'on-perm' should be 'on-premises'.","section":"3.7.3"},{"comment":"The notation in the turnaround-time formula is unclear; x_j d_j could be read as a product, but the surrounding text suggests it means the completion time x_j + d_j. Please define it explicitly.","section":"3.2"},{"comment":"'BurstyIdle' should be 'Bursty + Idle' for consistency with Section 3.1.","section":"3.5"},{"comment":"The sentence 'Claude 3.7, Or-Tools achieves the highest wait time fairness' is missing punctuation or a verb; please clarify which method achieves the highest fairness.","section":"5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has the ingredients of a useful empirical study, and the public artifact is a plus. My main concern is that the wait-time inconsistency is not cosmetic: it directly affects Figures 3, 4, 7, and 8 and the abstract's claims. I would not reject outright because the fix is feasible within the manuscript's scope, but the authors need to rerun or carefully restrict the quantitative claims before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: the paper is a useful proof-of-concept for using reasoning-capable LLMs in HPC scheduling, but the central wait-time and fairness claims are not interpretable as written.\n\nWhat's new and good: this is the first evaluation of O4-Mini and Claude 3.7 with a ReAct-style loop and scratchpad memory for multiobjective HPC scheduling. The framework separates natural-language reasoning from constraint enforcement, which is sensible and gives interpretable traces. The paper includes seven synthetic scenarios plus a Polaris trace, and the code and prompts are public. The overhead analysis is honest: 1–2 hours for 100 jobs, so not real-time viable. These are real contributions.\n\nThe soft spot is load-bearing. Section 3.3 fixes all submission times at s_j=0 and defines wait time as start time. Section 3.5 then says the average wait time for all methods, including FCFS, was 0.0 and drops it from normalization. That is impossible under any contention, e.g., Long Job Dominant with 128-node jobs on 256 nodes. FCFS cannot have zero average wait if any job starts after time 0. Worse, Sections 3.1 and 4 describe Poisson dynamic arrivals that contradict the simultaneous-arrival model. Fairness is computed from wait times via Jain's index; if wait times were truly 0, fairness would be 1.0 for all methods, yet the paper reports fairness differences. This means Figures 3, 4, and 7's wait-time and fairness comparisons are void. This is not a minor typo; it undermines the headline claim of multiobjective balance.\n\nAlso, the OR-Tools baseline objective is never stated. If OR-Tools is minimizing utilization or makespan alone, its fairness degradation is an artifact of objective choice. The authors need to specify what OR-Tools optimizes before claiming the LLMs balance objectives better.\n\nThe makespan, utilization, and throughput results may still hold, but they are secondary to the paper's stated contributions. The real-trace section is honest about its assumptions and the limitations, but it too reports wait-time and fairness improvements that inherit the measurement problem.\n\nMy recommendation: send it to peer review, but with a clear expectation of major revision. The topic is timely and the framework is worth evaluating, but the wait-time metric and the OR-Tools objective must be fixed before the central claims can be trusted. If the arrival model is clarified and the experiments re-run consistently, this could be a solid contribution. As is, I would not cite it for wait-time or fairness results, though I might cite it as an example of the approach.","headline":"The paper's first-of-its-kind LLM scheduling evaluation is worth a look, but the wait-time and fairness comparisons are invalidated by an internal contradiction in the problem setup.","tokens_in":15223,"tokens_out":3002,"would_cite":false,"duration_ms":28528,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A ReAct-style LLM scheduler can balance makespan, wait time, utilization, throughput, and fairness for small HPC workloads without retraining.","keywords":["Large Language Models","ReAct prompting","multiobjective scheduling","HPC job scheduling","scratchpad memory","constraint enforcement","zero-shot reasoning","Jain's fairness index"],"falsifier":"Re-run the same seven workload scenarios and the 100-job Polaris trace with OR-Tools configured to maximize a weighted objective that includes Jain's fairness index; if that version attains LLM-level fairness without sacrificing utilization or throughput, the paper's central comparison collapses because the LLM advantage becomes a baseline-configuration artifact.","tokens_in":14197,"feed_emoji":"⚖️","tokens_out":11263,"duration_ms":103605,"temperature":0.7,"pith_summary":"This paper tries to establish that a current reasoning LLM, prompted with a ReAct-style reason-and-act loop, can make multiobjective HPC job-scheduling decisions that compete with classical heuristics and an optimization solver on small workloads while explaining each decision in natural language. The authors evaluate O4-Mini and Claude 3.7 against FCFS, SJF, and Google OR-Tools over seven synthetic workload patterns and a 100-job real trace, and report that the LLM schedulers maintain balanced performance across makespan, wait time, utilization, throughput, and fairness without domain-specific training. Their advantage shows most clearly in heterogeneous, bursty, and long-job-dominated scenarios, where heuristics suffer convoy effects and OR-Tools trades fairness away for utilization. If true, this would give HPC operators an inspectable, zero-retraining scheduling option for small-to-medium batch workloads, while the paper's overhead measurements show that cloud-based reasoning APIs are too slow for real-time scheduling today.","feed_headline":"LLM schedulers balance HPC job queues and explain picks","feed_subtitle":"Tests on 10-100 job workloads show reasoning models match optimizer throughput while keeping fairness.","key_machinery":"The load-bearing mechanism is the ReAct (Reason+Act) loop augmented with a scratchpad. At each decision point, the LLM receives a text prompt containing the current system state, the waiting jobs with their resource demands, and a running scratchpad that logs prior thoughts, actions, and feedback; it responds with a Thought and one of StartJob, BackfillJob, Delay, or Stop. A separate constraint-enforcement module (the discrete-event simulator) checks the proposed action against node and memory capacities, executes it if feasible, and otherwise returns natural-language feedback that is appended to the scratchpad. This design separates the LLM's reasoning from feasibility guarantees and gives the agent memory and self-correction without any fine-tuning.","core_discovery":"The authors claim that separating LLM reasoning from constraint enforcement is what makes zero-shot scheduling work: the LLM proposes actions, a discrete-event simulator validates feasibility, and rejected actions generate natural-language feedback that is appended to the agent's scratchpad for later steps. On seven workload scenarios with 10 to 100 jobs, they observe that OR-Tools achieves the highest utilization and throughput but degrades wait-time and user-level fairness, while the LLM schedulers keep all objectives in balance; in Long Job Dominant and Bursty Idle scenarios, the LLMs sharply reduce wait and turnaround times relative to FCFS and SJF. In the Heterogeneous Mix at 80-100 jobs, Claude 3.7 and O4-Mini maintain throughput around 1.4x and utilization above 1.4x versus FCFS while preserving fairness, whereas OR-Tools reaches higher utilization (up to about 1.8x) at the cost of worse wait and turnaround times. On the 100-job Polaris trace, the LLM schedulers match OR-Tools on latency-related metrics while keeping utilization on par with baselines. The authors also find that overhead grows with queue size, with Claude 3.7 faster and more stable than O4-Mini, and conclude that current cloud-based LLM services are not viable for real-time scheduling but may suit batch or periodic optimization.","pith_inferences":["The paper measures 'balance' qualitatively across normalized metrics rather than as a formal Pareto front; a natural extension would be to convert the seven objectives into a scalar or Pareto evaluation and test whether LLM choices are actually on the frontier.","Because the paper never states what objective OR-Tools was configured to optimize, an immediate test is to rerun the comparison with a fairness-aware OR-Tools objective; if that configuration closes the fairness gap, the LLM advantage is partly an artifact of baseline setup.","The observation that per-call latency, not call count, drives overhead suggests a cheap extension: route only the hardest decision steps to a slow reasoning model and let a fast heuristic handle routine steps, which could cut wall-clock time without losing the balance reported here.","The scratchpad mechanism is a kind of in-context learning, so a testable sequel is to feed the agent explicit per-user fairness statistics and ask it to rebalance, which may amplify the fairness gains already observed in the real-trace experiment."],"forward_implications":["If the central claim holds, zero-shot LLM schedulers can serve as interpretable alternatives for small-to-medium batch scheduling windows, where knowing why each job was chosen matters more than sub-second latency.","The advantage is workload-dependent: LLM reasoning pays off in heterogeneous, bursty, and long-job-dominated mixes, while homogeneous or resource-sparse workloads leave little room for improvement over simple heuristics.","Constraint satisfaction is guaranteed by the simulator rather than by the model, so the scheduler remains safe even when the LLM proposes infeasible actions.","The reported overhead implies that practical deployment today would be limited to periodic or strategic scheduling decisions, or to on-premises fast reasoning models, rather than to online real-time job submission.","The successful transfer to a 100-job Polaris trace, under an assumed idle initial state, suggests the framework generalizes from synthetic logs to real job histories without retraining."],"supporting_citations":[{"why":"Supplies the ReAct reason-and-act prompting framework that the LLM scheduling agent is built on.","marker":"[Yao et al.(2023)]"},{"why":"Provides the OR-Tools optimization baseline whose schedules are compared against the LLM schedulers.","marker":"[Perron and Furnon(2024)]"},{"why":"Defines Jain's index, the fairness metric used in the multiobjective evaluation.","marker":"[Sediq et al.(2013)]"},{"why":"Characterizes FCFS with backfilling, a baseline heuristic and the source of the convoy-effect comparison.","marker":"[Srinivasan et al.(2002)]"},{"why":"Establishes the HPC job-scheduling problem as NP-hard and motivates the reinforcement-learning baseline the authors deliberately avoid.","marker":"[Wang et al.(2021)]"}],"fun_headline_variants":["LLMs balance HPC job objectives with explainable picks","Reasoning LLMs handle HPC scheduling, but overhead hampers real-time","LLM schedulers match optimizers on HPC fairness, not speed","Study: LLM-based HPC scheduling trades speed for transparent reasoning","LLM+scratchpad scheduling: multiobjective HPC balance, extra compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the unstated premise that the baselines, especially OR-Tools, were configured in a way that makes the multiobjective comparison fair; because the paper never specifies what objective OR-Tools was set to optimize, the fairness gap between OR-Tools and the LLMs could be an artifact of baseline setup rather than a property of LLM reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs balance HPC job objectives with explainable picks","Reasoning LLMs handle HPC scheduling, but overhead hampers real-time","LLM schedulers match optimizers on HPC fairness, not speed","Study: LLM-based HPC scheduling trades speed for transparent reasoning","LLM+scratchpad scheduling: multiobjective HPC balance, extra compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000248,"raw_usage":{"total_tokens":1633,"prompt_tokens":1116,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":436}},"tokens_in":732,"tokens_out":517,"duration_ms":6043,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:44:28.599734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same seven workload scenarios and the 100-job Polaris trace with OR-Tools configured to maximize a weighted objective that includes Jain's fairness index; if that version attains LLM-level fairness without sacrificing utilization or throughput, the paper's central comparison collapses because the LLM advantage becomes a baseline-configuration artifact.","supporting_citations":[],"review_version":1}