{"id":"06c41f14-fcd9-44c5-b196-40c207ea8df9","arxiv_id":"2501.18191","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new SST component simulates HPC job scheduling with five policies and DAG workflows, with qualitative validation against CQsim and workload traces.","lead":"The paper introduces a job scheduling and resource management component for the SST parallel discrete-event simulator, supporting five standard scheduling policies and DAG-based workflow dependencies. It compares simulated job wait times and node usage against CQsim and real workload logs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy validation may be circular: the text says trace data determine job start/end times, which would make wait-time and occupancy matches to DAS-2/CQSim predetermined; the paper must show scheduling policy actually produces those times.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I find: the Section 4.1 sentence 'We use the trace data to determine the start time and end time of jobs in our simulator' creates a potential circularity. The abstract's 'good accuracy' claim depends on the simulator actually recomputing start times from scheduling decisions, not reading them from the trace. This is not a disagreement with HPC scheduling consensus; it is an internal ambiguity that, under one reading, makes the headline metrics predetermined. I agree with the reader. I do not identify an independent second concern as equally load-bearing: the scalability and workflow claims are secondary and would not rescue the accuracy claim. No code is provided, so I cannot resolve the ambiguity by inspection; hence the concrete test requires either code release or a trace-input ablation. The conditional verdict is appropriate: the paper can be accepted in principle if the authors clarify input semantics and show policy sensitivity. I would not move the verdict to reject because the ambiguity may be resolvable and the SST integration has some plausibility, but the circularity must be resolved before the accuracy claim is relied upon.","tokens_in":8994,"tokens_out":5166,"duration_ms":53455,"concrete_test":"Require the authors to make the trace-reading code and SST configuration public, then run two experiments with the DAS-2 trace: (1) feed the trace without any start/end fields (only submit time, runtime, node request) under FCFS and record wait-time and occupancy curves; (2) run the identical input under SJF. If (1) still matches DAS-2/CQSim within the paper's claimed accuracy, the scheduler is genuinely producing start times. If (1) degrades or (2) produces curves identical to (1) or to the trace, then trace start/end times are being replayed and Section 4.2's accuracy validation is circular. As a minimal first check, inspect whether the trace parser reads the 'start_time' or 'end_time' columns.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is that the SST scheduler actually decides when jobs start and end, and then its outputs are compared with ground truth. Section 4.1 undermines this: 'We use the trace data to determine the start time and end time of jobs in our simulator.' On the literal reading, the simulator consumes the very quantities it is supposed to predict. Then Figure 3(a)'s occupied-node curve is just a binning of trace start/end times and trace node counts, and Figure 4(a)'s wait times are trace start times minus trace submission times, so a close match to DAS-2/CQSim is guaranteed. The ambiguity matters because Listing 1's TaskEvent has executionTime but no slot for a trace-supplied start/end timestamp, so the intended input semantics are not documented. If the sentence means only 'execution durations are taken from the trace,' the authors must say so and show the scheduler generates start times; if it means actual start/end times are replayed, both validation metrics collapse. Either way, the accuracy claim is currently unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a job scheduling and resource management component for the Structural Simulation Toolkit (SST), a parallel discrete-event simulator. The component supports five job scheduling algorithms (FCFS, FCFS with Best Fit, FCFS with Backfilling, SJF, LJF) and a workflow management module for DAG-based task dependencies. Validation is reported against CQSim using the GWA-DAS2 trace and against a SIPHT workflow, and MPI scalability results are presented for the DAS-2, SDSC-SP2, and Galactic workflow workloads.","tokens_in":9223,"tokens_out":4115,"duration_ms":41049,"significance":"If its claims were established, the component would be a useful extension of SST, addressing the need for scalable and workflow-capable HPC scheduling simulation. The paper has concrete strengths: a modular design integrated into a known simulator, a clearly described task-event interface (Listing 1), a workflow input format with a JSON example, and an effort to test on real workload traces. However, the current evidence is not sufficient: the accuracy validation is at risk of circularity because Section 4.1 says trace data determine job start and end times, and the reported matches in Figures 3 and 4 are purely visual with no quantitative error measures. The parallel performance claim also lacks concrete speedup or efficiency numbers.","major_comments":[{"comment":"The sentence \"We use the trace data to determine the start time and end time of jobs in our simulator\" is load-bearing for the accuracy claim. If taken literally, the simulator consumes the very quantities it is supposed to predict: job start and end times come from the DAS-2 trace rather than from the scheduling policy. In that case, the node-occupancy match in Figure 3(a) and the wait-time match in Figure 4(a) are not independent validation but a restatement of the trace. The paper must clarify the intended input semantics: either the scheduler actually generates start times and only execution durations and resource demands are taken from the trace, or the validation is circular. Listing 1's TaskEvent has executionTime but no field for trace-supplied start/end timestamps, so the intended semantics are not documented.","section":"Section 4.1"},{"comment":"The accuracy claim is supported only by visual curve inspection: the text says the occupancy and wait-time outputs are \"similar\" and \"closely matches\" CQSim, but no quantitative error metric (e.g., mean absolute error, RMSE, relative error, correlation) is reported. Please add numerical error measures for wait time and node occupancy, and state which scheduling algorithm (FCFS, backfilling, etc.) was used for the validation runs and all relevant configuration parameters so the comparison is reproducible.","section":"Section 4.2, Figures 3 and 4"},{"comment":"There is a direct inconsistency about the workflow experiments. The last paragraph of Section 4.1 says \"we plan to extend our testing to encompass real-life workflow simulations using Pegasus,\" yet Section 4.2 reports results for the Galactic workflow from the Pegasus gallery (Figure 6) and a validation against real-life SIPHT measurements (Figure 7). Please correct the status description. In addition, Figure 7's validation is not described: the ground-truth source, the compared metric (per-task wait time? makespan?), the workflow configuration, and the error measure are all missing, so the workflow validation claim is currently unsupported.","section":"Section 4.1 vs Section 4.2"},{"comment":"Algorithm 1 describes a simple immediate-allocation loop that schedules a job only when enough cores are available, which corresponds to FCFS without backfilling or best-fit. The paper claims support for five scheduling algorithms, but the pseudocode does not specify how backfilling, best-fit, SJF, or LJF are implemented inside the event-driven framework, nor how the priority queue is ordered differently per policy. Without this specification, the paper does not establish that the five algorithms are actually simulated. Please provide the per-policy queue ordering and event rules, and indicate which algorithm produced each reported result.","section":"Section 2.2, Algorithm 1"},{"comment":"The parallel performance claim is not quantified. Figures 5 and 6 appear to lack labeled axes, and the text gives no speedup, parallel-efficiency, or wall-clock-time values, no hardware/SST configuration (number of ranks, host machine, synchronization settings), and no comparison against ideal scaling or against other simulators. Please report concrete timings and speedups at the rank counts shown, and add axis labels to the figures.","section":"Section 4.2, Figures 5 and 6"}],"minor_comments":[{"comment":"The phrase \"introduces a workflow management components\" should be \"introduces a workflow management component.\"","section":"Abstract"},{"comment":"The sentences about the 4seq, 5seq, and 6seq epigenomic datasets mention that these are analyzed, but no result or plot involving them appears anywhere in the paper; either report the corresponding analysis or remove the mention.","section":"Section 4.1"},{"comment":"The in-text citation \"Simulator. 2024\" in Section 2.2 does not match the reference list entry \"SST Simulator. 2024\"; please unify the citation format. Also, the reference for Mitchell et al. (2019) is listed with the title \"Alea 2: Job Scheduling Simulator,\" which appears to be a copy-paste error for a paper about Apache Airflow; please correct it.","section":"References"},{"comment":"The wait-time comparison across the five scheduling algorithms would be more informative with quantitative summary statistics (mean and median wait time, or a table), and with a statement of the workload and resource configuration used; currently the figure is described only qualitatively.","section":"Section 4.2, Figure 4(b)"},{"comment":"The task representation section lists fields such as resource_requirements and dependencies, but it does not state whether these fields are stored in the same TaskEvent class as in Listing 1 or in a separate class; please clarify the relationship between the job-level TaskEvent and the workflow-level Task representation.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the circular validation described in Section 4.1. If the authors confirm that the scheduler only reads execution durations and produces start/end times, and they add quantitative error measures, the paper could become a solid contribution. If the literal reading is correct, the central validation claim collapses and would require a new experimental setup. The workflow section also needs a clear statement of what was actually run versus planned. I would send the paper back for major revision rather than reject, because the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a real piece of work: a job-scheduling and resource-management element inside SST, plus a workflow/DAG component. That integration is genuinely new for SST, and the scalability results on DAS-2 and SDSC-SP2 suggest the component parallelizes well. The five scheduling policies are textbook, but having them inside a cycle-accurate, parallel discrete-event simulator is useful for people who want to study scheduling under realistic architecture models. The workflow JSON input is a nice touch.\n\nThe problem is the validation. Section 4.1 says \"We use the trace data to determine the start time and end time of jobs in our simulator.\" Read literally, that makes the accuracy claims in Figures 3(a) and 4(a) circular: if the simulator consumes the trace's start/end times, then its occupied-node curve and wait-time distribution are just the trace's own values, and matching CQsim is guaranteed. The paper never clarifies whether the trace provides only submission times and run durations, with the scheduler actually computing start times. That is the load-bearing distinction, and it is unresolved. Also, the validation is entirely visual; there are no quantitative error measures. Code is not released, so there is no independent check.\n\nThe workflow validation is thinner still: a few \"simplified workflow scenarios\" and one wait-time comparison against \"real-life measurements\" from SIPHT, with no description of those measurements. That part reads as preliminary, and the paper says as much.\n\nMinor issues: the references include a copy-paste error (the Mitchell et al. 2019 entry is titled \"Alea 2\"), and the text has a few typos. These are trivial by comparison.\n\nWhere does this leave us? The design is coherent and the integration is honest engineering. The central accuracy claim is unverified, but it is probably fixable: state clearly that the trace supplies only submission times and execution durations, add even one quantitative error metric, and release the code. If the authors meant what Section 4.1 literally says, then the accuracy claim collapses, and they need to redo the validation. If it is a wording slip, the paper is a solid workshop-level contribution to the SST ecosystem.\n\nMy call: send it to peer review. A referee should force the clarification, because the tool could be useful to the HPC simulation community, and the flaw is fixable. I would not cite it until the validation is straightened out, but I would bring it to a reading group as a case study in how easy it is to make validation circular.","headline":"A useful SST scheduling element whose accuracy validation may be circular, since the text says trace data determine job start and end times.","tokens_in":9692,"tokens_out":3324,"would_cite":false,"duration_ms":31296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An SST-based simulator accurately reproduces HPC job wait times and node usage, and scales in parallel, the paper claims.","keywords":["HPC job scheduling","discrete-event simulation","SST","workflow management","resource allocation","parallel simulation","backfilling","workload traces"],"falsifier":"An immediate check is to feed the same trace to two different scheduling policies in the simulator and see whether the reported wait times change. If the simulator uses the trace's recorded start times as fixed inputs, every policy would produce identical wait times, and the validation against the trace's wait times would be tautological. A reader could compare a subset of individually reported job wait times from the simulator against the trace's wait times for those exact jobs, and verify whether the simulator's wait times equal the trace's wait times to the tick.","tokens_in":8844,"feed_emoji":"⏱️","tokens_out":8092,"duration_ms":64015,"temperature":0.7,"pith_summary":"The paper introduces a job scheduling and resource management component built inside the structural simulation toolkit (SST), a parallel discrete-event simulator. It claims that this simulator reproduces real HPC job behavior--specifically job wait times and node occupancy--when replaying a large workload trace, and that it scales well as more MPI ranks are added. The simulator supports five scheduling policies (FCFS, shortest job first, longest job first, FCFS with best fit, and FCFS with backfilling) and adds a workflow layer that models task dependencies as a directed acyclic graph. The significance would be a single simulation framework where HPC architects can test both scheduling policies and workflow dependencies at scale without disturbing a production system.","feed_headline":"SST simulator matches HPC wait times, scales in parallel","feed_subtitle":"A new SST component replays real workload traces and enforces workflow dependencies, the paper reports.","key_machinery":"The structural simulation toolkit (SST) is a modular, parallel discrete-event simulator that uses conservative synchronization; the paper's component plugs scheduling decisions into this engine as event handlers. Jobs arrive as serialized TaskEvent objects, are placed into a priority queue, and trigger resource allocation and deallocation as simulation time advances. The workflow component stores tasks and their interdependencies as a directed acyclic graph using adjacency lists, so that a task becomes eligible only when all of its dependency tasks have completed. This combination--event-driven resource accounting plus DAG-based dependency tracking--is what the paper relies on to produce accurate wait times, node usage, and workflow behavior.","core_discovery":"The central claim is that job scheduling and resource management for HPC systems can be modeled faithfully inside SST's event-driven parallel engine. The authors report that their component's node-occupancy curve and job wait times closely match a reference cluster scheduling simulator when both replay the same real-world workload trace, and that parallel speedup grows with the number of MPI ranks on two large traces (about 1.1 million jobs and about 73,000 jobs). They also report that a workflow management component, which represents dependencies among tasks as a DAG and triggers tasks when their predecessors complete, reproduces the wait-time behavior of a real bioinformatics workflow. The paper frames this as validation of both accuracy and scalability in one framework.","pith_inferences":["A stronger version of this validation would feed only arrival times and resource requests to the simulator and let each policy compute start times; the current setup's reliance on trace start and end times leaves open whether the wait-time match reflects the scheduler or the trace.","The same component could be extended to test policies the trace never used, such as replacing first-come-first-served with backfilling on a repeat of the same workload, which would show whether the simulator can predict the effect of a policy change.","The workflow DAG representation could be combined with the network and memory models already in SST to study how data movement between dependent tasks affects overall workflow makespan, a direction the paper does not pursue."],"forward_implications":["HPC scheduling policies can be exercised and compared within a parallel discrete-event engine that also models network and memory effects.","Million-job workload traces can be replayed at scale, since the simulator's runtime improves as MPI ranks increase.","Workflow-driven workloads can be simulated with dependency constraints enforced, allowing evaluation of scientific pipelines before deployment.","The five implemented policies provide a basis for controlled head-to-head experiments on identical traces."],"supporting_citations":[{"why":"Describes the structural simulation toolkit (SST), the parallel discrete-event engine the component is built inside.","marker":"Rodrigues et al. 2011"},{"why":"CQsim, the reference cluster scheduling simulator the paper validates against.","marker":"SPEAR Lab. 2023"},{"why":"The Grid Workloads Archive, source of the DAS-2 trace used for validation.","marker":"Iosup et al. 2008"},{"why":"The SDSC-SP2 workload log used to test scalability.","marker":"San Diego Supercomputer Center. 2000b"},{"why":"Pegasus, the workflow system whose style of workflows the component is meant to support and validate against.","marker":"Deelman et al. 2015"},{"why":"SIPHT workflow, the real bioinformatics workflow used for workflow wait-time validation.","marker":"Juve 2014"},{"why":"Supplies the DAG-based representation for workflow dependencies.","marker":"Gupta et al. 2017"}],"fun_headline_variants":["SST scheduler: HPC wait times matched, parallel speedup shown","New SST component models HPC scheduling and workflows","SST simulates HPC job queues with real trace accuracy","Scalable SST scheduling: accurate wait times, workflow DAGs","HPC scheduling fidelity in SST: validated and parallel"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulator's wait-time validation is meaningful, which requires that the scheduling policy, not the trace file, generates job start times; the paper says it uses the trace to determine start and end times, so this premise is not settled.","fun_headline_variants_meta":{"raw":{"variants":["SST scheduler: HPC wait times matched, parallel speedup shown","New SST component models HPC scheduling and workflows","SST simulates HPC job queues with real trace accuracy","Scalable SST scheduling: accurate wait times, workflow DAGs","HPC scheduling fidelity in SST: validated and parallel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1754,"prompt_tokens":800,"completion_tokens":954,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":870}},"tokens_in":416,"tokens_out":954,"duration_ms":7644,"temperature":1.0,"reasoning_tokens":870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:20:13.627116+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An immediate check is to feed the same trace to two different scheduling policies in the simulator and see whether the reported wait times change. If the simulator uses the trace's recorded start times as fixed inputs, every policy would produce identical wait times, and the validation against the trace's wait times would be tautological. A reader could compare a subset of individually reported job wait times from the simulator against the trace's wait times for those exact jobs, and verify whether the simulator's wait times equal the trace's wait times to the tick.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SIPHT workflow, the real bioinformatics workflow used for workflow wait-time validation."},{"cited_title":"Choudhary, and P","cited_arxiv_id":null,"evidence_quote":"Supplies the DAG-based representation for workflow dependencies."},{"cited_title":"Hemmert, B","cited_arxiv_id":null,"evidence_quote":"Describes the structural simulation toolkit (SST), the parallel discrete-event engine the component is built inside."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CQsim, the reference cluster scheduling simulator the paper validates against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Grid Workloads Archive, source of the DAS-2 trace used for validation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pegasus, the workflow system whose style of workflows the component is meant to support and validate against."}],"review_version":1}