{"id":"f2d4c8ca-8fae-48ff-84f2-dc82c33ca3ee","arxiv_id":"2506.08850","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"aRL, a DQN-based edge scheduler with EDF-guided exploration and action masking, achieves higher hit-ratio and faster convergence than vanilla RL and heuristic baselines in simulated soft real-time task scheduling.","lead":"This paper introduces aRL, a reinforcement learning scheduler for edge computing that uses earliest-deadline-first exploration and action masking to speed up learning. In simulations, aRL reaches higher schedule hit-rates than vanilla DQN and standard heuristics while using less runtime, RAM, and power.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The runtime comparison is confounded: aRL's EDF-guided exploration may itself clear the 98% hit-ratio threshold, so the 46% speed-up over vRL is not shown to come from learning.","rationale":"The reader's weakest-assumption analysis identifies the same core issue: the convergence-based runtime metric can be satisfied by the EDF exploration heuristic itself, so the headline runtime reduction does not isolate the effect of learning. I read the paper in good faith: the method is a reasonable combination of action masking and informed exploration, the experimental setup is described in detail, and a source-code repository is promised. However, the central novelty is 'agile reinforcement learning,' i.e., faster convergence of a learned policy, and the current evidence does not rule out the possibility that the learned Q-network is largely unnecessary for the reported convergence speed. This is a load-bearing concern because the quantitative claims (46% runtime reduction, 65.5% RAM reduction, 60.6% power reduction) are all measured against this convergence criterion. The proposed ablation would settle whether the learned component contributes, and would not require new mathematical machinery. The reader's CONDITIONAL verdict remains appropriate: the paper is plausible and useful, but the main quantitative claim needs an additional control experiment and ideally confidence intervals across the 31 runs. I do not see grounds for REJECT, since the concern is testable and the method may well deliver the advertised benefit once the ablation is run.","tokens_in":10480,"tokens_out":4096,"duration_ms":50639,"concrete_test":"Run an ablation 'aRL-NoQ' that keeps Algorithm 1 exactly but removes policyNet updates and always takes the exploration action (epsilon = 1 throughout). If aRL-NoQ also exceeds the 98% hit-ratio threshold for 100 consecutive episodes and its convergence runtime is within noise of aRL, then the reported 46% speed-up is attributable to EDF-guided exploration and action masking, not to learned policy improvement. Also report the earliest episode at which the greedy (non-exploring) policy alone reaches the 98% threshold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 defines runtime for RL algorithms as learning time until the generated schedule's hit-ratio exceeds 98% for 100 consecutive episodes. In Algorithm 1, the exploration branch (lines 4-8) returns mapAction(GetEDFtask()), i.e., it schedules the earliest-deadline unassigned task to an available edge server satisfying resource constraints. This is a constructive deadline-aware heuristic, not random exploration. If this heuristic alone produces schedules with hit-ratio >= 98%, the convergence timer stops before the Q-network has contributed meaningful policy improvement, and the reported 46% runtime reduction over vRL compares informed exploration plus masking against random exploration plus the same DQN, rather than demonstrating that the learned policy converges faster. The paper does not report an ablation using only EDF exploration and action masking without Q-learning, nor an ablation using random exploration with masking, so the causal attribution of the headline speed-up to the RL learning component is unestablished. The fact that the plain EDF baseline has lower hit-ratio does not resolve this, because aRL's exploration policy is a modified EDF with feasible-server selection, not the same baseline EDF.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Agile Reinforcement Learning (aRL), a DQN-based scheduler for soft real-time tasks in edge computing. The method augments DQN with two mechanisms: action masking, which restricts assignments to unassigned tasks and feasible servers, and informed exploration, which replaces random exploration with an EDF-guided heuristic that selects the earliest-deadline unassigned task. The authors claim that this combination yields faster convergence and higher hit-ratio than vanilla DQN (vRL), EDF, and BestFit, with reported reductions of 46% in runtime, 65.5% in RAM usage, and 60.6% in power consumption relative to vRL. The evaluation is carried out in the EdgeSimPy simulator on a video surveillance scenario with 52 edge users and 4 heterogeneous edge servers, and the paper provides a link to source code.","tokens_in":10779,"tokens_out":4762,"duration_ms":46337,"significance":"If the claims are robust, the paper makes a useful practical contribution: it combines a simple, interpretable exploration heuristic with action masking to reduce the training time of a DQN scheduler, which is a known bottleneck for RL-based edge task scheduling. The problem formulation explicitly includes real-time constraints (deadline, response time, hit-ratio), which is appropriate for soft real-time applications. The availability of source code and 31-run logs is a strength and supports reproducibility. However, the significance is currently limited by the evaluation design: the convergence-based runtime metric is confounded with the exploration heuristic, and the absence of error bars or significance tests weakens the quantitative claims. The central idea is worth pursuing, but the evidence as presented is not yet convincing.","major_comments":[{"comment":"The headline claim that aRL converges 46% faster than vRL is confounded by the definition of convergence. Section 5 states that convergence is achieved when the hit-ratio of the generated schedule exceeds 98% for 100 consecutive episodes. In Algorithm 1, the exploration branch returns mapAction(GetEDFtask()), i.e., it schedules the earliest-deadline unassigned task to an available server satisfying resource constraints. This exploration policy is itself a constructive deadline-aware heuristic, so it may by itself produce schedules that meet the 98% threshold. Consequently, the reported runtime reduction could reflect the contribution of the EDF-guided exploration and action masking rather than the learned Q-network. The paper does not include an ablation with EDF exploration plus action masking but without Q-learning updates, nor an ablation with random exploration plus masking. Without such controls, the causal attribution of the speed-up to the learning component is not established. I recommend adding these ablations and/or reporting hit-ratio as a function of learning episodes separately from the exploration heuristic's standalone performance.","section":"Section 5 (Convergence criterion) and Section 4 (Informed Exploration; Algorithm 1, lines 4–8)"},{"comment":"Equation (3.5) defines execution time as eτj = (cτj × mτj)/(fsk × Nsk), where cτj is required processor cycles and mτj is required RAM. The preceding sentence states that execution time is obtained by dividing the total required number of processor cycles by the processing capacity, which would yield cτj/(fsk × Nsk). The product cτj × mτj is dimensionally inconsistent and appears to be a typo. This is load-bearing because eτj enters the response-time formula (3.4), the deadline constraint C1.1, and the reward function, and hence directly affects the measured hit-ratio. Please correct the formula or provide a justification for the multiplication.","section":"Section 3.1, Eq. (3.5)"},{"comment":"The experiments are repeated 31 times, but the paper reports only average hit-ratio, runtime, RAM usage, and power consumption, with no confidence intervals, standard deviations, or significance tests. The central quantitative claims (46% runtime reduction, 65.5% RAM reduction, 60.6% power reduction) are therefore not supported with evidence that these differences are stable or statistically meaningful. I request error bars or variance information for the RL-based methods, and at least a statement of the test used to compare aRL and vRL.","section":"Section 5 (Experimental results, Figures 3–5)"}],"minor_comments":[{"comment":"The loop condition 'while (t ≤ |T |) or (hitT asks== |T |)' appears to use OR where the text in Section 4 says the episode terminates when either the number of actions equals the number of tasks or all tasks are assigned. With OR, the loop continues even after all tasks are assigned, so the episode is not actually bounded by |T| actions. Also, the exploration branch 'Return mapAction(GetEDFtask())' does not specify how the edge server is chosen; please clarify.","section":"Section 4, Algorithm 1"},{"comment":"The utilization constraint is written as (UPsk + upτj) < 1, but the definition of upτj as eτj/dτj is not standard for a multi-server system with heterogeneous servers and may assume a single-server capacity of 1.0. Please clarify how utilization is normalized across different server speeds.","section":"Section 3.1, Eq. (3.6) and Eq. (3.7)"},{"comment":"There are several notation inconsistencies: Eq. (3.15) minimizes f2(Θ) while Eq. (3.14) defines f2(Φ); Algorithm 1 line 18 has malformed conditions; and the abstract contains a typo ('andlarge-scaleproblems'). Please proofread.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short conference submission with a potentially useful idea, but the evaluation is not yet rigorous enough to support the strong quantitative claims. The confounding of the runtime metric with the EDF exploration heuristic is the main concern; I would require ablations or a redesigned convergence metric before accepting. The Eq. (3.5) error is also serious and should be corrected. The 31-run logs are a good asset, and the authors should be encouraged to report variance. This is a major-revision rather than a reject because the core approach is plausible and the issues are addressable within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a clear, well-structured paper on a real problem. The MDP formulation includes hit-ratio, deadlines, and resource constraints, which is more real-time aware than most edge-scheduling RL papers. The vRL baseline is a reasonable control that shares the DQN architecture, and the authors promise code. The combination of action masking with a constructive EDF-based exploration is a sensible engineering idea for cutting learning time.\n\nThe problem: the headline number—46% faster than vRL—is not isolated to learning. Convergence is declared when hit-ratio exceeds 98% for 100 episodes, and aRL explores using a feasible-server EDF heuristic. If that heuristic alone clears 98%, the timer stops before the Q-network has learned anything useful. The paper gives no ablation with exploration-only (no Q-learning) or with masking-only, and no random-exploration-plus-masking condition. So the speedup could come entirely from the heuristic exploration, not from the RL loop. The fact that a plain EDF baseline gets only ~76% doesn't settle it, because aRL's exploration is a modified EDF that selects feasible servers.\n\nAlso, the reported averages over 31 runs have no confidence intervals or significance tests, Eq. (3.5) has a unit mismatch (cycles times RAM over frequency times cores), and the while condition in Algorithm 1 is typo'd (the OR should likely be an AND). These are fixable.\n\nBottom line: the paper is worth a serious referee—the idea is plausible and the review will help the authors design the right ablations—but the empirical claims should not be taken at face value. I'd ask for major revision before acceptance.","headline":"A sensible incremental idea on DQN for edge scheduling, but the headline speedup is confounded by the convergence criterion.","tokens_in":11248,"tokens_out":4049,"would_cite":false,"duration_ms":42585,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By replacing random exploration with an earliest-deadline heuristic and masking irrelevant actions, a deep Q-network scheduler for edge computing converges faster and schedules more soft real-time tasks on time than vanilla RL or standard…","keywords":["reinforcement learning","action masking","informed exploration","edge computing","task scheduling","soft real-time","deep Q-network","hit-ratio"],"falsifier":"Run aRL with uniform random exploration, identical action masking and reward function, and measure episodes to the same 98% threshold; if the runtime gap over vRL disappears, the claimed 46% saving comes from EDF-guided exploration rather than from faster learning.","tokens_in":10273,"feed_emoji":"⚡","tokens_out":8245,"duration_ms":89129,"temperature":0.7,"pith_summary":"The paper proposes aRL, a reinforcement-learning scheduler for soft real-time tasks offloaded onto heterogeneous edge servers. Its central claim is that a deep Q-network can learn to produce deadline-satisfying schedules quickly if two mechanisms curb the action space: informed exploration that always picks the earliest-deadline unassigned task, and action masking that forbids reassigning a task and limits each episode to one action per task. In an EdgeSimPy video-surveillance scenario, aRL reports a higher hit-ratio than EDF, BestFit, and a vanilla DQN baseline, while converging 46% faster and using 65.5% less RAM and 60.6% less power than the vanilla agent. The paper's intended significance is that slow training, the main obstacle to RL-based real-time schedulers, can be removed by borrowing structure from classical scheduling heuristics.","feed_headline":"EDF-guided RL scheduler trains 46% faster, hits more deadlines","feed_subtitle":"Informed exploration and action masking let a DQN scheduler reach 98% hit-ratio in 46% less time than vanilla RL.","key_machinery":"The named machinery is aRL's dual mechanism. Action masking uses a decision matrix $G$ whose entries mark whether task $\\tau_j$ has been assigned to server $s_k$; the single-assignment constraint restricts every action to the set of unassigned tasks $\\Gamma_{ua}$, and the action bound $|A| \\le |T|$ caps each episode at one action per task, so rollouts are short and every episode ends within a predictable horizon. Informed exploration replaces uniform random action selection with a modified Earliest Deadline First rule: during exploration the agent picks the unassigned task with the earliest absolute deadline and assigns it to an available server subject to capacity and utilization thresholds. The reward function accumulates positive rewards for satisfying the deadline, capacity, utilization, and single-assignment constraints and negative rewards for violations, so the DQN's value estimates are shaped by exactly the objectives the scheduler is meant to optimize.","core_discovery":"On the paper's own terms, the discovery is that random exploration is what makes RL schedulers slow, and that replacing it with an EDF-guided exploration policy while masking irrelevant actions concentrates learning on decisions that matter. The policy network still picks actions once the exploration threshold is passed, but the agent's exploration behavior is already deadline-aware. In the evaluated workload, aRL attains the highest hit-ratio among EDF, BestFit, and vRL, exceeding the heuristics by 24% and vRL by 1.9%, while reaching convergence 46% faster than vRL. The authors present these results as evidence that informed exploration and action masking improve the predictability of the agent, which in turn shortens the learning phase and makes RL practical for soft real-time edge scheduling.","pith_inferences":["The paper does not isolate the contribution of informed exploration from that of action masking; a natural extension is ablating each mechanism separately to see which one drives the runtime saving.","Because the exploration policy is itself EDF, the 98% convergence criterion may be partially satisfied by the heuristic rather than by the learned component; a fair test would compare against a random-exploration agent with identical masking to isolate the learning effect.","The same template could apply to other scheduling objectives by swapping EDF for a different prior, such as least-loaded server, best-fit resource match, or energy-aware ordering, suggesting the result may transfer beyond deadline-hit-ratio metrics.","Action masking shortens rollouts and stabilizes the number of actions per episode, which should reduce variance in value estimates; this could matter even more at the larger task scales the paper does not test."],"forward_implications":["If aRL's claim holds, deadline-guided exploration plus action masking offers a general recipe for making DQN schedulers trainable in edge environments where the state-action space is large and timing constraints are tight.","The measured runtime reduction to convergence implies that RL-based schedulers can be retrained or adapted when the edge environment changes, which is the setting that usually makes RL impractical for real-time workloads.","The higher hit-ratio over both EDF and BestFit indicates that a learned value function can improve on the heuristic that guides exploration rather than merely reproducing it.","The reductions in RAM and power consumption during learning mean the training process itself is cheaper, which matters on resource-constrained edge servers."],"supporting_citations":[{"why":"Defines soft real-time scheduling and establishes hit-ratio as the critical metric the scheduler optimizes.","marker":"[2]"},{"why":"Formulates edge-computing task scheduling as NP-hard and supplies the multi-resource constraints used in the problem statement.","marker":"[4]"},{"why":"Provides the prior priority-based RL scheduler and the experimental protocol of 31 independent runs used to average the reported metrics.","marker":"[20]"},{"why":"EdgeSimPy is the simulator in which aRL and all baselines are implemented and measured.","marker":"[21]"},{"why":"Defines the convergence criterion, hit-ratio above 98% for 100 consecutive episodes, that determines the runtime comparison for the RL methods.","marker":"[22]"}],"fun_headline_variants":["Deadline-aware RL scheduler cuts training time by 46%","Informed exploration makes RL edge scheduling practical","Action masking speeds RL scheduler convergence by 46%","EDF-guided exploration boosts RL hit-ratio and speed","Exploration that meets deadlines: RL scheduler trains 46% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that crossing the 98%-for-100-episodes hit-ratio threshold measures the quality of the learned policy, even though the agent explores by following EDF, so the threshold could be met largely by the exploration heuristic alone.","fun_headline_variants_meta":{"raw":{"variants":["Deadline-aware RL scheduler cuts training time by 46%","Informed exploration makes RL edge scheduling practical","Action masking speeds RL scheduler convergence by 46%","EDF-guided exploration boosts RL hit-ratio and speed","Exploration that meets deadlines: RL scheduler trains 46% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000593,"raw_usage":{"total_tokens":2784,"prompt_tokens":953,"completion_tokens":1831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1751}},"tokens_in":569,"tokens_out":1831,"duration_ms":16085,"temperature":1.0,"reasoning_tokens":1751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:00:17.111968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run aRL with uniform random exploration, identical action masking and reward function, and measure episodes to the same 98% threshold; if the runtime gap over vRL disappears, the claimed 46% saving comes from EDF-guided exploration rather than from faster learning.","supporting_citations":[{"cited_title":"Soft real-time scheduling","cited_arxiv_id":null,"evidence_quote":"Defines soft real-time scheduling and establishes hit-ratio as the critical metric the scheduler optimizes."},{"cited_title":"TF-DDRL: ATransformer-enhanced DistributedDRL Technique for Scheduling IoT Applications in Edge and Cloud Computing Environments","cited_arxiv_id":null,"evidence_quote":"Formulates edge-computing task scheduling as NP-hard and supplies the multi-resource constraints used in the problem statement."},{"cited_title":"A Task Scheduler for Mobile Edge Computing Using Priority-based Reinforcement Learning","cited_arxiv_id":null,"evidence_quote":"Provides the prior priority-based RL scheduler and the experimental protocol of 31 independent runs used to average the reported metrics."},{"cited_title":"EdgeSimPy: Python-Based Modeling and Sim- ulation of Edge Computing Resource Management Policies","cited_arxiv_id":null,"evidence_quote":"EdgeSimPy is the simulator in which aRL and all baselines are implemented and measured."},{"cited_title":"Convergent policy optimization for safe reinforce- ment learning","cited_arxiv_id":null,"evidence_quote":"Defines the convergence criterion, hit-ratio above 98% for 100 consecutive episodes, that determines the runtime comparison for the RL methods."}],"review_version":1}