{"id":"653e405d-19f1-4b1c-8282-2b3ab9d189ca","arxiv_id":"2507.02690","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An RL-driven heterogeneous graph neural network that selects among four graph structures per process instance achieves modest average accuracy gains on six business process event logs.","lead":"This paper proposes RLHGNN, a model that builds four kinds of graphs from business process event logs and uses reinforcement learning to pick one graph per process instance before predicting the next activity. The authors report small average accuracy gains over six public datasets and about one millisecond inference latency, but the gains are not consistent across datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RL reward in Eq. (3) is never tied to a concrete model: the paper never states which network computes Accuracy(s,a), and Section V-B's 'baseline establishment' split is the only clue.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and my concern is a sharper form of the reader's noted 'key RL reward details are missing.' The single most load-bearing assumption is not merely that the three edge types capture useful signal; it is that the RL agent has a well-defined training objective at all. Without a specification of which model produces Accuracy(s,a), the DQN cannot be trained from the text, and the adaptive selection mechanism - the paper's central claimed contribution - is not reproducible. The ablations in Table V are consistent with this concern: the average GMean advantage of RLHGNN over the fixed Comprehensive structure is 0.006, and over Context-aware is 0.010, while on BPI12CW the fixed Context-aware structure beats RLHGNN. These differences are small enough that a missing or misaligned reward definition, or seed noise, could explain them. My proposed concrete test directly traces the reward in the released code and compares the two plausible training variants; if neither matches the text, the central claim should be treated as unverified rather than accepted on the reported numbers. I do not find evidence of fabrication or internal inconsistency in the graph construction; the repeat-edge semantics are clearly described, and the complexity analysis is reasonable. The concern is purely about the RL training loop's specification and its coupling to the prediction model, which is the load-bearing part of the claimed novelty.","tokens_in":1097,"tokens_out":1261,"duration_ms":86486,"concrete_test":"Use the released repository (https://github.com/Joker3993/RLHGNN) to trace the reward computation: identify the exact model that produces Accuracy(s,a) in Eq. (3). Then run two variants: (a) reward from four frozen pre-trained fixed-structure models, and (b) reward from the current adaptive HGNN being trained, with the Q-network and HGNN updated as specified in the code. Compare Table IV/V on BPI12CW and BPI2020P across 10 random seeds. If variant (a) is the implemented design, verify whether the deployed HGNN is one of those frozen models; if not, the Q-values are misaligned and the RL component cannot be responsible for the reported gain. If variant (b) is implemented, record the DQN loss and HGNN accuracy curves to check convergence. If neither variant is implemented consistently with the text, the central RL claim is unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RLHGNN's core novelty is the RL policy that picks one of four graph structures per prefix. For the DQN to learn, every (state, action) needs a reward; Eq. (3) defines R(s,a)=α·Accuracy(s,a)-β·ComputeCost(a)+γ·Efficiency(s,a), but Accuracy(s,a) is never defined. The only procedural clue is Section V-B: 'training data further divided equally between baseline establishment and RL agent training.' If the reward is computed with four fixed, pre-trained structure-specific HGNNs (the 'baselines'), then the Q-network is optimized for those frozen models, while the deployed model in Eqs. (7)-(10) is a single HGNN with structure-specific weights that is presumably trained jointly with or after the RL agent. Nothing in the text specifies how the deployed model's parameters are learned, whether the Q-network and HGNN are updated alternately, or how non-stationarity is handled. If instead the deployed HGNN provides the reward, the DQN targets are non-stationary and the training loop is a moving-target problem that is not described. In either reading, the reported results are not reproducible from the manuscript. The consequence is direct: the 'RL-driven adaptive selection' is the paper's central contribution, and Table V shows it buys only 0.731 vs 0.725 average GMean over the fixed Comprehensive structure, with fixed Context-aware beating RLHGNN on BPI12CW (0.876 vs 0.872). Without a well-defined reward model, this small gap cannot be attributed to RL selection, to the edge semantics, or to noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLHGNN, a framework for next-activity prediction in business processes. It constructs heterogeneous process graphs with three edge types (forward, backward, repeat), builds four candidate graph structures of increasing complexity, and uses a Deep Q-Network to select a graph structure per process instance. The selected graph is processed by HeteroGraphConv with GraphSAGE aggregators to produce a next-activity distribution. Experiments on six BPI event logs compare RLHGNN with six baselines and include ablations against fixed graph structures, complexity analysis, and latency measurements. The central claim is that RL-based instance-level structure selection consistently outperforms static sequence- and graph-based state-of-the-art approaches while keeping inference latency near one millisecond.","tokens_in":19546,"tokens_out":5183,"duration_ms":53822,"significance":"If the core contribution is fully specified and the reported gains are statistically robust, the paper would be a useful contribution: it introduces an adaptive, per-instance structure-selection mechanism for process graphs, evaluates on six public datasets, reports training and inference costs, and makes source code available. These are real strengths. However, the current evidence does not yet establish the claimed advantage: the reward function that drives the RL selection is not operationalized, the 'consistent' improvement is contradicted by the paper's own tables, the average gains over the best fixed structure are very small, and no significance or variance information is reported. The central mechanism therefore needs either clearer specification or substantially stronger evidence before the main claims can be accepted.","major_comments":[{"comment":"The reward function R(s,a) = alpha * Accuracy(s,a) - beta * ComputeCost(a) + gamma * Efficiency(s,a) is never operationalized. The paper does not state which network produces Accuracy(s,a), how ComputeCost(a) and Efficiency(s,a) are measured, or whether rewards come from four frozen structure-specific HGNNs or from the deployed joint model. Section V-B's statement that training data is 'further divided equally between baseline establishment and RL agent training' is the only procedural clue; no alternating update schedule, DQN loss, or treatment of non-stationary targets is given. Because the RL selection policy is the paper's central contribution, this omission prevents reproduction and even formal evaluation of the method as specified.","section":"IV-D, Eq. (3)"},{"comment":"The claim that RLHGNN 'consistently outperforms all static structures' is contradicted by Table V. On BPI12CW the fixed Context-aware structure achieves a higher GMean (0.876 vs 0.872) and F1-score (0.797 vs 0.784) than RLHGNN. Across all six datasets the average advantage over the fixed Comprehensive structure is only 0.006 in GMean (0.731 vs 0.725) and 0.008 in F1 (0.576 vs 0.568). No confidence intervals, standard deviations, or significance tests are reported for the three-fold cross-validation, so these small differences may well be within noise.","section":"VI-B, Table V"},{"comment":"The statement that RLHGNN 'consistently outperforms state-of-the-art approaches' is not supported by Table IV. On BPI2020P, RLHGNN's accuracy is 0.873 versus MHG-Predictor's 0.886; on BPI13CP, RLHGNN's F1-score is 0.491 versus MHG-Predictor's 0.512. The paper achieves the highest average accuracy and F1, but not consistent per-dataset superiority. The wording should be corrected, and the per-dataset failures should be analyzed rather than only acknowledged in the text.","section":"Abstract and VI-A, Table IV"},{"comment":"The repeat-edge construction connects every occurrence of an activity to successors of every other occurrence, which creates O(m^2) bidirectional edges per repeated activity and can introduce long-range links between positions that are neither temporally nor semantically adjacent. The paper provides no analysis of graph density or evidence that these edges are informative. Table V offers little support: the Pattern-aware structure (forward plus repeat) is no better than Basic on average (GMean 0.702 vs 0.700; F1 0.533 vs 0.529). This weakens the premise that the three handcrafted edge types provide complementary signal for the RL policy to exploit.","section":"IV-C, ConnectRepeatedActivities"}],"minor_comments":[{"comment":"Table VII labels the unit as 'hour' for inference latency, but the numeric values and the text in Section VI-C ('average inference latency of 1.018 milliseconds') clearly refer to milliseconds. The unit should be corrected and used consistently in the table and text.","section":"Table VII and VI-C"},{"comment":"The sentence 'To address these limitations, We present in this paper RLHGNN' uses an uppercase 'We' in the middle of a sentence; please fix the capitalization.","section":"I, last paragraph of Introduction"},{"comment":"The claimed efficiency advantage E[Cost(Gi)] < Cost(G4) depends on the RL policy selecting simpler structures more often, but the paper does not report the distribution of selected structures or per-structure measured latency. Adding such empirical evidence would make the complexity argument concrete.","section":"IV-F and Table VII"},{"comment":"The statement that RLHGNN 'outperforms the strongest baseline (HiGPP) by 3.4% in accuracy and 3.8% in F1-score' on BPI12CW is accurate for those two numbers, but it appears before the per-dataset caveat and should be qualified as one dataset rather than presented as the general pattern.","section":"VI-A, BPI12CW discussion"}],"recommendation":"major_revision","confidential_remarks":"The two strongest graph baselines (HiGPP and MHG-Predictor) come from the authors' own earlier work, and the edge-type grounding also references those papers; this is acceptable but the novelty over those methods should be made more explicit. I also recommend that the editor verify the GitHub repository and, if possible, the reported timing numbers, since the latency table currently has a unit inconsistency that could indicate a deeper reporting problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper's genuinely new piece is the RL-based per-instance choice among four heterogeneous graph structures for next activity prediction—nothing in the cited prior work does exactly that. The framework is clearly described, the three edge types are grounded in process mining ideas, and the evaluation covers six public logs with ablations against fixed structures. Credit where it's due: the authors admit where their method loses (BPI2020P accuracy; BPI12CW to the fixed Context-aware structure), which is more honest than most submissions in this area.\n\nThe soft spots are real. The abstract says 'consistently outperforms state-of-the-art approaches,' but Table IV shows RLHGNN below MHG-Predictor on BPI2020P, and Table V shows the fixed Context-aware structure beating RLHGNN on BPI12CW. Average gains over the best static structure are tiny: 0.731 vs 0.725 GMean, 0.576 vs 0.568 F1. There are no error bars or significance tests, so a 0.01 accuracy edge could be noise.\n\nThe load-bearing problem is the reward definition. Eq. (3) defines R(s,a) in terms of Accuracy(s,a) but never says which model computes that accuracy. Section V-B's split—training data divided equally between 'baseline establishment' and 'RL agent training'—is the only hint. If four fixed, pre-trained per-structure HGNNs provide the rewards, then the Q-network is optimized against those frozen models, while the deployed model in Eqs. (7)-(10) is a single HGNN with structure-specific weights. The paper never explains how that deployed model is trained, whether it is updated alternately with the Q-network, or how non-stationarity is handled. If instead the deployed HGNN supplies the reward, the DQN targets are non-stationary and that training loop is not described either. Either way, the center of the paper is not reproducible from the manuscript.\n\nThis is an incremental paper, not a breakthrough. The RL selection idea is worth taking seriously, but the evidence that it helps is weak and the method description has a missing core. I'd send it to peer review—the idea is important enough and the gaps are fixable—but the referee should demand a concrete specification of the reward model, significance testing, and a softened headline claim. For a reader who cares about next-activity prediction, the ablation tables are useful even if the RL mechanism is unproven.","headline":"An honest incremental paper whose RL-selection core is under-specified and whose headline overstates small gains; send to review, but demand the reward model be pinned down.","tokens_in":20051,"tokens_out":2740,"would_cite":false,"duration_ms":28066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RLHGNN claims that a reinforcement-learning agent choosing per-instance among four heterogeneous graph structures outperforms static sequence- and graph-based next-activity predictors, averaging 0.782 accuracy and 0.578 macro F1 on six…","keywords":["next activity prediction","heterogeneous graph neural networks","reinforcement learning","process mining","business process monitoring","graph structure selection","predictive process monitoring"],"falsifier":"On BPI2020P the paper already reports MHG-Predictor beats RLHGNN in accuracy (0.886 versus 0.873). A decisive test would be to compute, on a log with known loops and parallel branches, the accuracy of RLHGNN against each of its four fixed structures and against a random or oracle structure selector; if the RL-selected structures do not beat the best fixed structure by more than noise, or if the learned policy's choices match random selection, the adaptive-selection claim collapses.","tokens_in":18978,"feed_emoji":"🎯","tokens_out":4926,"duration_ms":53473,"temperature":0.7,"pith_summary":"RLHGNN claims that next-activity prediction in business processes improves if the model can choose, for each running case, how much graph structure to build: only sequential forward edges, forward plus backward edges, forward plus repeat edges, or all three. The choice is made by a Deep Q-Network trained as a Markov Decision Process, and the chosen graph is processed by HeteroGraphConv with relation-specific GraphSAGE aggregators. Across six public event logs the paper reports average accuracy 0.782 and macro F1 0.578, above six sequence- and graph-based baselines, with about 1 ms inference latency. The authors position the contribution as instance-level adaptability: simple cases get cheap structures, complex parallel or looping cases get richer ones, without manual configuration.","feed_headline":"RL agent picks graph shape per process case, top accuracy 0.782","feed_subtitle":"Policy chooses among four heterogeneous process graphs, beating fixed-graph and sequence baselines across six event logs.","key_machinery":"The carrying mechanism is the heterogeneous process graph plus its per-instance selector. Nodes are activities of a prefix trace; edges are typed forward (consecutive activities), backward (reverse of consecutive activities), and repeat (bidirectional links between occurrences of the same activity and successors of other occurrences). Four progressive structures G1 through G4 combine these edge types, and a three-layer Deep Q-Network over hand-crafted structural, temporal, and pattern features picks one structure per instance. The predictor is HeteroGraphConv with GraphSAGE aggregators: LSTM aggregation on forward and backward edges preserves ordering, mean aggregation on repeat edges pools recurring patterns, and structure-specific weight matrices combine the relation outputs before a softmax classifier.","core_discovery":"The central claim is that an RL-driven per-instance choice among four heterogeneous process graph structures, rather than a fixed sequence or graph encoding, yields better next-activity predictions than state-of-the-art static methods. The paper constructs graphs whose edges carry three process-mining semantics: forward (directly-follows), backward (dependency), and repeat (loop/recurrence); these are combined into four structures of increasing complexity. A DQN policy selects the structure for each prefix trace, and a HeteroGraphConv with LSTM aggregators for forward and backward edges plus mean aggregation for repeat edges produces the activity distribution. On six real-world datasets the method averages 0.782 accuracy and 0.578 macro F1, with the largest gains on structurally diverse logs; the paper also reports that on the sparse BPI2020P log a static heterogeneous baseline achieves higher accuracy, which the authors attribute to insufficient structural regularity for policy learning.","pith_inferences":["A natural extension is to compare the RL policy's selections against an oracle selector and a random selector; the paper does not isolate how much of the gain comes from selection versus from heterogeneous aggregation itself.","The repeat-edge construction connects every pair of repeated occurrences, so on logs with high repetition the graph can grow quadratically; sampling repeat edges or learning edge weights is an untested refinement.","The DQN state features are hand-crafted; replacing them with learned prefix embeddings could transfer better to out-of-distribution cases, including the sparse regime where the paper reports weaker performance.","The paper's own suggestion of a static fallback for sparse logs could be formalized as a hybrid controller that switches between RL selection and a fixed comprehensive graph when data is scarce."],"forward_implications":["If RLHGNN is right, predictive process monitoring can treat graph construction as a per-case decision instead of a fixed preprocessing choice, so simple cases spend less compute and complex cases get richer structure.","The reported average inference latency of about one millisecond implies the adaptive mechanism is compatible with real-time service orchestration and monitoring loops.","The three process-mining edge semantics (directly-follows, dependency, and loop) become a reusable template for building heterogeneous graph features for business process prediction.","The ablation results imply that no single fixed structure dominates on all logs, so deployed predictors on structurally mixed data should consider instance-level or at least dataset-level structure selection."],"supporting_citations":[{"why":"Grounded the forward-edge construction in directly-follows relations from process discovery.","marker":"[25]"},{"why":"Supports backward edges as dependency-based historical context.","marker":"[26]"},{"why":"Supports repeat edges via loop detection and recurrence analysis.","marker":"[27]"},{"why":"Supplies the Deep Q-Network algorithm used for structure selection.","marker":"[41]"},{"why":"Provides the GraphSAGE inductive aggregation used inside HeteroGraphConv.","marker":"[43]"},{"why":"Heterogeneous graph baseline whose multi-layer design RLHGNN extends with per-instance selection.","marker":"[40]"},{"why":"Strong static graph baseline used to benchmark the accuracy gains.","marker":"[39]"}],"fun_headline_variants":["RL chooses per-case graph structure to predict next process step","Reinforcement learning picks process graph per trace, 0.782 avg accuracy","Adaptive graph selection via RL beats static graphs in next-activity prediction","Per-instance graph selection with RL improves business process predictions","RL-driven heterogeneous graphs hit 0.782 accuracy for next-activity prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the forward, backward, and repeat edge semantics add real predictive signal beyond a flat sequence model, and that the RL policy can learn when each structure is needed; if those handcrafted edges do not help, the adaptive selection mechanism has little to optimize.","fun_headline_variants_meta":{"raw":{"variants":["RL chooses per-case graph structure to predict next process step","Reinforcement learning picks process graph per trace, 0.782 avg accuracy","Adaptive graph selection via RL beats static graphs in next-activity prediction","Per-instance graph selection with RL improves business process predictions","RL-driven heterogeneous graphs hit 0.782 accuracy for next-activity prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3423,"prompt_tokens":986,"completion_tokens":2437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2358}},"tokens_in":602,"tokens_out":2437,"duration_ms":18287,"temperature":1.0,"reasoning_tokens":2358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:22:44.331459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On BPI2020P the paper already reports MHG-Predictor beats RLHGNN in accuracy (0.886 versus 0.873). A decisive test would be to compute, on a log with known loops and parallel branches, the accuracy of RLHGNN against each of its four fixed structures and against a random or oracle structure selector; if the RL-selected structures do not beat the best fixed structure by more than noise, or if the learned policy's choices match random selection, the adaptive-selection claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Grounded the forward-edge construction in directly-follows relations from process discovery."},{"cited_title":"Process mining with the heuristics miner-algorithm,","cited_arxiv_id":null,"evidence_quote":"Supports backward edges as dependency-based historical context."},{"cited_title":"Discov- ering block-structured process models from event logs - a constructive approach,","cited_arxiv_id":null,"evidence_quote":"Supports repeat edges via loop detection and recurrence analysis."},{"cited_title":"Mhg-predictor: A multi-layer heterogeneous graph-based predictor for next activity in complex business processes,","cited_arxiv_id":null,"evidence_quote":"Heterogeneous graph baseline whose multi-layer design RLHGNN extends with per-instance selection."},{"cited_title":"Higpp: A history-informed graph-based process predictor for next activity,","cited_arxiv_id":null,"evidence_quote":"Strong static graph baseline used to benchmark the accuracy gains."}],"review_version":1}