{"id":"a6d3023b-6ad6-412b-9c86-c86e6f843414","arxiv_id":"2412.10484","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An ISM-based virtual fault tree and a graph convolutional network are combined to predict Fussell-Vesely importance on two small nuclear subsystems, with claims of millisecond inference and high accuracy.","lead":"This paper combines expert-built influence diagrams with a graph neural network to estimate Fussell-Vesely importance, a nuclear-safety risk metric, and reports fast predictions on two small subsystems. The idea is plausible, but the evidence is inconsistent: rare-event predictions are off by orders of magnitude, and the claimed speed advantage is contradicted by the paper's own timing data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-event FV predictions in Table 6 are off by 100-500x for rare events, so aggregate R2/MSE cannot support the claimed accuracy; the central claim fails regardless of ISM graph validity.","rationale":"The reader identified the ISM-to-Boolean-logic gap as the weakest assumption. That is a legitimate structural concern and remains unresolved, but the most load-bearing problem is more direct: the paper's own per-event evidence shows the trained model is not accurate where FV importance matters. Even if the ISM graph were a perfect encoding, a GNN that predicts 5.55e-7 as 2.93e-4 has not evaluated FV importance; it has only produced a rough ranking. Since the abstract and Section 5.2 claim high accuracy and the risk-informed decision support depends on actual FV values, this empirical failure is sufficient to reject the central claim. I therefore agree with the REJECT verdict without adopting the reader's primary reasoning; the ISM structural issue is secondary. A corrected paper would need per-event stratified error reporting, consistent metric tables, and ideally cross-topology training/testing before the claim could be reconsidered.","tokens_in":14318,"tokens_out":7439,"duration_ms":79418,"concrete_test":"Recompute the per-event predictions for the Section 5.4 case study (or rerun the trained GCN) and report relative error |y_hat - y|/y for every event, stratified by reference FV magnitude. Specifically check events with y<1e-3: CCF-SI-RF2-ALL, BUS-A-UN, and BUS-B-UN. Also verify Table 5's internal consistency: confirm that sqrt(0.0035)=0.0592 and sqrt(0.0036)=0.0600 for the first two rows, and that the Claude row is corrected to RMSE=48,305.68 and MAE=2,986.07. If the relative errors for y<1e-3 remain above 100%, the aggregate accuracy claim is not supported and the rejection stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the GNN surrogate actually evaluates FV importance, not merely fits aggregate error metrics. Table 6 is the only per-event comparison, and it contradicts the claimed accuracy. For CCF-SI-RF2-ALL the predicted FV is 1.985e-3 versus a reference 2.02e-5 (~98x too high); for BUS-A-UN and BUS-B-UN both predictions are 2.929e-4 versus reference 5.55e-7 (~527x too high). These are exactly the low-probability events that FV importance is supposed to prioritize in risk assessment. The aggregate metrics in Table 5 (MSE 0.0035, RMSE 0.0588, R2 0.9788) are dominated by the two near-1.0 FV values, so they mask this failure. The metric reporting is also internally inconsistent: Section 5.2 states MSE 0.0191, RMSE 0.1381, MAE 0.0979, R2 0.8832, which do not match Table 5; and the Claude row swaps RMSE and MAE (sqrt(2,333,439,021.98)=48,305.68, not 2,986.07). A model that cannot resolve FV at 1e-5 to 1e-7 cannot support the claimed 'rapid, data-driven calculation of FV importance' for safety-critical decisions, irrespective of whether the ISM reachability graph is a logically valid surrogate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a hybrid framework that combines Interpretive Structural Modeling (ISM) with a Graph Convolutional Network (GCN) to estimate Fussell-Vesely importance for basic events. The ISM step converts expert-elicited pairwise relations among six basic events into a directed acyclic graph, called a virtual fault tree; the GCN then takes event probabilities as node features and is trained to predict FV values. The paper presents aggregate error metrics, a comparison with an MLP and Claude 3.5 Sonnet, a time-efficiency comparison, and a scenario case study for a simplified nuclear-power-plant system. The central claims are that the virtual fault tree reduces complexity and that the GNN enables rapid, accurate, real-time FV calculation.","tokens_in":14678,"tokens_out":6827,"duration_ms":69040,"significance":"If the central claim were established, the idea of replacing fault-tree gate logic with an expert-elicited influence graph and learning FV importance with a graph neural network could be a useful direction for fast risk screening. The paper also makes a reasonable effort to compare against an MLP and an LLM, and it explicitly acknowledges the small scale of the tested fault trees. However, the evidence offered for the central claim is not convincing: per-event predictions for rare events are off by orders of magnitude, the reported accuracy metrics are internally inconsistent, and the time-efficiency claim is contradicted by the paper's own numbers. No code or data are provided, and the reported experiment is an interpolation test on two small systems rather than a demonstration of generalization to new fault-tree topologies. In its current form, the contribution does not support the abstract's claims of accurate and rapid FV evaluation.","major_comments":[{"comment":"The per-event FV predictions for exactly the rare events that FV importance is designed to prioritize are off by orders of magnitude. The predicted FV for CCF-SI-RF2-ALL is 1.985e-3 versus a reference of 2.02e-5 (about 98x too high), and for BUS-A-UN and BUS-B-UN the predictions are 2.929e-4 versus references of 5.55e-7 (about 527x too high). The aggregate metrics in Table 5 are dominated by the two events with FV near 1.0 and therefore mask this failure. Please report per-event errors, preferably on a logarithmic scale or stratified by FV magnitude, and state explicitly whether the model can resolve FV values at the 1e-5 to 1e-7 scale.","section":"Section 5.4, Table 6"},{"comment":"The numerical results for the proposed model are inconsistent between the text and the table. Section 5.2 reports MSE 0.0191, RMSE 0.1381, MAE 0.0979, and R2 0.8832, while Table 5 reports MSE 0.0035, RMSE 0.0588, MAE 0.0236, and R2 0.9788 for the same model. In addition, the Claude row swaps RMSE and MAE: sqrt(2,333,439,021.98) = 48,305.68, so RMSE should be 48,305.68 and MAE should be 2,986.07. The paper must state which numbers are final; as written, the accuracy and comparison claims are not reliably assessable.","section":"Section 5.2, Table 5"},{"comment":"The time-efficiency claim is contradicted by the reported numbers. The text says construction times were 1 minute 18.70 seconds and 2 minutes 0.16 seconds for the traditional and virtual fault trees, respectively, and then concludes that the virtual fault tree 'offers a significant advantage in terms of construction efficiency.' As written, the virtual tree took longer. Please clarify the measurement protocol, the units, the system under test, and either correct the claim or the reported times.","section":"Section 5.3"},{"comment":"The central methodological claim is that an ISM-derived DAG over basic events can serve as a virtual fault tree and that message passing over this graph can reproduce FV importance values defined by Boolean gate logic and minimal cut sets. No logical equivalence is established between the ISM reachability graph and the fault-tree semantics, and the GCN is trained on FV labels produced by RiskSpectrum from the same two fault trees on which it is later evaluated. The reported accuracy is therefore an interpolation test on a single data generator, not a predictive test on new fault-tree topologies. Please provide either a formal equivalence argument or an experiment with held-out topologies.","section":"Sections 3.2-3.3"},{"comment":"The experimental protocol is not sufficiently documented to evaluate the reported point estimates. The paper states that 304 and 316 data points were collected for the CS and SI systems but does not specify the train/validation/test split, the number of independent runs, the range of probability fluctuations used for data generation, or the variability of the metrics. In addition, Eq. (1) defines a single linear propagation D^{-1}AXW with no activation or self-loops, which does not match the three-layer GCN described in Section 4.3. Please document the full protocol and the exact layer update used.","section":"Section 4.2-4.3"}],"minor_comments":[{"comment":"The R2 formula is incorrect: it is missing the squared terms and should read R2 = 1 - Sum((yi - yhat_i)^2) / Sum((yi - ybar)^2).","section":"Eq. (10)"},{"comment":"The caption 'The Parameters α, β, and γ in the HCR Model' appears unrelated to the reachability-set table shown; please correct the caption.","section":"Table 3"},{"comment":"The caption 'Simplified system diagram for a NPP' seems to describe a different figure than the SSIM matrix shown; please verify all figure captions.","section":"Figure 6"},{"comment":"Please clarify the exact GCN update: Eq. (1) has no nonlinearity or self-loop term, while the text states three convolutional layers with hidden dimension 32.","section":"Section 4.3"},{"comment":"The parenthetical '(time units to be clarified)' should be resolved and the time measurement should identify which system and which construction task are being timed.","section":"Section 5.3"},{"comment":"The author-contributions section lists five individuals (Qi Ben, Peng Pengcheng, Liang Jingang, Tong Jiejuan, Wang Haitao) who are not present in the byline; this needs editorial verification.","section":"Author contributions"}],"recommendation":"reject","confidential_remarks":"For the editor: please verify authorship, because the byline contains two names while the author-contributions section names seven. The data were generated with proprietary RiskSpectrum software, and no code or data are provided, so the experiments are not currently reproducible. The topic is closer to reliability engineering than to core machine learning, but the scientific issues above are independent of that scope question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the paper's headline accuracy claim doesn't survive its own Table 6. For the rare events that FV importance is supposed to prioritize, the model is off by 100-500x. The time-efficiency claim is also contradicted by the construction times the authors report. So the preprint as written is not close to publishable. But the core idea—use ISM to build a simplified event graph, then train a GCN to predict FV importance—is genuinely new and worth a second look.\n\nWhat's good: the hybrid framing is sensible, the paper gives a clear motivation for avoiding minimal cut set enumeration, and the comparison against MLP and Claude is a reasonable sanity check. The authors also admit in the conclusion that they only tested small fault trees. That's honest.\n\nWhere it falls apart: Table 6 shows per-event predictions that are off by orders of magnitude for the low-probability events—CCF-SI-RF2-ALL predicted at 1.985e-3 vs a reference 2.02e-5, and both bus events predicted at 2.93e-4 vs 5.55e-7. The aggregate metrics (MSE 0.0035, R2 0.9788) are dominated by the two near-1.0 values, so they hide this. There are also mechanical problems: Section 5.2 gives one set of metrics, Table 5 gives another; the Claude row swaps RMSE and MAE; and the virtual fault tree took 2m0.16s to build vs 1m18.70s for the traditional tree, yet the text calls that a \"significant advantage.\" Those aren't typos in peripheral sections; they undercut the two central claims. The deeper conceptual worry—that the ISM reachability graph doesn't encode AND/OR gate logic, so the GCN is just fitting labels for one fixed system—is real but secondary to the empirical failures.\n\nBottom line: the idea may have legs, but this manuscript doesn't support it. I'd desk-reject with an invitation to resubmit after a major revision: fix the metrics, report per-event error, release the data/code, and validate on at least one structurally different, larger system. A serious referee would spend most of their time on mechanical errors.","headline":"The accuracy and speed claims don't survive the paper's own per-event numbers and time comparison, but the ISM+GCN idea has merit for a resubmission.","tokens_in":15195,"tokens_out":3977,"would_cite":false,"duration_ms":39815,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph model predicts Fussell-Vesely risk importance in milliseconds","keywords":["Fussell-Vesely importance","minimal cut sets","interpretive structural modeling","graph neural networks","virtual fault tree","fault tree analysis","real-time risk assessment","reliability engineering"],"falsifier":"Train the model on one small fault-tree topology and test it on a different topology of the same size that shares no gates or cut sets; if predictions degrade sharply or the ranking of critical events flips, the virtual fault tree has not captured the Boolean logic. A sharper test: keep the ISM reachability graph identical but change an AND gate to an OR gate inside the fault tree and recompute reference FV values; the GNN's output should change if the graph geometry encodes the logic, and should stay flat if it only fits probabilities.","tokens_in":14109,"feed_emoji":"🛡️","tokens_out":9026,"duration_ms":83991,"temperature":0.7,"pith_summary":"The paper proposes a way to compute Fussell-Vesely (FV) importance — a reliability metric that ranks how much each basic failure event contributes to overall system failure — without building a full fault tree or enumerating minimal cut sets. Its method replaces the Boolean fault tree with a 'virtual fault tree': a small directed graph of basic events and their dependencies, constructed from expert judgment via Interpretive Structural Modeling (ISM). A graph convolutional network reads this graph plus occurrence probabilities and predicts each event's FV importance. On two nuclear-plant subsystems the predictions match reference values computed by standard fault-tree software (R² = 0.9788), and inference takes milliseconds rather than minutes. The paper argues this makes real-time, risk-informed ranking of critical events practical for large dynamic systems.","feed_headline":"Graph model predicts Fussell-Vesely risk importance in milliseconds","feed_subtitle":"Expert-built event graph plus graph network gives millisecond FV rankings with R² of 0.9788","key_machinery":"The load-bearing object is the 'virtual fault tree': a directed acyclic graph produced by Interpretive Structural Modeling, in which nodes are basic events only and edges encode expert-elicited direct influence (a self-interaction matrix promoted to a reachability matrix via Boolean closure, then leveled into a hierarchy). The paper feeds this DAG, with basic-event occurrence probabilities as node features, into a graph convolutional network whose propagation rule is $Z = f(A,X) = D^{-1}AXW$, averaging feature messages from neighbors. The claim is that this message-passing over the dependency graph is enough to predict FV importance, which in classical reliability analysis is defined through Boolean AND/OR gates and minimal cut sets.","core_discovery":"On its own terms, the paper's central claim is that an ISM-derived reachability graph over basic events — a directed acyclic graph capturing expert-judged influences — can stand in for the full fault tree as the input representation for computing FV importance. Feeding this graph and the basic-event probabilities into a three-layer graph convolutional network yields FV values with MSE 0.0035, RMSE 0.0588, MAE 0.0236 and R² 0.9788 on the safety-injection and containment-spray systems, improving on a structure learned by HillClimbSearch and on a multilayer perceptron, and outperforming an LLM baseline. The authors assert that this hybrid structure-data approach captures dependencies between basic events, eliminates the need to handle intermediate events and minimal cut sets, and enables real-time recalculation as data streams update.","pith_inferences":["Cross-topology generalization remains open: the study reports only within-system holdout performance, so a decisive next test is training on one fault tree and evaluating on another with different gate structure.","Because each node's only feature is its occurrence probability, the ISM edges must carry all logical structure; an ablation that randomly permutes edges while keeping probabilities fixed would measure how much accuracy is genuinely structural.","The demonstration uses six basic events in two small systems, so the sufficiency of the reachability DAG for larger, repairable, or time-dependent systems is untested.","If the surrogate approach proves general, the same expert-graph-plus-GNN pattern could be applied to other importance measures, such as Risk Achievement Worth or Birnbaum importance, which share the same minimal-cut-set foundation."],"forward_implications":["FV importance becomes a real-time quantity: each change in basic-event probabilities can be re-ranked in milliseconds, supporting dynamic risk control.","The virtual fault tree contains only basic events, so model complexity and space no longer scale with intermediate-event structures.","The explicit ISM edges encode inter-event dependencies, avoiding the independence assumption of classical fault tree analysis.","The model's predicted importance ranking aligns with reference FV ordering in the case study, giving operators a prioritized list of events to monitor."],"supporting_citations":[{"why":"Supplies the Interpretive Structural Modeling procedure used to build the virtual fault tree from expert pairwise judgments.","marker":"[30]"},{"why":"The Delphi questionnaire protocol through which the three experts established the direct relationships between the six basic events.","marker":"[47]"},{"why":"The GNN survey that grounds the choice of graph neural networks for learning on graph-structured reliability models.","marker":"[25]"},{"why":"Defines the Fussell-Vesely importance measure that the framework predicts, and the hand-calculation approach it is meant to replace.","marker":"[12]"},{"why":"Gives the minimal-cut-set calculation that classical FV importance relies on, the cost the paper claims to avoid.","marker":"[6]"},{"why":"HillClimbSearch algorithm used to generate the data-driven comparison structure whose performance is contrasted with the ISM structure.","marker":"[48]"},{"why":"BDeu scoring function used by the HillClimbSearch baseline to select the alternative graph structure.","marker":"[49]"}],"fun_headline_variants":["Graph network computes FV importance in real time","Virtual fault trees enable fast FV importance via GNN","Millisecond FV importance from event graph and GNN","GNN on virtual fault tree speeds FV risk evaluation","Real-time FV importance using graph neural networks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the expert-drawn influence graph of basic events contains the same information as the full fault-tree logic, so that a graph network can reproduce Fussell-Vesely values without ever seeing the AND/OR gates or minimal cut sets.","fun_headline_variants_meta":{"raw":{"variants":["Graph network computes FV importance in real time","Virtual fault trees enable fast FV importance via GNN","Millisecond FV importance from event graph and GNN","GNN on virtual fault tree speeds FV risk evaluation","Real-time FV importance using graph neural networks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1559,"prompt_tokens":963,"completion_tokens":596,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":519}},"tokens_in":579,"tokens_out":596,"duration_ms":6934,"temperature":1.0,"reasoning_tokens":519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:25:04.534227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the model on one small fault-tree topology and test it on a different topology of the same size that shares no gates or cut sets; if predictions degrade sharply or the ranking of critical events flips, the virtual fault tree has not captured the Boolean logic. A sharper test: keep the ISM reachability graph identical but change an AND gate to an OR gate inside the fault tree and recompute reference FV values; the GNN's output should change if the graph geometry encodes the logic, and should stay flat if it only fits probabilities.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Interpretive Structural Modeling procedure used to build the virtual fault tree from expert pairwise judgments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Delphi questionnaire protocol through which the three experts established the direct relationships between the six basic events."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GNN survey that grounds the choice of graph neural networks for learning on graph-structured reliability models."},{"cited_title":"Fussell, How to hand-calculate system reliability and safety charac- teristics, IEEE Transactions on Reliability, vol","cited_arxiv_id":null,"evidence_quote":"Defines the Fussell-Vesely importance measure that the framework predicts, and the hand-calculation approach it is meant to replace."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the minimal-cut-set calculation that classical FV importance relies on, the cost the paper claims to avoid."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HillClimbSearch algorithm used to generate the data-driven comparison structure whose performance is contrasted with the ISM structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BDeu scoring function used by the HillClimbSearch baseline to select the alternative graph structure."}],"review_version":1}