{"id":"37848d4b-9c40-4397-97cb-e1891e0b0ff7","arxiv_id":"2506.13903","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A graph built from classification rules yields a feature-importance score and a rule-set distance that perform competitively with and more robustly than common importance metrics.","lead":"Rule-based classifiers in medicine are hard to read once they accumulate hundreds of rules. This paper turns the rules into a feature graph so clinicians can see which measurements matter alone or together, and compares rule sets from different models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The feature graph, centrality, and distance all inherit the unvalidated relevance scores of Eqs. (7)-(8); the synthetic test is visual only, so the central claims stand or fall with those input metrics.","rationale":"I considered whether the lack of error bars in Table 1 or the visual-only synthetic evaluation is more load-bearing. Those are real limitations, but they affect the strength of the empirical evidence rather than the logical dependence of the claimed importance metric on the input relevance functions. The reader's weakest assumption identifies this same dependence, so my assessment agrees with the reader. Since the reader already recommends CONDITIONAL, I do not adjust the verdict; the proposed test would supply the missing independent validation and would either support or refute the centrality and robustness claims. The concern is about an unvalidated input metric, not about the authors or their intent.","tokens_in":11548,"tokens_out":5410,"duration_ms":66089,"concrete_test":"On the synthetic datasets of §4.1, rebuild the feature graph after replacing I_box and I_nabla with two alternatives already mentioned in the paper: (a) support/confidence and (b) impurity gain. Quantitatively compare how well each choice separates independent versus combined predictive features (e.g., diagonal-to-off-diagonal weight ratio) against the known generation process. If the known structure is recovered only for Eqs. (7)-(8), the method's advertised interpretability is metric-dependent; if recovered for all choices, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every downstream object — the adjacency matrix A' in Eq. (3), the importance score in Eq. (5), and the distance in Eq. (4) — is a function of the feature relevance I_box(D, v_h, R_k) and rule relevance I_nabla(D, R_k) defined in Eqs. (7)-(8). These functions are adopted from the LLM literature and are never independently validated in the paper. The synthetic evaluation (§4.1) only shows heatmaps side-by-side with Gini importance; it does not quantitatively test whether P and q recover the known relevance structure, and Section 4.3 compares centrality to other importance metrics only through downstream DT accuracy on top-k features, which is an indirect and noisy proxy. The robustness comparison in §5.3 is likewise computed on rankings derived from these same P and q values. Since the paper itself defers 'support, confidence, lift, and impurity gain' to future work (Section 6), the choice of Eqs. (7)-(8) is explicitly untested. If those scores misweight features — e.g., if they favor features in high-covering rules or ignore rule interactions — the graph edges, centrality rankings, and the claimed 'competitive performance and superior robustness' all shift. The central claim is therefore conditional on an unvalidated input metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a graph-based framework for estimating feature contributions in rule-based classifiers. It constructs a tripartite graph among features, rules, and classes, projects it onto a feature-level weighted graph via Eq. (3), and defines a feature importance score as degree centrality (Eq. 5) and a distance metric between rule sets as the Frobenius norm of the adjacency difference (Eq. 4). The graph adjacency is built from feature relevance and rule relevance scores adopted from the Logic Learning Machine literature (Eqs. 7-8). The authors evaluate the approach on synthetic data, two clinical datasets, and 15 public benchmarks, comparing the importance score against permutation importance, Gini importance, and SHAP in terms of top-k classification accuracy and rank robustness.","tokens_in":11764,"tokens_out":5854,"duration_ms":56271,"significance":"If the proposed importance metric is shown to be reliable, the framework would provide a unified, model-agnostic way to visualize feature interactions and compare rule sets, which is of practical value in high-stakes domains such as healthcare. The paper makes the implementation publicly available and implements alternative relevance metrics, both of which are positive features. However, the central claims are conditional on the validity of the input relevance scores and on the projection in Eq. (3) not distorting the contribution structure; the current evidence is not yet sufficient to support the strong claims of competitive performance and superior robustness.","major_comments":[{"comment":"The complement-product form in Eq. (3) can saturate: when even a few rules have moderately large p_ki * p_kj * q_k, the product over k of (1 - p_ki p_kj q_k) approaches zero, so A_ij tends to 1 and becomes insensitive to the magnitude of the relevance. As a result, the degree centrality in Eq. (5) increasingly reflects the number of rules in which a pair of features co-occurs rather than the strength of their joint contribution. This directly affects the validity of the proposed importance metric. Please provide a diagnostic (e.g., the distribution of A_ij values) and consider an alternative aggregation such as a sum or max that preserves magnitude information.","section":"Section 3, Eq. (3)"},{"comment":"The synthetic evaluation is entirely visual: no quantitative measure is reported for how well the graph, or the underlying relevance scores of Eqs. (7)-(8), recover the known ground-truth feature structure. Because the entire framework inherits the properties of these input scores, the evaluation should include a quantitative recovery test (e.g., Spearman correlation between the computed centrality and the ground-truth importance, with standard errors). The paper already states that alternative relevance metrics (support, confidence, lift, impurity gain) are implemented; comparing the results under these alternatives would substantially strengthen the claim that the framework is robust to the choice of input metric.","section":"Sections 4.1 and 5.1"},{"comment":"The accuracy values are reported as point estimates without any measure of variance (standard deviation or confidence interval). Given that the robustness claim is central, the average Spearman correlations should also be reported with their variability across folds. Without error bars, it is impossible to assess whether the differences between methods are meaningful. Please report means and standard deviations across CV folds (or repeated runs) for both accuracy and rank robustness.","section":"Section 5.3, Table 1"},{"comment":"The paper states that the fidelity of the extracted rule sets was evaluated with accuracy and F1, and that the number of rules was set to 15, but these fidelity results are never reported. Since the clinical insights in Section 5.2 are derived from surrogate rule sets, the reader needs to see how faithful these surrogates are to the original black-box models.","section":"Section 4.2"}],"minor_comments":[{"comment":"The abstract and several places in the text contain spacing/formatting errors, e.g., 'discerncrucialfeatures' and 'uncovercrucial'; please fix the PDF compilation and proofread the text.","section":"Abstract and throughout"},{"comment":"In Table 1, 'Hill V alley' should be 'Hill Valley'.","section":"Table 1"},{"comment":"The centrality values and the pair-wise distance matrix are described qualitatively; please include the numerical values in a table so that the claims can be verified.","section":"Section 5.2"},{"comment":"The graph is described as undirected, but this is only valid if the adjacency matrix is symmetric; please state explicitly that A' is symmetric given non-negative P and q.","section":"Section 3"},{"comment":"The term 'model-agnostic' is used for the importance metric, but the benchmark evaluation only uses decision trees; please clarify the scope of the experiments.","section":"Section 5.3"},{"comment":"The rule relevance I_nabla = covering * (1 - error) is one of several possible choices; consider reporting how the main results change if lift or confidence is used instead.","section":"Eq. (6)"},{"comment":"Reference [6] is a water-distribution application of LLM; for the relevance metrics defined in Section 3, please also cite the original LLM source where these metrics are introduced.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The second author is affiliated with Rulex, which provides the LLM implementation used in the experiments, and the relevance metrics in Eqs. (7)-(8) come from that ecosystem. The paper does not hide this and implements alternative metrics, but the lack of a sensitivity analysis to those alternatives remains a concern. I would ask the authors to run the synthetic recovery experiment with at least one alternative relevance metric before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid, workmanlike contribution to interpretability for rule-based models. What is actually new: the weighted projection in Eq. (3), which uses a complement-product form to aggregate joint feature relevance across rules, plus class-specific feature graphs, a Frobenius-norm distance between normalized adjacency matrices, and a degree-centrality feature importance score. These are sensible extensions of known bipartite-projection ideas, and the paper does a good job of specifying the framework formally and making the code public. The benchmark evaluation on 15 datasets, with four rule-based methods and two detailed clinical case studies, is a real strength. The clinical findings (glucose–age and glucose–BMI interactions on Pima, class-specific interactions on breast tissue) are plausible and well connected to the medical literature.\n\nThe soft spots are real but not fatal. The stress-test concern is the most important: every downstream object—adjacency, centrality, distance—inherits the feature and rule relevance scores from Eqs. (7)–(8), which are adopted from the LLM literature (one author is involved) and never independently validated within the paper. The synthetic evaluation is only qualitative heatmaps; it does not quantitatively show that the graph recovers known feature structure. The benchmark performance is measured indirectly, by training DTs on top-k features, which can wash out the contribution of the importance metric itself. Table 1 also has no error bars, and the robustness column is a single Spearman value per metric/dataset, so \"superior robustness\" is asserted on thin evidence. The saturation behavior of Eq. (3)—where the product quickly approaches one when many rules mention the same pair—is a minor concern but worth a sentence of analysis. These issues are fixable: the paper already mentions alternative relevance metrics in the code, and the method is agnostic to P and q, so a sensitivity analysis (e.g., using support, confidence, lift, or impurity gain) would substantially strengthen the claims.\n\nThat said, the central framework is sound and the paper is honest about its scope. It is internally consistent, and the input-relevance dependency is a feature of any interpretability pipeline, not a unique flaw. With added error bars, a quantitative synthetic check, and a sensitivity analysis of P and q, the claims would hold up. I think this deserves a serious referee, not a desk rejection. I would read it myself if I were working on rule-based interpretability, though I am unlikely to cite it directly in my own near-term work.","headline":"A clearly specified graph-based framework for analyzing feature contributions in rule-based classifiers, with a novel weighted projection and degree-centrality importance score, but the validation is thinner than the claims warrant: synthetic results are visual only and the input relevance metric is adopted without independent scrutiny.","tokens_in":12299,"tokens_out":2404,"would_cite":false,"duration_ms":27833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the degree centrality of its projected feature graph is a valid model-agnostic feature importance metric for rule-based classifiers, performing comparably to permutation importance and with more stable rankings.","keywords":["rule-based systems","feature importance","feature graphs","interpretable machine learning","graph centrality","rule set comparison","feature interactions","clinical decision support"],"falsifier":"On a synthetic rule set whose true pairwise contributions are known by construction, compare the centrality ranking from Equation (5) with the ranking obtained when Equation (8) is replaced by support, confidence, lift, or impurity gain; if the ranking materially changes with the injected score, the graph is measuring that score rather than the model's actual feature use.","tokens_in":11328,"feed_emoji":"🕸️","tokens_out":8914,"duration_ms":94333,"temperature":0.7,"pith_summary":"This paper tries to establish a unified way to read rule-based classifiers: turn the rule set into a graph whose nodes are input features, and let the connections record when two features contribute jointly to the same rules. The authors define feature importance as the total weight attached to a node (degree centrality) and claim this score is model-agnostic, computationally cheap, and more stable across data splits than permutation importance, Gini importance, or average Shapley values. On 15 public benchmarks the score matches permutation importance in top-k accuracy and is the most robust of the four. The same graph carries interaction information that a one-dimensional importance list cannot, and a Frobenius distance between graphs lets users compare rule sets over the same features. The practical payoff, illustrated on two clinical datasets, is the ability to see which pairs of clinical measurements are predictive only in combination.","feed_headline":"Graph centrality rivals permutation importance on 15 benchmarks","feed_subtitle":"A graph built from rules shows which features matter alone versus together, and the rankings stay stable across data splits.","key_machinery":"The central object is the projected feature graph: a weighted undirected graph on input features obtained from a tripartite network of features, rules, and class labels. Its adjacency matrix is computed by Equation (3) as a normalized product over rules, with each rule contributing a factor $1 - p_{ki} p_{kj} q_k$, where $p$ is a feature-to-rule relevance score and $q$ is a rule-to-class relevance score. Degree centrality of this adjacency matrix is the proposed importance measure, self-edges keep features that appear alone in rules, and the class-restricted projection gives class-specific feature graphs. The Frobenius distance between two normalized adjacency matrices is the proposed distance between rule sets.","core_discovery":"The central claim is that projecting a tripartite features-rules-classes network onto the feature set produces a feature graph whose degree centrality is a valid feature importance metric, independent of the rule-induction algorithm. In that graph, an edge between two features is strong when the pair appears together with high relevance in at least one rule that matters, while self-edges preserve the contribution of features that appear alone in rules. The authors evaluate this on synthetic data with known independent and combined predictive structure, on two clinical datasets with four rule-induction schemes, and on 15 public benchmarks against permutation importance, Gini importance, and average Shapley values. They report that the centrality score performs comparably to permutation importance in top-5 and top-10 feature-selection accuracy and is the most stable ranking across cross-validation folds. The same construction restricted to rules with one class gives class-specific interaction graphs, and the Frobenius distance between adjacency matrices serves as a rule-set distance that does not require pairwise rule matching.","pith_inferences":["A testable extension is to feed the edge weights, not just node centrality, into feature selection: strong edges between a predictive feature and a spurious one might flag correlations that survive single-feature rankings but not leave-one-out evaluation.","Because the projection multiplies contributions, a single highly relevant rule containing a rare feature pair can dominate an edge; comparing multiplicative aggregation with additive aggregation on the same synthetic data would reveal whether that emphasis is a strength or a distortion.","The framework's stated agnosticism to the input relevance scores suggests it could be used as a general contribution-to-graph wrapper around any local explanation method, although the paper only demonstrates it with scores adopted from logic learning machines.","The robustness result hints at a practical rule of thumb: when data splits are noisy, graph centrality may be a safer basis than permutation importance for deciding which patient information to collect, a claim that could be tested directly on the same benchmarks with different training seeds."],"forward_implications":["Rule sets from different induction schemes, such as decision trees, logic learning machines, association rules, and extracted neural-network rules, can be compared on a common feature-importance scale without re-training or pairwise rule matching.","Clinicians can see which measurements are predictive only in combination, so data collection can prioritize pairs like glucose with age or with body mass index rather than treating all risk factors independently.","Replacing a rule set in an expert system can be guided by graph distance, choosing the update whose feature-contribution graph is closest to the existing one and thereby preserving continuity of clinical explanations.","Class-specific projections expose interactions that aggregate importance hides, such as phase angle and impedance carrying the signal for carcinoma while a single spectral feature suffices for adipose tissue in the breast-tissue case.","The reported rank stability across decision-tree depths suggests that the centrality score can serve as a reliable feature-ranking tool when data splits are noisy, reducing the need for extensive hyperparameter tuning."],"supporting_citations":[{"why":"Supplies the permutation importance method that is the main accuracy and robustness baseline for the new centrality score.","marker":"[1]"},{"why":"Provides the logic-learning-machine relevance scores for rules and features, Equations (7)-(8), which the whole graph construction consumes.","marker":"[6]"},{"why":"Gives the layered rule-network representation that the tripartite features-rules-classes graph extends.","marker":"[15]"},{"why":"Provides the bag-of-words rule-set comparison and the combined feature relevance background the distance metric is positioned against.","marker":"[13]"},{"why":"Defines average Shapley values, the third baseline in the benchmark comparison.","marker":"[16]"},{"why":"Defines Gini importance, the decision-tree baseline used in the benchmark comparison and synthetic experiments.","marker":"[20]"},{"why":"Supplies the bipartite-projection weighting strategies that motivate the paper's weighted, uniqueness-preserving projection.","marker":"[28]"},{"why":"Provides the rule-extraction scheme used to turn a neural network into rule sets for the clinical case studies.","marker":"[24]"}],"fun_headline_variants":["Feature graphs rank rule-based model inputs stably","Rule graphs turn feature importance into a network","Feature centrality matches permutation scores on 15 benchmarks","New metric ranks rule features via graph links","Graph importance for rule models proves robust across splits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole graph is built from two input scores, feature-to-rule relevance and rule-to-outcome relevance, adopted from an existing method without independent validation, so if those scores misrepresent true contributions the edge weights, centrality rankings, and graph distances all inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Feature graphs rank rule-based model inputs stably","Rule graphs turn feature importance into a network","Feature centrality matches permutation scores on 15 benchmarks","New metric ranks rule features via graph links","Graph importance for rule models proves robust across splits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2130,"prompt_tokens":972,"completion_tokens":1158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1089}},"tokens_in":588,"tokens_out":1158,"duration_ms":9395,"temperature":1.0,"reasoning_tokens":1089,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:26:17.608756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic rule set whose true pairwise contributions are known by construction, compare the centrality ranking from Equation (5) with the ranking obtained when Equation (8) is replaced by support, confidence, lift, or impurity gain; if the ranking materially changes with the injected score, the graph is measuring that score rather than the model's actual feature use.","supporting_citations":[{"cited_title":"Bioinformatics26(10), 1340–1347 (2010)","cited_arxiv_id":null,"evidence_quote":"Supplies the permutation importance method that is the main accuracy and robustness baseline for the new centrality score."},{"cited_title":"Computers12(6), 123 (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the logic-learning-machine relevance scores for rules and features, Equations (7)-(8), which the whole graph construction consumes."},{"cited_title":"In: 2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)","cited_arxiv_id":null,"evidence_quote":"Gives the layered rule-network representation that the tripartite features-rules-classes graph extends."},{"cited_title":"IEEE Journal of Biomedical and Health Informatics (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the bag-of-words rule-set comparison and the combined feature relevance background the distance metric is positioned against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Gini importance, the decision-tree baseline used in the benchmark comparison and synthetic experiments."},{"cited_title":"Physical review E76(4), 046115 (2007)","cited_arxiv_id":null,"evidence_quote":"Supplies the bipartite-projection weighting strategies that motivate the paper's weighted, uniqueness-preserving projection."},{"cited_title":"In: CEUR WORKSHOP PROCEED- INGS","cited_arxiv_id":null,"evidence_quote":"Provides the rule-extraction scheme used to turn a neural network into rule sets for the clinical case studies."}],"review_version":1}