{"id":"0705b56e-58c6-434c-b5cb-cde70edaccef","arxiv_id":"2504.12446","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors describe and prototype a heuristic that extracts hierarchical decision trees from feedforward neural networks by tracing activation paths, without proving equivalence for unseen inputs.","lead":"This paper proposes a step-by-step method and a Java prototype for converting trained feedforward neural networks into decision trees by tracing which connections matter for each input. It is relevant to explainable AI because decision trees are easier for people to audit than neural networks, but the paper does not prove the tree matches the network on new inputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equivalence is asserted, not established: the tree is built only from a finite set of traced inputs and pruned by an undefined relevance threshold, so it cannot be shown to reproduce the FNN on unseen inputs.","rationale":"Good-faith reading: the paper describes a prototype that traces individual forward passes and renders each trace as a decision path. That is a plausible XAI visualization, and the authors do provide runnable code. But the headline claim in the Abstract and Section V is stronger: equivalent symbolic models that preserve functionality. The construction is input-set dependent (Section V, steps 1-3) and relies on an undefined 'sufficiently relevant' threshold (Sections III.D and III.E). Both gaps are load-bearing: without a coverage argument the tree cannot be a full model of the FNN; without a threshold error bound it is not even guaranteed to match on the traced inputs. The reader's threshold concern is valid, and their rationale also notes per-input path construction; my emphasis is on the finite-coverage problem, which by itself invalidates the equivalence claim even if the threshold were set to zero. A direct empirical check with a tiny ReLU network can settle it. I therefore keep the rejection, with the recommendation that the paper be reframed as a trace-explanation tool unless equivalence over the full input domain is proven.","tokens_in":17379,"tokens_out":5151,"duration_ms":57630,"concrete_test":"Build a small deterministic FNN with continuous inputs, e.g., 2 inputs, one hidden layer with 4 ReLU units, and 2 softmax outputs. Run the Section V procedure with a finite set of 100 distinct input vectors to construct the hierarchical decision tree. Then sample 1e5 inputs uniformly from the same domain that are not in the original 100, and compare each tree prediction with the original FNN's argmax, recording (a) inputs for which the tree has no applicable path, and (b) inputs where the tree's class differs. Repeat with the relevance threshold set to 0 and to several positive values. If (a) or (b) is non-empty for any threshold, the tree is not functionally equivalent to the FNN. Optionally enumerate all linear regions of the ReLU network and check that the tree's leaves coincide exactly with those regions.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that derived symbolic decision models are 'equivalent' to the FNN and 'preserve functionality' (Abstract; Section V). The construction in Section V, however, works per input vector: 'For each of the input vectors to be examined,' it derives a decision path and then combines the paths into a tree. A finite set of traced paths cannot represent the FNN's decision function over a continuous input domain; no coverage argument is supplied. Section III.D steps 1-3 and Section III.E additionally discard all weighted connections whose product v_j*w_j is not 'sufficiently large' or 'sufficiently relevant,' with no definition of 'sufficient' and no error bound. Dropping even a small contribution can change an output-layer argmax, so the pruned tree may disagree with the original network even on traced inputs. The abstract also claims scalability to deeper networks through iterative subpath refinement, but recursive application of an undefined pruning step does not transfer equivalence. Thus the central claim is unsupported: the method yields per-input explanations under an unspecified heuristic, not an equivalent symbolic model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for deriving decision trees from trained feedforward neural networks (FNNs), using the filler/role distinction from Smolensky's integrated connectionist/symbolic architecture to map neuron activations and weighted connections to decision tree edges and nodes. The derivation proceeds per input vector: activations are computed, 'relevant' weighted connections are selected by an informally defined threshold, and the resulting decision paths are merged into a hierarchical tree. A Java prototype reads Keras .h5 files, converts convolutional and pooling layers into feedforward form, and visualizes the extracted tree. The central claim, stated in the abstract and Section V, is that the resulting symbolic structures are 'equivalent' to the network and 'preserve its functionality', with scalability to deeper networks via iterative subpath refinement.","tokens_in":17574,"tokens_out":2967,"duration_ms":33874,"significance":"If the equivalence claim were established, this would be a valuable contribution to explainable AI, offering a transparent, faithful symbolic surrogate for FNN decisions. The paper's strengths include a concrete step-by-step procedure, illustrated examples, explicit handling of convolutional and pooling layers, and a publicly available prototype. However, the central notion of equivalence is not formally defined or proven, and the method relies on an unspecified relevance threshold that can affect fidelity. The paper provides no quantitative validation that extracted trees reproduce network outputs on unseen inputs. As presented, the contribution is closer to a heuristic, per-instance explanation tool than to an equivalence-preserving transformation, so the stated significance is not yet supported.","major_comments":[{"comment":"The central claim of equivalence is not established. Section V states that for 'each of the input vectors to be examined' a decision path is derived and then combined into a tree; there is no argument that the finite tree obtained from a finite set of traced inputs reproduces the network's decision function on inputs outside that set. Since the input domain is continuous, a coverage or generalization argument is required to justify the term 'equivalent'. Without it, the method at best provides per-instance explanations, not an equivalent symbolic model.","section":"Abstract; Section V"},{"comment":"The pruning criterion is undefined. The method repeatedly discards weighted connections whose product v_j*w_j is not 'sufficiently large' or 'sufficiently relevant', but no threshold, justification, or error bound is given. A connection with a small individual contribution can be decisive for the output-layer argmax, especially when contributions accumulate; thus the pruned tree may disagree with the original network even on the traced input. The paper needs to specify how the threshold is chosen and to demonstrate, at minimum empirically, that the choice does not alter the final decision across a representative set of inputs.","section":"Section III.D, steps 1-3; Section III.E, steps 1-2"},{"comment":"No quantitative validation is provided. Section IV describes a prototype and Section V concludes that the prototype 'proved that the desired derivation of equivalent symbol-based decision models from feedforward neural networks can be efficiently implemented.' This only demonstrates implementation feasibility. The paper reports no experiments comparing the extracted decision tree's outputs with the original network on held-out data, no accuracy/fidelity metrics, and no sensitivity analysis for the relevance threshold. Without such measurements, the central claim of functionality preservation is unsubstantiated.","section":"Section IV; Section V"},{"comment":"The scalability claim to deeper networks is not supported. The abstract states that the method 'enable[s] scalability to deeper networks through iterative refinement of subpaths for each hidden layer', but Section III.E simply repeats the same undefined pruning step per layer. Recursively applying a heuristic that lacks an error bound does not transfer any equivalence guarantee to deeper networks. The paper provides no complexity analysis and no demonstration that iterative subpath refinement preserves the network's output mapping.","section":"Abstract; Section III.E"}],"minor_comments":[{"comment":"There are typographical errors in the running example: 'by is not considered' should likely be 'bv is not considered', and 'bu' is used without being defined as the input value of the first input neuron; the notation should be made consistent.","section":"Section III.D"},{"comment":"The text contains odd spacing artifacts such as 'K ERAS', 'T ENSOR FLOW', and 'J AVA'; these should be corrected to the standard software names during copyediting.","section":"Throughout"},{"comment":"The formal definition of a decision tree is difficult to parse; for example, the notation s×t×b_ijr → e_r and the statement 'm = number of initial edges' are ambiguous. Please clarify with a cleaner formalization, or rely on a standard reference.","section":"Section III.C"},{"comment":"The related-work discussion is very broad and does not compare the proposed extraction method with existing decompositional approaches (e.g., DeepRED, LORE, or LRP-based methods). Adding such a comparison would help position the contribution.","section":"Section II"},{"comment":"The prototype description focuses on implementation details (HDF5 parsing, JavaFX GUI, layer conversion) but does not describe how large networks perform; a brief runtime or scalability experiment would strengthen the claims in Section V.","section":"Section IV"}],"recommendation":"reject","confidential_remarks":"The paper's core claim of equivalence goes beyond what the method can support as presented. The authors may be able to reframe the contribution as a heuristic explanation tool with explicit caveats and a thorough empirical fidelity study; in that case a revised submission could be viable. As it stands, the lack of a formal equivalence proof, an undefined threshold, and the absence of quantitative evaluation are load-bearing gaps. I would also flag that the manuscript cites the authors' own prior work but does not critically differentiate the new method from it, and that the 'proof' language in Section V is likely to mislead readers about what has been demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a prototype paper with an unsupported equivalence claim. If read as a methods paper that proves the derived tree matches the network on all inputs, it fails. If read as a heuristic explanation tool with a nontrivial prototype and public code, there is something there.\n\nWhat the paper actually does: for each input vector to be examined, it propagates the vector through the network, prunes weighted connections that are not “sufficiently large” (never defined), derives a decision path, and then merges the paths into a tree. Section V says it in plain language: “For each of the input vectors to be examined,” derive a path, then combine. A tree built from finitely many traces can at best explain those inputs; it does not, by construction, extend to unseen inputs, and no coverage or fidelity argument is supplied. The pruning step can silently change the output-layer argmax, so the “equivalent” in the title is not earned. This is not a subtle statistical issue; it is the load-bearing claim.\n\nCredit where it is due: the prototype does real work. It reads Keras .h5 files, converts Conv1D/2D/3D, pooling, and flattening layers into feedforward form, and merges paths into a hierarchical tree. The code is public. The exposition of fillers and roles following Smolensky is systematic, and the step-by-step walkthroughs in Figures 11 and 12 are clear. As a demonstration of a possible extraction pipeline, it is a legitimate engineering contribution.\n\nThe soft spots beyond the missing proof: no comparison to the existing decision-tree extraction literature (Craven and Shavlik, among others), which has been around since the 1990s; no quantitative evaluation of fidelity on held-out inputs; and a free “sufficiently relevant” threshold with no guidance on setting it. The authors’ own 2018 paper [16] already introduced the basic idea, so this is an extension rather than a new result, and the paper should say so more explicitly.\n\nBottom line: I would not cite this as a method that yields equivalent symbolic decision models. I would keep it in mind as a prototype implementation and as a clear statement of one possible mapping from activations to symbolic structure. If the authors reframe it as a heuristic explanation tool, report fidelity statistics, and define the threshold, it could become useful. A serious editor should still send it to peer review — the prototype and code are real — but reviewers should reject the equivalence claim as currently stated.","headline":"The prototype is real but the equivalence claim is not established: the method traces finite input paths with an undefined pruning threshold, so the title promise fails as stated.","tokens_in":18067,"tokens_out":2925,"would_cite":false,"duration_ms":31704,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained feedforward nets can be rewritten as transparent decision trees.","keywords":["Artificial neural networks","connectionism","decision trees","explainable AI","symbolic AI models","symbolism","feedforward neural networks","model interpretability"],"falsifier":"Take a trained classifier, pick an input whose tree path is built with a given relevance cutoff, then search for a perturbation that only alters connections below the cutoff or inputs that change their weighted contributions, and watch whether the network's argmax changes while the extracted tree's decision stays fixed. Any single such divergence demonstrates the extracted tree is not equivalent to the network.","tokens_in":17174,"feed_emoji":"🌳","tokens_out":5395,"duration_ms":52349,"temperature":0.7,"pith_summary":"This paper tries to show that a trained feedforward neural network, including convolutional networks expressed in feedforward form, can be converted into a decision tree that reproduces the network's decisions and exposes how each decision was reached. The conversion works by tracing, from the output layer backward, which weighted connections actually matter for a given input, and representing the surviving input-to-neuron combinations as symbolic decisions along a tree path. The authors claim the resulting tree effectively captures the network's decision process and that the procedure scales to deeper networks by replacing each relevant edge with a subpath derived from the preceding hidden layer. If the claim holds, trained networks would gain a transparent, human-readable audit trail without retraining or altering the network itself.","feed_headline":"Trained feedforward nets can be rewritten as transparent decision trees","feed_subtitle":"A new derivation traces each relevant weighted connection so a network's choices can be audited branch by branch.","key_machinery":"The central machinery is the filler/role decomposition borrowed from tensor-product symbol representations: a symbol is written $s = \\sum_i f_i \\otimes r_i$, where $f_i$ is the filler and $r_i$ its role. In the network, each input neuron's activation value is a filler, the information it encodes is its role, and a neuron's input sum $\\sum_i v_i w_i$ is the combination rule that decides which (filler, role) tuples survive. The second load-bearing component is the recursive relevance cut: an edge is kept only if the product $v_j \\times w_{jk}$ is sufficiently large for the receiving neuron's activation, and this criterion is applied backward from the output layer to every hidden layer, producing subpaths that replace edges of an already-built path. Iterating this for many input vectors and merging the resulting paths yields the hierarchical decision tree.","core_discovery":"The paper's central claim is that a feedforward neural network's decision for a given input vector can be re-expressed as a decision path, and that the union of such paths over inputs forms a hierarchical decision tree that is functionally equivalent to the network. The derivation identifies the symbolic components of the network's distributed representation: input values play the role of fillers, their positions and the information they encode play the role of roles, and each neuron's weighted input sum is the combination rule that binds them. Starting from the output layer, the method keeps only neurons and connections whose weighted activation is sufficiently large to influence the output, then repeats this relevance check layer by layer so that every surviving edge of the tree is expanded into a subpath from the previous hidden layer. The result is presented as a step-by-step algorithmic procedure, with a prototype that reads a stored model from an .h5 file, converts convolutional and pooling layers into feedforward form, and renders the merged decision tree in a graphical interface.","pith_inferences":["If the relevance threshold is made adaptive or formally tied to the output margin, the method could double as a verification tool: the gap between the pruned tree and the true network output measures fidelity, and a user could demand whatever fidelity the application requires.","The same filler/role reading could produce other symbolic artifacts, such as rules or state machines, from the same extracted paths, potentially connecting this procedure to broader neuro-symbolic compilation.","A natural stress test is to compare tree and network on adversarial inputs engineered near decision boundaries, since these are exactly the cases where pruned small contributions are most likely to flip the argmax.","The paper's equivalence is per input-vector derivation; whether the merged tree is equivalent on unseen inputs is an empirical question that the authors do not address."],"forward_implications":["Every input decision of a feedforward network can be given a concrete branch-by-branch explanation in the vocabulary of the original input features.","The extraction runs on a stored trained model and produces a static tree, so the audit object is independent of the training framework.","Because convolutional layers are first flattened into feedforward form, the same derivation covers convolutional classifiers, not only fully connected networks.","The method is modular per hidden layer: adding depth is handled by replacing edges with subpaths, so deeper networks do not require a new algorithm."],"supporting_citations":[{"why":"Introduces the filler/role tensor-product representation that anchors the symbolic bridge.","marker":"[36]"},{"why":"Frames the symbolic-versus-connectionist distinction that the derivation is meant to bridge.","marker":"[13]"},{"why":"Provides the deep-learning and convolution concepts the method depends on, including input coding.","marker":"[18]"},{"why":"Supplies the decision-tree formalism and learning context used for the target model.","marker":"[21]"},{"why":"Supports treating convolutional layers as feedforward structures with hidden cells.","marker":"[34]"},{"why":"Prior work on creating equivalent symbolic AI models that this method extends toward tree extraction.","marker":"[16]"},{"why":"Earlier state-machine extraction from feedforward networks that motivates the tree-based counterpart.","marker":"[39]"},{"why":"Supplies the parser used to read stored model files in the prototype.","marker":"[42]"},{"why":"Provides the runtime and convolution semantics the prototype emulates.","marker":"[14]"}],"fun_headline_variants":["Neural net decisions become audit-ready decision trees","Extracting symbolic trees from feedforward networks","Rewriting neural nets as transparent decision paths","From neural nets to decision trees: a traceable path","Converting feedforward nets into interpretable trees"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The extraction assumes that weighted connections whose product $v_j \\times w_{jk}$ is not judged sufficiently large can be discarded without ever changing the network's final decision; if small contributions collectively flip the output layer's argmax, the extracted tree diverges from the network.","fun_headline_variants_meta":{"raw":{"variants":["Neural net decisions become audit-ready decision trees","Extracting symbolic trees from feedforward networks","Rewriting neural nets as transparent decision paths","From neural nets to decision trees: a traceable path","Converting feedforward nets into interpretable trees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2943,"prompt_tokens":982,"completion_tokens":1961,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1891}},"tokens_in":598,"tokens_out":1961,"duration_ms":13388,"temperature":1.0,"reasoning_tokens":1891,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:31:09.823017+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained classifier, pick an input whose tree path is built with a given relevance cutoff, then search for a perturbation that only alters connections below the cutoff or inputs that change their weighted contributions, and watch whether the network's argmax changes while the extracted tree's decision stays fixed. Any single such divergence demonstrates the extracted tree is not equivalent to the network.","supporting_citations":[{"cited_title":"Smolensky and G","cited_arxiv_id":null,"evidence_quote":"Introduces the filler/role tensor-product representation that anchors the symbolic bridge."},{"cited_title":"Logical vs. analogical or symbolic vs. connectionist or neat vs. scruffy,","cited_arxiv_id":null,"evidence_quote":"Frames the symbolic-versus-connectionist distinction that the derivation is meant to bridge."},{"cited_title":"G ´eron, Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow","cited_arxiv_id":null,"evidence_quote":"Supplies the decision-tree formalism and learning context used for the target model."},{"cited_title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,","cited_arxiv_id":null,"evidence_quote":"Supports treating convolutional layers as feedforward structures with hidden cells."},{"cited_title":"Understanding neural network decisions by creating equivalent symbolic AI models,","cited_arxiv_id":null,"evidence_quote":"Prior work on creating equivalent symbolic AI models that this method extends toward tree extraction."},{"cited_title":"Extracting state machines from feedforward neural networks,","cited_arxiv_id":null,"evidence_quote":"Earlier state-machine extraction from feedforward networks that motivates the tree-based counterpart."},{"cited_title":"Mudd, jHDF - Pure Java HDF5 library , 2015, version 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the parser used to read stored model files in the prototype."},{"cited_title":"Abadi et al","cited_arxiv_id":null,"evidence_quote":"Provides the runtime and convolution semantics the prototype emulates."}],"review_version":1}