{"id":"46ed2cbd-e6b8-4bde-b332-d0cbb84e61a7","arxiv_id":"2507.22665","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper presents a cluster-based visualization system with a new tree distance metric, plus Feature Plot and Rule Plot views, to make random forests more interpretable.","lead":"A group of researchers built a tool that groups the many decision trees inside a random forest into clusters of similar trees, so a person can understand the model without reading every tree. They also designed two new charts, the Feature Plot and the Rule Plot, and tested them with a case study and a small user study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) defines an asymmetric tree distance: it averages over one tree's rules only, so d(T1,T2) can differ from d(T2,T1); MDS and complete-linkage clustering consume this matrix without any stated symmetrization, making the clustering and projection that the central claim rests on under-specified.","rationale":"The reader's weakest-assumption analysis identifies a genuine, load-bearing gap: Eq. (1) as written is asymmetric for trees with different numbers of leaves, and no symmetrization is described before the dissimilarity matrix is used for complete-linkage clustering and MDS. I agree that this is the most important correctness risk. The central claim about interpretable clusters is only meaningful if the clusters and the projection are well-defined from the stated method; asymmetry makes them dependent on an unspecified convention. A quick check on the actual datasets can settle whether the asymmetry is large enough to change cluster structure. The small user study (n=2) and lack of quantitative baselines are real limitations, but they are secondary to the formal under-specification of the core distance. The reader's CONDITIONAL verdict remains appropriate: the method is plausible and the visual design is described in detail, but the published description needs an explicit resolution of the asymmetry or evidence that it is immaterial.","tokens_in":20136,"tokens_out":5905,"duration_ms":79990,"concrete_test":"Independently re-implement Eqs. (1)-(3) and train the same sklearn RandomForest on Glass (and Penguin) as in Section 6. Compute the full pairwise matrix D and check max_{i,j} |D_ij - D_ji|. If this maximum exceeds a small tolerance, symmetrize with the most natural convention, e.g., (D + D^T)/2, then rerun complete linkage, dynamic hybrid cut, and MDS exactly as described, and compare cluster memberships, medoids, and the resulting Feature Plot/Rule Plot summaries against the unsymmetrized pipeline. If clusters or representative trees change materially, the paper must specify and justify its symmetrization step; if they do not, the asymmetry has no practical effect for these datasets and the concern is resolved empirically.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that clusters of decision trees, visualized through the Feature Plot and Rule Plot, let users interpret a random forest without per-tree inspection or an oversimplified single-tree summary. That claim requires the clusters to be well-defined objects derived from a meaningful pairwise similarity. The paper's only formal definition of similarity is Eq. (1): d(T1,T2) = 1/|R(T1)| * sum over ri in R(T1) of min over rj in R(T2) of dR(ri,rj). Because the average is over rules of T1 only, the quantity is not symmetric when |R(T1)| differs from |R(T2)|. This is not a corner case: random forests trained with scikit-learn on the Glass dataset used in the evaluation produce trees with different numbers of leaves, so asymmetry of the same order as legitimate distance differences is expected. The paper then feeds this pairwise matrix into complete-linkage hierarchical clustering and MDS (Sections 4 and 5.1) without ever stating that d was symmetrized or that a directed variant was handled. Complete-linkage and MDS are defined for symmetric dissimilarities; applying them to an asymmetric matrix makes the dendrogram, the dynamic hybrid cut, cluster membership, and the MDS projection depend on an unstated ordering or storage convention. The representative medoid and all subsequent Rule Plot mappings are computed from these clusters, so the interpretation workflow inherits that ambiguity. The paper also calls Eq. (1) a distance metric; even after symmetrization it is not shown to satisfy metric axioms, though non-metric dissimilarities can still be usable in clustering and MDS if applied correctly. The load-bearing issue is therefore not that the distance is non-metric, but that the asymmetry is unaddressed and the downstream steps require symmetry.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a visual analytics system for interpreting random forests by clustering decision trees and visualizing the clusters with two new views, the Feature Plot and the Rule Plot, together with a detailed per-tree view. The proposed tree dissimilarity in Eq. (1) averages, over each rule of one tree, the minimum rule distance to same-class rules of the other tree, where the rule distance combines prediction agreement and per-feature interval overlap. The trees are then clustered with complete-linkage hierarchical clustering and dynamic hybrid cut, projected with MDS, and summarized by a medoid representative. The system is evaluated through a case study on the Glass dataset and a small user study with two participants, with SUS and ICE-T scores. The paper argues that this cluster-based approach is a middle ground between inspecting every tree and collapsing the forest into a single summary tree.","tokens_in":20494,"tokens_out":4769,"duration_ms":59022,"significance":"If the clustering step were well-defined, the approach would be a useful addition to random-forest interpretability: it preserves structural diversity within the forest, visualizes feature usage and decision rules at a cluster level, and the task elicitation in Section 3 grounds the design in real user needs. The Feature Plot and Rule Plot are thoughtful designs, and the Glass case study demonstrates nontrivial insights that go beyond single-tree summaries. However, the central clustering and projection pipeline rests on a pairwise dissimilarity that is asymmetric as defined in Eq. (1), while the paper feeds it into symmetric-input methods without any stated symmetrization. The current manuscript therefore does not yet establish a sound basis for the clusters, projections, and medoid representatives on which the entire interpretation workflow depends.","major_comments":[{"comment":"The dissimilarity in Eq. (1) is not symmetric: d(T1,T2) = (1/|R(T1)|) * sum_{ri in R(T1)} min_{rj in R(T2)} dR(ri,rj) averages over the rules of T1 only, while d(T2,T1) averages over the rules of T2. The two values generally differ, even when the trees have the same number of leaves, because min-assignment is not symmetric. Sections 4 and 5.1 then feed this matrix into complete-linkage hierarchical clustering and MDS without describing any symmetrization step, although both methods require symmetric dissimilarities. This is not a corner case: decision trees in a scikit-learn random forest on the Glass dataset vary in size. Consequently, the dendrogram, the dynamic hybrid cut, the cluster memberships, the MDS projection, and the medoid representatives are under-specified and may depend on an arbitrary orientation or storage convention. The authors should either explicitly symmetrize the matrix (for example by taking the maximum, minimum, or average of the two directed values) and justify the choice, or explain how they handled the directed matrix.","section":"Section 4, Eq. (1)"},{"comment":"The paper repeatedly calls Eq. (1) a 'distance metric' (Section 4, the abstract, and the list of contributions). As defined it is not a metric: it is asymmetric and, even after symmetrization, it is not established that it satisfies the triangle inequality. I recommend using the term 'dissimilarity measure' or 'distance measure' throughout, and reserving 'metric' for a quantity that has been verified to satisfy the metric axioms.","section":"Section 4 and contribution 1"},{"comment":"The user study includes only two participants, and the reported SUS and ICE-T scores are given as raw values without statistical analysis or a comparison to a baseline. In particular, the sentence in Section 8 that the 'case and user study demonstrate the effectiveness of our approach' is stronger than the evidence supports. The case study is informative, but the user-study component should either be framed as a pilot usability probe or be supplemented with a more systematic assessment, such as task-completion measurements, a formal comparison with an existing random-forest visualization or distance method, or a larger participant sample.","section":"Section 6.2"}],"minor_comments":[{"comment":"There is a typo in the first paragraph: 'the set of of rules' should be 'the set of rules'.","section":"Section 4"},{"comment":"The text 'includesemanticas well asstructural components' is missing spaces in the author version; please correct the formatting in the final version.","section":"Section 4"},{"comment":"The statement that MDS 'truthfully reflects the relative distances between the trees' is too strong; MDS only approximates pairwise distances and the approximation error (stress) should be mentioned or shown.","section":"Section 5.1"},{"comment":"The grammar in 'the SUS survey results in a score of (70 and 82.5)' should be revised, for example to 'the SUS survey resulted in scores of 70 and 82.5'.","section":"Section 6.2"},{"comment":"The choice of MDS over other dimensionality-reduction methods is motivated only briefly; a sentence on why the projection is faithful enough for cluster interpretation would help readers assess the influence of projection stress on the cluster hulls.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—quick take: a real systems paper with a useful new idea for random forest interpretation, but the math has a load-bearing gap I'd want fixed before relying on the clustering results. The Feature Plot and Rule Plot are genuinely new aggregate visualizations, and coupling them with a prediction-aware rule distance is the right instinct. The Glass case study feels credible, and the task elicitation from practitioners is a nice touch.\n\nThe soft spot is Eq. (1). As written, d(T1,T2) averages over the rules of T1 only, so it is asymmetric whenever the two trees have different numbers of rules—which is the normal case for scikit-learn random forests. That matrix is fed to complete-linkage clustering and MDS, both of which assume a symmetric dissimilarity, and the paper never describes symmetrizing. That makes the dendrogram, the dynamic hybrid cut, and the 2D projection depend on an unstated ordering or storage convention. This is an easy fix—average the two directed distances or use a symmetrized version—but it needs to be written down. Also, calling d a 'distance metric' oversells it; even after symmetrization there is no triangle inequality proof, though non-metric dissimilarities are usable in clustering and MDS if acknowledged.\n\nThe evaluation is thin: user study with n=2, and no quantitative comparison against the earlier tree distances they survey. The authors are honest about that in Section 7, which helps. The case study is detailed but anecdotal.\n\nI would not cite this in my own work in the next year, but I would bring it to a reading group: the visualization designs are worth dissecting and the asymmetry flaw is instructive. It deserves a serious peer review—send it to a capable referee with a request to fix the distance definition and add at least a synthetic sanity check on clustering.","headline":"Useful RF visualization system with a genuinely new tree distance, but the distance is asymmetric as written and the evaluation is too thin to fully back the claims.","tokens_in":21026,"tokens_out":3120,"would_cite":false,"duration_ms":33076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that random forests should be interpreted through clusters of similar decision trees, grouped by both split rules and predictions, rather than through individual trees or a single simplified summary.","keywords":["random forest","decision tree","tree clustering","distance metric","explainable AI","visual analytics","feature plot","rule plot"],"falsifier":"Take two trees from a trained random forest with different numbers of leaves, compute $d(T_1,T_2)$ and $d(T_2,T_1)$ using Eq. (1); if the values differ, recompute the full distance matrix with the symmetrized average and re-run the complete-linkage clustering and MDS. If cluster memberships change materially, the reported cluster structure is an artifact of the one-sided averaging.","tokens_in":19945,"feed_emoji":"🌲","tokens_out":9176,"duration_ms":99778,"temperature":0.7,"pith_summary":"This paper claims that the right unit of interpretation for a random forest is neither the individual tree nor one aggregated summary, but clusters of trees that make decisions in similar ways. To form these clusters it introduces a distance metric that compares the decision rules (paths with split thresholds) of two trees and only matches rules that predict the same class. The resulting clusters are visualized with two new idioms: the Feature Plot, which shows which features are split at each depth, and the Rule Plot, which maps all rules in a cluster onto the cluster's representative tree. The authors show through a Glass-dataset case study and a small user study that this middle-ground view supports tasks like judging model reliability, finding misclassified classes, and identifying important features. The motivation is that a forest's structure is multi-level, so a single summary hides the variability that clustering preserves.","feed_headline":"Group similar trees to make random forests interpretable","feed_subtitle":"New rule-and-prediction distance plus two plots lets users inspect tree subgroups instead of one oversimplified summary.","key_machinery":"The load-bearing object is the rule-interval distance metric defined in Eqs. (1)--(3), which represents each tree as a set of root-to-leaf rules, each rule as a per-feature interval, and compares two trees by averaging the distance from each rule of the first tree to its closest same-class rule in the second. This metric is what makes the cluster-level visualizations possible: it produces the pairwise dissimilarity matrix used for complete-linkage hierarchical clustering with dynamic hybrid cut, the MDS projection in the Sidebar, and the mapping of all cluster rules onto the medoid tree in the Rule Plot. The Feature Plot complements it by aggregating feature frequencies per depth, adding a topological view that the rule distance deliberately discards.","core_discovery":"The central discovery is that decision trees in a random forest can be meaningfully grouped by a distance that combines semantics and structure: $d(T_1,T_2)$ is the average, over all rules in $T_1$, of the distance to the closest rule in $T_2$ that yields the same class, where rule distance is the average per-feature interval dissimilarity $d_f = 1 - \\frac{|I_1 \\cap I_2|}{\\max(b_1-a_1, b_2-a_2)}$. Because this ignores the order of splits, two trees with the same decision logic but different node order get distance zero, while methods based only on split variables would treat them as different. The distance feeds a complete-linkage hierarchical clustering with dynamic hybrid cuts and an MDS projection, and each cluster is summarized by its medoid tree. The Feature Plot shows feature usage per tree level; the Rule Plot maps every rule of every tree in a cluster to the closest same-class rule of the medoid and visualizes the aggregated feature intervals and classification outcomes. The paper reports that this cluster view, rather than a single summary tree, lets users see how well parts of the data are covered and which tree subgroups drive classifications.","pith_inferences":["A testable extension the paper leaves open: symmetrizing the distance as $(d(T_1,T_2)+d(T_2,T_1))/2$ and re-running the clustering would show whether the reported cluster structure depends on the arbitrary ordering of trees in the forest.","Because the rule distance only needs root-to-leaf intervals and class labels, it could plausibly be lifted to boosted tree ensembles or rule lists, a generalization the paper does not claim.","A controlled study with a single-summary baseline and an all-trees baseline would be needed to confirm that cluster-level views actually reduce cognitive load; the reported user study is qualitative and has no control condition.","The color-based feature encoding limits scaling to datasets with many features; aggregating or selecting features could extend the system, at the cost of the fine-grained inspection the plots currently provide."],"forward_implications":["Users can locate subgroups of trees responsible for specific classes or misclassifications and judge model confidence from cluster sizes and feature patterns.","The Feature Plot gives feature importance a topological reading: a feature split early and often across a cluster is more discriminative than one that appears only deep in a few trees.","The Rule Plot lets analysts compare rule groups across clusters, see which feature ranges separate classes, and filter to specific feature values or misclassifications while keeping the aggregation stable.","The approach scales to large numbers of trees because each tree is reduced to its rule set and only representative trees need deep inspection.","The same Feature and Rule Plots can visualize a single decision tree or any classifier that emits decision paths or rules, not only random forests."],"supporting_citations":[{"why":"Defines random forests as an ensemble of decision trees, the model this paper interprets.","marker":"[10]"},{"why":"Introduces earlier decision-tree distance metrics that the new rule-based metric extends and compares against.","marker":"[16]"},{"why":"Proposes representative-tree extraction from ensembles, which the paper builds on for medoid selection.","marker":"[7]"},{"why":"Presents a split-variable tree distance that the paper's interval-based distance is designed to improve upon.","marker":"[28]"},{"why":"Provides the dynamic hybrid cut algorithm used to assign trees to clusters from the dendrogram.","marker":"[29]"},{"why":"Supplies the random-forest implementation used to train models on uploaded datasets.","marker":"[43]"},{"why":"Supplies the Glass dataset on which the case study demonstrates the visualization tasks.","marker":"[21]"},{"why":"Supplies the Penguin dataset used in the user study's training phase.","marker":"[26]"},{"why":"Contributes the enhanced node-link decision-tree visualization that the detailed tree view adapts.","marker":"[56]"},{"why":"Describes a prior rule-extraction visualization system that motivates the Rule Plot design.","marker":"[14]"}],"fun_headline_variants":["Cluster trees by logic and predictions for clearer insight","Group similar trees to see forest's inner logic","New distance metric and plots reveal tree clusters","Visualize random forests via clustered subgroups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The grouping assumes that the similarity between two trees does not depend on which tree is considered first, but the formula as written averages over only the first tree's rules, so swapping the trees can change the distance.","fun_headline_variants_meta":{"raw":{"variants":["Cluster trees by logic and predictions for clearer insight","Group similar trees to see forest's inner logic","New distance metric and plots reveal tree clusters","Visualize random forests via clustered subgroups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1942,"prompt_tokens":983,"completion_tokens":959,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":903}},"tokens_in":599,"tokens_out":959,"duration_ms":8784,"temperature":1.0,"reasoning_tokens":903,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:24:16.233112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two trees from a trained random forest with different numbers of leaves, compute $d(T_1,T_2)$ and $d(T_2,T_1)$ using Eq. (1); if the values differ, recompute the full distance matrix with the symmetrized average and re-run the complete-linkage clustering and MDS. If cluster memberships change materially, the reported cluster structure is an artifact of the one-sided averaging.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines random forests as an ensemble of decision trees, the model this paper interprets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces earlier decision-tree distance metrics that the new rule-based metric extends and compares against."},{"cited_title":"Banerjee, Y","cited_arxiv_id":null,"evidence_quote":"Proposes representative-tree extraction from ensembles, which the paper builds on for medoid selection."},{"cited_title":"Laabs, A","cited_arxiv_id":null,"evidence_quote":"Presents a split-variable tree distance that the paper's interval-based distance is designed to improve upon."},{"cited_title":"Chatzimparmpas, R","cited_arxiv_id":null,"evidence_quote":"Describes a prior rule-extraction visualization system that motivates the Rule Plot design."}],"review_version":1}