{"id":"c01d2fd8-5e37-4bb6-bb3e-24224566fdcc","arxiv_id":"2507.22556","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VAR is a visual analytics application that uses radial basis function interpolation to create heatmaps and scatter plots for horizontal comparison of models in a Rashomon set.","lead":"A new visual analytics tool, VAR, plots many equally accurate machine learning models in a shared feature space using radial basis function interpolation, showing them as heatmaps and dot plots. The goal is to help developers choose the model that behaves best under specific conditions, such as missing data or fairness constraints.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RBF visual mapping in §3.2 is unvalidated as a faithful representation of model similarity; without a fidelity check, VAR's heatmap and dot patterns may mislead rather than inform model selection.","rationale":"The reader's weakest assumption identifies exactly the load-bearing risk: the RBF interpolation is arbitrary without validation. My reading supports this and adds two concrete details: the interpolation algorithm is under-specified to the point of non-reproducibility, and §3.3 contradicts §3.2 about whether interpolation is used at all. The user study cannot rescue the claim because it has no ground-truth measure of whether visual similarity corresponds to true model quality. This does not require a different verdict from the reader's CONDITIONAL: the paper could become acceptable with a precise algorithm, released artifacts, and a quantitative fidelity check, but in its current form the central visual-analysis claim is unsubstantiated.","tokens_in":14272,"tokens_out":2952,"duration_ms":43881,"concrete_test":"Reconstruct the RBF pipeline exactly as specified in §3.2–3.3 on a released or regenerated TreeFarms FICO Rashomon set (152 models). Compute pairwise distances between models in the original performance/feature space (e.g., accuracy, F1, leaf count, feature usage) and pairwise visual distances from the resulting RBF heatmap coordinates and color values. Compute the rank correlation (Kendall tau) between these distance matrices. If the correlation is not significantly positive—or is comparable to a random embedding—the visual patterns are artifacts and the central claim fails. As a second check, permute model labels in the color field: if users still report the same insights, the color encoding is not informative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that VAR lets developers identify an optimal model and understand Rashomon-set characteristics—depends on the RBF mapping preserving meaningful structure. Section 3.2 defines the kernel φ(r)=r·log(1+r^0.5)/(1+r^0.1) and asserts a balance between local adaptability and global smoothness, but gives no argument, validation, or release of the interpolation pipeline. The mapping from model performance vectors to the visual x/y axes and color field is never specified: the input dimension, the set of centers, the choice of RBF values versus solving an interpolation system, and the role of the selected feature axes are all absent. Section 3.3 adds an internal inconsistency: it says the RBF-dot mode 'avoids interpolation,' contradicting §3.2's claim that RBF interpolation underlies the visualization. Appendix 8 shows that 16 kernel choices produce visibly different heatmaps but provides no criterion for selecting a faithful kernel or for deciding what the colors mean. Because visual proximity and color gradients are not tied to any quantitative model-similarity metric, the claimed ability to 'identify the optimal model under specific conditions' is unsupported; a misleading interpolation could actively steer users toward worse models. The user evaluation in §5 is qualitative, with five experts and no ground-truth selection task, so it cannot detect this failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VAR, a web-based visual analytics tool for horizontally comparing models within a Rashomon set of machine learning models. VAR combines radial basis function (RBF) based heatmaps and scatter plots to visualize model performance across user-selected feature dimensions. The authors motivate the tool through interviews with ML developers, describe two visualization modes (RBF-heatmap and RBF-dot), and present case studies on FICO and COMPAS datasets along with qualitative feedback from five expert users. The central claim is that VAR enables developers to identify the optimal model under specific conditions and to better understand the overall characteristics of a Rashomon set.","tokens_in":14554,"tokens_out":2849,"duration_ms":34140,"significance":"The problem addressed is real and timely: as Rashomon-set enumeration methods mature, practitioners need tools to compare many near-optimal models horizontally, beyond the structural views offered by TimberTrek or GamChanger. If the RBF-based visual mapping were rigorously specified and validated, VAR could be a useful addition to the interpretable ML toolbox. The paper's strengths include a clear problem motivation grounded in interviews, the use of two public datasets, and an iterative design process that demonstrably incorporated expert feedback. However, the paper ships no code, no machine-checked derivation, and no quantitative validation; its effectiveness rests on anecdotal expert comments. The central methodological step, the RBF interpolation, is not described precisely enough to reproduce, and no fidelity check connects visual patterns to actual model similarities or performance differences. These issues currently prevent the paper from substantiating its stated contribution.","major_comments":[{"comment":"The RBF interpolation procedure is critically underspecified. The paper defines the kernel φ(r)=r·log(1+r^0.5)/(1+r^0.1) and states that model performance vectors are mapped into a continuous space, but it does not specify how the input vector x is constructed from model performance values, what the centers c_i are, whether the visualization solves an interpolation system or simply evaluates the kernel at grid points, how the two displayed dimensions are chosen, or how the color field is computed from kernel values. Without this algorithm, the heatmaps and dot plots cannot be reproduced, and the claim that users can trust the visual patterns to identify optimal models is not auditable.","section":"§3.3"},{"comment":"There is an internal contradiction about whether interpolation is used. Section 3.2 states that RBF interpolation underlies the mapping of model performance into the visual space, and Section 3.3 says 'both visualization modes rely on RBF interpolation.' A few paragraphs later, however, Section 3.3 says the RBF-dot mode 'avoids interpolation' and is 'more precise for examining actual data points.' This is inconsistent, especially because the same subsection says the dot color is 'calculated by the RBF based on its position.' The authors must clarify exactly which quantities are interpolated in each mode and reconcile these statements.","section":"§3.3"},{"comment":"The kernel-function comparison in Appendix 8 shows that 16 different kernel choices produce visibly different heatmaps from the same data, yet the paper offers no criterion for selecting a faithful kernel and no ground truth for what the visual colors should represent. If the visualizations change substantially with the kernel choice, then the claim in Section 3.2 that the chosen kernel 'maintains a balance between local adaptability and global smoothness' is not enough to guarantee that the displayed patterns reflect true model similarities or performance gradients. The authors should provide a validation, for example by showing that spatial proximity or color values in the chosen visualization correlate with a quantitative model-similarity metric, or by performing a user study with known ground-truth model rankings.","section":"Appendix 8"},{"comment":"The user evaluation in Section 5 is purely qualitative: five experts give anecdotal comments, with no ground-truth model-selection task, no pre-registered success criteria, and no quantitative comparison against a baseline method. The paper's abstract claims that VAR enables developers to 'identify the optimal model under specific conditions,' but the evaluation does not measure whether experts' selections matched the true optimal model or whether VAR outperformed numerical analysis in any measurable way. The effectiveness claim therefore remains unsupported. A structured evaluation (e.g., tasks where the correct selection is known, with accuracy and time measurements) is needed.","section":"§5"}],"minor_comments":[{"comment":"The abstract states that VAR combines heatmaps and scatter plots but does not mention the RBF method that is central to the paper; including it would better set reader expectations.","section":"Abstract and Title"},{"comment":"The caption for Figure 1 says (a) is 'Comparison of the performance of 152 models on the test set' but then says 'The color represents the train loss.' This is confusing; clarify whether the displayed metric is train or test, and similarly for Figure 9.","section":"Figure 1"},{"comment":"The phrase 'focusing primarily on filtering and comparing existing model results' is a bit vague; consider rephrasing to describe the concrete interaction (point size, color, axis selection) more precisely.","section":"§3.3"},{"comment":"The text says 'we observed that it is commonly known that shallower decision tree models represent simpler tree structures and should have lower prediction accuracy,' but the subsequent finding is that shallower trees can still perform well on the test set. The wording could be tightened to state the conventional expectation as a hypothesis rather than a fact.","section":"§4.1"},{"comment":"The experts are described as coming from 'O University' and 'Z Institution'; for a journal publication, the authors should either disclose these institutions or use more conventional anonymization, and should confirm that the experts consented to being quoted.","section":"§5.1"},{"comment":"Several references are incomplete or inconsistently formatted, e.g., reference [41] lists author names in a nonstandard order and lacks a venue, and reference [31] is given as an arXiv ID without year or DOI. Please clean up the bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This paper is a single-author preprint that reads more like a workshop report than a completed journal article. The central methodological step (RBF interpolation) is not specified, the effectiveness claim is supported only by anecdotal feedback, and there is no code or data release. The issues are serious but fixable within the manuscript's scope: the authors can specify the algorithm, validate the visual mapping, and redesign the user study with measurable outcomes. If these are not substantially addressed, the paper would not meet the standard for a journal in visualization or machine learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a genuine but incremental systems paper. It builds a web tool that combines RBF-interpolated heatmaps with scatter plots to compare decision trees in a Rashomon set horizontally, a real gap that Dong and Rudin's variable importance clouds only partly cover. The paper is honest about its limits; it repeatedly says the findings are not the focus and the tool is early stage.\n\nWhat's good: the design process is iterative, with 18 meetings and expert feedback; the case studies on FICO and COMPAS show how the tool is used; the kernel comparison appendix at least acknowledges that kernel choice matters; and the authors give full parameter settings for TreeFarms. The writing is clear about what the tool does.\n\nThe load-bearing spot is Section 3.2. The RBF mapping from performance vectors to visual space is never defined: no centers, no vector construction, no interpolation system, no error analysis. The kernel φ(r)=r·log(1+r^0.5)/(1+r^0.1) is justified only by vague 'balance' language. Section 3.3 then says the RBF-dot mode 'avoids interpolation,' which contradicts Section 3.2's claim that both modes rely on RBF interpolation. Appendix 8 shows 16 kernels produce visibly different heatmaps but gives no criterion for choosing a faithful one. So the stress-test's worry is real: without a fidelity check, the colors and spatial proximity could mislead rather than inform model selection. The user evaluation is five experts, qualitative, no ground-truth selection task; it can show usability but not correctness of the mapping. No code or data are released, which makes the interpolation pipeline uncheckable.\n\nNone of this makes the paper worthless. The gap is real, the design rationale is sensible, and the limitations are stated openly. A revised version with a precise RBF specification, a sanity check that visual proximity tracks a quantified model-similarity metric, and released code would substantiate the claims. If the target venue is a visualization conference, this could be a late-breaking work or a short paper; for a more rigorous journal it needs more.\n\nMy call: send to peer review rather than desk reject. A serious referee can push for the missing specification and a small quantitative evaluation. It deserves a round of revision, not rejection on the merits.","headline":"A genuine but under-specified visualization tool for Rashomon sets; worth a referee, but the RBF mapping needs explicit definition and fidelity validation.","tokens_in":15035,"tokens_out":2408,"would_cite":false,"duration_ms":27883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes VAR, a visual analytics tool that maps each model's performance vector into a continuous RBF-interpolated space, rendered as heatmaps and scatter plots, so that developers can compare and select models across a Rashomon…","keywords":["Rashomon set","visual analytics","radial basis functions","decision trees","model selection","missing data","interpretable machine learning"],"falsifier":"Take the 152-model credit-risk Rashomon set used in the paper, measure each model's true performance on a specific missing-data pattern, and compare those values with the heatmap color at the corresponding positions; if the interpolated surface misranks models relative to the measured values on substantial regions, the visualization is not faithful. A complementary check is a controlled trial where developers choose a target model from the same set using VAR versus using the raw numerical table, with the best model for each condition known in advance.","tokens_in":14079,"feed_emoji":"📊","tokens_out":9436,"duration_ms":103501,"temperature":0.7,"pith_summary":"VAR is a visual analytics tool for the Rashomon set, the collection of machine-learning models that score nearly equally well while making decisions through different structures. The paper claims that by mapping each model's performance across chosen features through radial-basis-function interpolation, then rendering the result as heatmaps and scatter plots, VAR allows developers to compare many models horizontally rather than inspecting one model's structure at a time. The intended payoff is the ability to pick the model that fits a specific deployment condition, such as a particular pattern of missing data, while also seeing global patterns of diversity and robustness in the whole set. The paper demonstrates the approach on two public tabular datasets and reports feedback from five expert users comparing VAR with their usual numerical-table workflow.","feed_headline":"Heatmaps make near-identical models easy to compare and select","feed_subtitle":"VAR combines heatmaps and scatter plots so developers can pick the model that fits a specific condition.","key_machinery":"The object that carries the argument is the radial-basis-function interpolation map $\\varphi(r) = \\frac{r \\log(1 + r^{0.5})}{1 + r^{0.1}}$, applied to a vector of each model's performance values with the models themselves as centers. The interpolation turns a discrete set of model metrics into a continuous surface, which the heatmap mode colors by predicted performance and the dot mode overlays with individual model points colored by a third metric. This is what converts the abstract claim that similar-performance models can be compared across features into a picture a developer can read for clusters, gradients, and outliers.","core_discovery":"The central claim is that an RBF-generated visual field makes the Rashomon set navigable for model selection. VAR represents each model by a vector of performance values over selected features, interpolates these vectors with a custom radial basis function, and presents the result in two complementary modes: a heatmap that colors the interpolated surface for global trends, and a dot plot in which each model is a point and a third metric is encoded in color. In the case studies the tool surfaces patterns such as shallow decision trees with few leaves matching deeper trees on test performance, and expert users report that they can screen models more quickly than with numerical tables. The contribution is a new way of seeing an existing model collection rather than a new model-building or optimization method.","pith_inferences":["Not stated in the paper, but implied by its design: the RBF machinery is model-agnostic, so any model family that yields a vector of per-condition performance scores could be fed into the same heatmap-and-dot view, including rule lists, additive models, or neural networks with feature-attribution vectors.","Not stated in the paper: the appendix's comparison of 16 kernel functions is visual only, and a numeric interpolation-error benchmark against held-out model performance would settle whether the custom kernel genuinely adds value.","The paper reports the 'shallow trees can match deep trees' pattern on two public datasets; a reader should treat it as illustrative rather than a general law."],"forward_implications":["Rashomon-set analysis becomes a set-level task: a developer can filter hundreds of near-optimal trees by the feature conditions that matter for deployment instead of inspecting one model at a time.","The two modes give complementary views, with the heatmap surfacing broad performance regions and the dot mode isolating individual models, so selection can proceed from landscape to specific point.","The case studies indicate that simpler decision trees with fewer leaves can remain competitive on test accuracy and F1 inside a Rashomon set, which points to cheaper and more interpretable deployment candidates.","Because the motivating scenario is missing data, VAR offers a way to ask which models stay valid when particular features are absent, a question that single-number accuracy tables answer poorly."],"supporting_citations":[{"why":"introduces the Rashomon set concept that defines the object VAR is designed to analyze.","marker":"[1]"},{"why":"provides the structural decision-tree exploration interface that VAR positions itself against for horizontal comparison.","marker":"[2]"},{"why":"proposes the scatter-plot-per-model variable-importance visualization that VAR extends with a color-encoded third dimension.","marker":"[19]"},{"why":"supplies the radial-basis-function visualization approach that VAR adapts to model performance comparison.","marker":"[31]"},{"why":"provides the sparse decision-tree optimization method whose threshold guessing is used to binarize the case-study data.","marker":"[41]"},{"why":"is cited as the source of the model-generation method that produced the Rashomon sets used in the case studies.","marker":"[45]"}],"fun_headline_variants":["Heatmap-dotplot pair reveals best ML model in Rashomon set","Rashomon set model selection made visual with VAR","Compare near-identical models fast with heatmaps and dots","VAR tool turns model performance into navigable visual map","Visual analysis spots optimal model among very similar ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guidance VAR gives is only as trustworthy as the smoothing map that turns model scores into colors and positions: if genuinely different models end up visually close, or genuinely similar models end up far apart, the tool would mislead rather than inform selection.","fun_headline_variants_meta":{"raw":{"variants":["Heatmap-dotplot pair reveals best ML model in Rashomon set","Rashomon set model selection made visual with VAR","Compare near-identical models fast with heatmaps and dots","VAR tool turns model performance into navigable visual map","Visual analysis spots optimal model among very similar ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1247,"prompt_tokens":827,"completion_tokens":420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":443,"tokens_out":420,"duration_ms":5582,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:31:42.932270+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 152-model credit-risk Rashomon set used in the paper, measure each model's true performance on a specific missing-data pattern, and compare those values with the heatmap color at the corresponding positions; if the interpolated surface misranks models relative to the measured values on substantial regions, the visualization is not faithful. A complementary check is a controlled trial where developers choose a target model from the same set using VAR versus using the raw numerical table, with the best model for each condition known in advance.","supporting_citations":[{"cited_title":"Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), Statistical science 16 (2001) 199–231","cited_arxiv_id":null,"evidence_quote":"introduces the Rashomon set concept that defines the object VAR is designed to analyze."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the structural decision-tree exploration interface that VAR positions itself against for horizontal comparison."},{"cited_title":"Variable Importance Clouds: A Way to Explore Variable Importance for the Set of Good Models","cited_arxiv_id":"1901.03209","evidence_quote":"proposes the scatter-plot-per-model variable-importance visualization that VAR extends with a color-encoded third dimension."},{"cited_title":"iGAiVA: Integrated Generative AI and Visual Analytics in a Machine Learning Workflow for Text Classification","cited_arxiv_id":"2409.15848","evidence_quote":"supplies the radial-basis-function visualization approach that VAR adapts to model performance comparison."},{"cited_title":"Hayden, Z","cited_arxiv_id":null,"evidence_quote":"provides the sparse decision-tree optimization method whose threshold guessing is used to binarize the case-study data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is cited as the source of the model-generation method that produced the Rashomon sets used in the case studies."}],"review_version":1}