{"id":"d2b066c8-cb1c-461e-83f0-222e7e7a3f5a","arxiv_id":"1908.04090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The authors built VISON, the first ontology of software visualization tools, populated with 70 publicly available tools, to support tool discovery and baseline selection for experiments.","lead":"This paper introduces VISON, a structured catalog of 70 publicly available software visualization tools, along with an ontology that lets people search them. It aims to help developers find existing visualization tools and help researchers choose a baseline tool for experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The discoverability claim rests on a manually populated catalog whose attribute accuracy is unverified and already internally inconsistent (structure-tool counts in §IV.A.2 disagree with Table I); a wrong attribute can change query results.","rationale":"VISON's strongest claim is as a searchable index: for it to support developers and researchers, the extracted characteristics in Table I and the OWL instance assertions must be correct. The paper gives a described protocol and a DOI, which is real support, and the total of 70 tools divides into 28+22+12+8, so the artifact is tangible. But the only validation of the index is two usage scenarios, and those do not establish whether the queries return the right tools. The Table I count mismatch is an objective signal that the manual extraction has at least one reporting error; if one attribute is wrong, others may be too. That is not a disagreement with the community consensus; it is an internal consistency problem in the central data artifact. The fix is not inherently difficult: verify the released OWL file against primary sources and run the scenarios with a gold standard. Since the paper is a resource contribution, this concern does not warrant rejection; it does warrant keeping the reader's CONDITIONAL verdict until the artifact and catalog are independently validated.","tokens_in":14758,"tokens_out":7227,"duration_ms":76076,"concrete_test":"Load the released ontology (DOI 10.5281/zenodo.3268626) and reconstruct the two usage-scenario queries from Figures 11-12; then compare the returned individuals with a gold standard built from the primary papers and from re-downloading the repositories for a random sample of 20 of the 70 tools. Independently re-verify, blind to the authors' spreadsheet, the attributes concern, environment, medium, license, and last-update, and compute Cohen's kappa. If kappa < 0.8 for any attribute, or if any query's returned result set changes, the catalog is not reliable enough to support the discoverability claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value is a populated ontology; if the 70 instances are mislabeled, every query result inherits the error. The population process (Section IV.A) is manual, single-protocol, with no inter-rater reliability or independent verification of extracted attributes. Internal evidence already indicates a problem: Section IV.A.2 states that twenty structure tools use the standard screen and three use immersive VR, but Table I lists only 22 structure rows, of which 19 are 'SCS' and one is 'S/I' (ExplorViz) plus two 'I3D' (PhysVis, CityVR). The stated totals do not add up. The usage scenarios are the only demonstration of discoverability, yet their OWL queries and returned tool sets are not given in full, so a reader cannot audit whether the right instances are retrieved. Because 'last update', environment, concern, and license values determine query results, an unverified manual annotation error can change the recommended tool set, collapsing the discoverability claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents VISON, an OWL ontology of software visualization tools, populated with a manually curated catalog of 70 publicly available tools drawn from 387 papers published in VISSOFT and SOFTVIS between 2002 and 2018. The authors follow the Noy and McGuiness ontology-engineering guidelines, report catalog details in Table I, and illustrate two usage scenarios (runtime performance analysis and free-licensed source-code tools) with OWL queries in Section V. The paper claims that VISON is the first software visualization ontology and that it can support developers in discovering suitable tools and researchers in identifying baselines; the artifact is publicly deposited on Zenodo.","tokens_in":14949,"tokens_out":5747,"duration_ms":53539,"significance":"If the catalog is accurate and representative, VISON is a useful queryable index and formal model for the software visualization domain. The strengths are the systematic extraction protocol, the public artifact, the explicit ontology metrics (150 classes, 696 class assertions), and the fact that the scenarios return tools from the encoded data. The contribution is modest but real: a structured, hyperlinked catalog and a reusable ontology. Credit is due for making the artifact available. The main weaknesses are that the discoverability claim is demonstrated only through two hand-picked scenarios with no user study, baseline, or ground-truth evaluation, and the manually populated catalog shows internal inconsistencies that undermine confidence in its attribute values.","major_comments":[{"comment":"The text in §IV.A.2 states that twenty structure tools are displayed on the standard computer screen and only three use immersive virtual reality (PhysVis, ExplorViz, CityVR). Table I, however, lists 22 structure rows, of which 19 are marked SCS, two are marked I3D (PhysVis and CityVR), and ExplorViz is marked S/I. The stated totals do not add up, and the same discrepancy appears in §IV.A.4, where Getaviz is described as supporting immersive virtual reality while Table I lists it as S/I. Because the medium attribute is one of the fields that determine query results in Section V, this internal inconsistency undercuts the reliability of the manually populated catalog and must be resolved by correcting the table, the text, and the ontology so that all three agree.","section":"§IV.A.2, Table I"},{"comment":"The two usage scenarios are the only evidence for the discoverability claim, but the OWL queries and the returned tool sets are shown only as screenshots; the full query text, the full result sets, and the justification that these sets are correct are not given in the manuscript or, as far as can be verified from the text, in the artifact. A reader cannot audit whether the queries retrieve all relevant tools and no irrelevant ones, nor whether the scenarios were chosen to match instances that the authors themselves encoded. Please provide the queries and result lists in machine-readable form, and ideally add a small ground-truth evaluation (e.g., recall and precision against the catalog, or an independent annotation of a sample of tools) so the discoverability claim is testable.","section":"§V, Figures 11–12"},{"comment":"The catalog was populated manually by the authors from their own prior classifications ([6], [7]), with no inter-rater reliability, no independent verification of the extracted attributes, and no per-tool provenance for the 'publicly available', 'last update', 'environment', 'license', and 'medium' values. The Threats to Validity paragraph discusses selection bias but not annotation reliability. Since every query in Section V inherits these attribute values, an annotation error can change the recommended tool set and collapse the discoverability claim. I recommend either adding an independent extraction check (e.g., a second annotator on a sample with inter-rater agreement reporting) or making the evidence for each attribute available in the artifact so that errors can be corrected by the community.","section":"§IV.A, Threats to Validity"}],"minor_comments":[{"comment":"The name 'Humprey' in the Figure 4 caption and in the sentence referencing reference [44] is a misspelling of 'Humphrey'.","section":"Figure 4 caption and §IV.A.1"},{"comment":"Reference [4] contains a typo: 'Proceeedings' should be 'Proceedings'.","section":"Reference [4]"},{"comment":"The heading 'Behavior/Evolution/Structure' in §IV.A.4 does not match the table group label 'E.-S.-B.'; use one consistent abbreviation throughout.","section":"§IV.A.4 vs Table I"},{"comment":"The table lists two distinct tools named 'Jive' (rows with years 2007 and 2016); consider disambiguating the names (e.g., 'Jive (2007)' and 'Jive (2016)') to avoid confusion when querying the ontology.","section":"Table I"},{"comment":"The screenshots in Figures 11 and 12 appear to be small and difficult to read; provide enlarged versions or text-based alternatives so that the queries and returned tool names are legible.","section":"Figures 11–12"},{"comment":"Please specify the query mechanism used (e.g., Protégé DL Query, SPARQL, or a custom reasoner) and the underlying reasoner configuration, since this affects how users reproduce the scenarios.","section":"§V"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-visualization venue. The most serious issue is the concrete inconsistency between the prose counts in §IV.A.2 and Table I, which undermines the reliability of the catalog. The discoverability claim is also supported only by two self-authored scenarios; that is a defensible limitation for an 'early results' paper, but the authors should make the queries and result sets auditable. The public artifact and systematic collection effort are genuine strengths. I do not recommend rejection; the catalog and ontology are useful resources if the data-quality issues are fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine, if modest, contribution. The authors built the first ontology of software visualization, populated it with a 70-tool catalog verified for public availability, and shipped the artifact with a DOI. That is real work, and the paper describes the process honestly. The two usage scenarios show the intended query mechanism working on sensible examples, and the catalog itself will save time for anyone surveying the field.\n\nThe new thing is the populated ontology and the availability-checked catalog. The ontology methodology is standard OWL in Protégé, but that is not a flaw; the value is in the curated instances. The paper does not overclaim much: it says \"early results\" and explicitly lists threats to validity, including the two-venue selection.\n\nWhere it is soft: the core claim is discoverability, but the only evidence is two hand-picked scenarios with no user study and no baseline against a keyword search. The queries and returned tool sets are not shown in full, so a reader cannot audit whether the right instances are retrieved. Also, the catalog is populated by a single manual protocol with no inter-rater reliability check. An annotation error in license, environment, or update date changes query results, so the resource's reliability depends on data quality that has not been independently verified.\n\nI checked the internal count issue flagged in the stress test. It is smaller than it looks. The structure section says 20 tools use the standard screen and three use immersive VR, but the table has 19 SCS, one S/I (ExplorViz), and two I3D. If S/I means the tool supports both, then 19+1=20 standard and 2+1=3 immersive, so the numbers are consistent. The paper could have defined the medium abbreviations more clearly, but I do not see a load-bearing arithmetic error.\n\nBottom line: this is a useful dataset and a reasonable first ontology for a small community. It deserves serious review, but a referee should ask for the full queries, clearer definitions of media types, and ideally an independent or second-annotator check on a sample of the catalog attributes. Even a small user evaluation would strengthen the discoverability claim. These are revision requests, not fatal flaws.","headline":"A genuine but modest resource contribution: a first populated software-visualization ontology and a 70-tool availability-checked catalog, with a discoverability claim that is illustrated, not yet proven.","tokens_in":15452,"tokens_out":3580,"would_cite":true,"duration_ms":30693,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that VISON, the first software visualization ontology built from 70 publicly available tools, lets developers and researchers find suitable visualization tools by querying semantic characteristics rather than reading…","keywords":["software visualization","ontology","tool discoverability","catalog of tools","OWL","development concerns","controlled experiments","tool maturity"],"falsifier":"Take a set of concrete developer queries, such as finding a free tool to visualize runtime performance in Java, and have independent experts, blind to VISON, list the tools they would recommend; if VISON's query results omit a substantial share of the experts' recommended tools, or include tools that are no longer available or do not match the stated concern, the discoverability claim is falsified. A simpler check is to visit the linked repository of every tool returned by VISON for a sample query and verify that the tool is still downloadable and that its last-update date is correct.","tokens_in":14582,"feed_emoji":"🔍","tokens_out":5598,"duration_ms":51750,"temperature":0.7,"pith_summary":"The paper claims that many software visualization tools exist but are rarely adopted because practitioners have no easy way to find one matching their concerns. It tries to close this gap by building VISON, an ontology that formally captures tool characteristics such as software aspect, development concern, execution environment, visualization technique, display medium, and evaluation evidence, and populating it with a curated catalog of 70 publicly available tools. The paper reports usage scenarios in which developers query for tools that fit a stated need, such as analysis of performance at runtime, and researchers identify baseline tools for controlled experiments. If correct, VISON is a queryable index and a shared formal model of the software visualization domain that can be extended and reused.","feed_headline":"70 software visualization tools, now queryable through one ontology","feed_subtitle":"Developers can ask for a free tool that analyzes runtime performance and get a shortlist from the catalog.","key_machinery":"The central object is the VISON ontology itself: a formal OWL model with 150 classes, 20 individual properties, 696 class assertions, and 1,547 object property assertions, built in a widely used ontology editor. It is populated with the curated catalog of 70 tools, each characterized by name, software aspect, concern, last update, execution environment, visualization technique, display medium, and evaluation. The ontology's query mechanism converts a developer's stated need into a class expression and returns matching tool instances; this is the mechanism that turns a static catalog into a discovery engine. It also carries the paper's broader claim that semantic relationships, not just taxonomies, are needed to identify suitable visualization tools.","core_discovery":"The paper's central claim is that discoverability of software visualization tools can be supported by an ontology, and that VISON is that ontology: to the paper's knowledge, the first ontology of software visualizations. The ontology expresses tools as instances of concepts spanning software aspect, development concern, execution environment, visualization technique, display medium, and evidence of effectiveness through evaluation. The catalog behind it was assembled by scanning 387 papers from the two dedicated software visualization venues between 2002 and 2018, keeping only named tools still publicly available, and ended with 70 tools. The paper demonstrates two OWL queries, one for runtime performance visualization and one for free source-code visualization tools, and reports that each returns suitable tools; these demonstrations are the evidence offered that the ontology serves both developers and researchers.","pith_inferences":["A direct test of the discoverability claim would be a user study in which practitioners with real development questions use VISON and rate whether the returned tools match their needs; the paper does not report such a study.","The catalog's restriction to two research venues and to tools that are still publicly available means the ontology likely underrepresents widely used commercial or industrial tools; extending the catalog beyond research venues would test whether the ontology generalizes.","The paper's count of 70 available tools from 387 papers suggests an availability gap in the field; if maintained over time, VISON could serve as a living indicator of tool availability and maturity."],"forward_implications":["A developer can translate a concrete need, such as a free tool for source-code analysis, into a query and receive a shortlist of tools with links to repositories instead of scanning papers.","A researcher proposing a new visualization tool can query VISON for an existing tool with the same concern or technique to serve as a baseline in a controlled experiment.","The ontology can grow by user contributions: adding new tools, new supported questions, or new evaluation results keeps the catalog current.","Because VISON exposes domain structure formally, higher-level search or recommendation frameworks can build on it without re-modeling the domain."],"supporting_citations":[{"why":"Supplies the initial dataset and classification of tool characteristics used to build the catalog.","marker":"[6]"},{"why":"Supplies the previous systematic review of software visualization evaluation, including evaluation data reused in the catalog.","marker":"[7]"},{"why":"Frames the definition of an ontology used as the paper's rationale for choosing an ontology over a taxonomy.","marker":"[9]"},{"why":"Makes the ontology publicly available, supporting the paper's claim of public availability.","marker":"[10]"},{"why":"Provides the prior meta-visualization approach and one of the catalogued tools, MetaVis.","marker":"[17]"},{"why":"Supplies the ontology-design guidelines that structure the development process described in Section III.","marker":"[25]"},{"why":"Supplies the structure/behavior/evolution classification used to group the tools in the catalog.","marker":"[26]"},{"why":"Supplies the average class and instance counts used to argue that VISON is not a small ontology.","marker":"[103]"}],"fun_headline_variants":["VISON ontology makes 70 software visualization tools searchable","One ontology, 70 tools: VISON helps you pick the right visualization","Discover visualization tools with VISON's catalog of 70","Find the visualization tool you need through VISON's ontology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually extracted characteristics in the catalog—what concerns each tool supports, its environment, maturity, and evidence—are accurate and that the 70 tools selected from two venues represent the tools practitioners actually need; if those data are wrong or miss the tools practitioners use, VISON's recommendations will mislead.","fun_headline_variants_meta":{"raw":{"variants":["VISON ontology makes 70 software visualization tools searchable","One ontology, 70 tools: VISON helps you pick the right visualization","Discover visualization tools with VISON's catalog of 70","Find the visualization tool you need through VISON's ontology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1884,"prompt_tokens":905,"completion_tokens":979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":908}},"tokens_in":521,"tokens_out":979,"duration_ms":9436,"temperature":1.0,"reasoning_tokens":908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:51:38.924407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of concrete developer queries, such as finding a free tool to visualize runtime performance in Java, and have independent experts, blind to VISON, list the tools they would recommend; if VISON's query results omit a substantial share of the experts' recommended tools, or include tools that are no longer available or do not match the stated concern, the discoverability claim is falsified. A simpler check is to visit the linked repository of every tool returned by VISON for a sample query and verify that the tool is still downloadable and that its last-update date is correct.","supporting_citations":[{"cited_title":"Artifact: VISON: An Ontology-Based Approach for Software Visualization Tool Discoverability,","cited_arxiv_id":null,"evidence_quote":"Makes the ontology publicly available, supporting the paper's claim of public availability."},{"cited_title":"Ontology development 101: A guide to creating your first ontology,","cited_arxiv_id":null,"evidence_quote":"Supplies the ontology-design guidelines that structure the development process described in Section III."},{"cited_title":"Diehl, Software Visualization","cited_arxiv_id":null,"evidence_quote":"Supplies the structure/behavior/evolution classification used to group the tools in the catalog."},{"cited_title":"Empirical findings on ontology metrics,","cited_arxiv_id":null,"evidence_quote":"Supplies the average class and instance counts used to argue that VISON is not a small ontology."}],"review_version":1}