{"id":"c19bfd2f-70ce-429c-bd99-8d2dd6a27522","arxiv_id":"2507.10244","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Helveg transforms C# codebases into interactive, filterable node-link diagrams for API documentation, and a second user study with five developers reports improved usability over the first prototype.","lead":"This paper describes Helveg, a prototype that turns C# codebases into interactive node-link diagrams meant to serve as API reference documentation. The authors redesigned the tool based on feedback from five developers and ran a second user study to test whether the changes improved readability and usability.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Same-participant, affiliated-interviewer design leaves the improved usability ratings and the comparative insight claim open to demand-characteristics bias; an independent comparison with a conventional API reference is needed.","rationale":"The reader's weakest assumption—that positive ratings from the second user study reflect genuine improvement rather than demand characteristics or confirmation of participants' own earlier feedback—is precisely the load-bearing point. The paper's central claim is a comparative usability/utility claim, and its strongest evidence is the Figure 5 Likert improvement and qualitative quotes, both drawn from a repeated-measures design with the same five developers and an interviewer affiliated with the sample project. This is not a matter of internal inconsistency; the paper is transparent about its small sample and limitations. But it is a matter of external validity: without an independent, blinded comparison, the data cannot support the strong 'hard to achieve using a typical API reference' assertion. I agree with the reader's conditional verdict. The paper deserves publication as a design and pilot study, but the central claim should be framed as preliminary until an independent evaluation with a comparison condition is run. I found no more load-bearing concern: the tool is open-source, the design rationale is grounded in prior work, and the qualitative findings are broadly consistent with the claims, so the methodological weakness is the primary threat.","tokens_in":19462,"tokens_out":2166,"duration_ms":29074,"concrete_test":"Recruit 8-12 new C# developers with no prior involvement with the project or authors. Randomly assign them to use either Helveg or a conventional API reference generated for the same KAFE codebase (e.g., DocFx), with tasks mirroring the study's overview goals: identify project purposes, key dependencies, the largest type, and compiler diagnostics. Use a blinded interviewer and pre-registered outcome measures (task accuracy, time, and a Likert questionnaire). If Helveg users perform comparably or better and ratings remain high, the concern is resolved; if performance or satisfaction drops relative to the API reference, or if ratings depend on the interviewer's affiliation, the central claim should be softened to an unvalidated hypothesis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Helveg is functional and can provide insight into a codebase that would be hard to achieve with a typical API reference—depends on the second user study's positive ratings and qualitative observations. That study reused the same five developers who had previously criticized the tool, and the interviewer was one of KAFE's developers. Participants could see their own earlier feedback implemented, which invites confirmation and demand effects; this is especially problematic because the study was explicitly framed as validating fixes to their complaints. The comparative clause ('hard to achieve using a typical API reference') is supported only by anecdotal statements and by inference from Figure 5, since neither study included a direct comparison condition with a conventional API reference on equivalent tasks. The icon-recognition improvement is also confounded: the second glyph test used different participants, different icons, and a different testing context than the first survey, so the 93.33% vs. 68.52% comparison is not a clean measure of design improvement. None of this makes the paper internally inconsistent, and the authors honestly acknowledge several of these threats in Section VI.C, but the load-bearing assumption that the reported Likert gains and insights reflect genuine utility rather than social desirability remains unsecured. Without it, the evidence does not yet distinguish 'improved for these five motivated participants' from 'improved for typical developers'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Helveg is an automatically generated interactive node-link diagram intended to complement or replace API reference documentation for C# codebases. The paper motivates the design with requirements R1-R7, then describes the glyph vocabulary (entity-kind icons, static/instance donut charts, accessibility corner icons, abstract/sealed contours, compiler-diagnostic animations), edge relations, filtering mechanisms (full-text, regex, JavaScript, and a filter builder), interaction and layout, and the user interface including a tree view, tutorial, and cheat sheet. It reports a first qualitative study from the prior VISSOFT version, a second user study with the same five professional developers after a redesign, and two glyph-recognition tests. The paper claims the redesign improved readability, intuitiveness, and interactivity, and concludes that Helveg is functional and can provide insight into a codebase that would be hard to achieve with a typical API reference.","tokens_in":19714,"tokens_out":5366,"duration_ms":64906,"significance":"The main contribution is a concrete, open-source prototype with a thoughtful design space exploration and unusually honest reporting of qualitative feedback. The paper also contributes useful lessons: tutorial/cheat-sheet features are essential for configurable tools, filter builders can replace programming-based filters, and custom icons inspired by existing iconography can outperform copied icons. The objective icon-recognition data, although confounded, provides a useful signal. If the comparative insight claim were established, Helveg would be a useful onboarding aid for unfamiliar C# codebases. However, the evidence as presented is exploratory: five participants, no control condition, the same participants in both studies, and no direct comparison against an API reference on equivalent tasks. The conclusion therefore goes beyond what the current evaluation can support, despite the authors' commendable acknowledgment of several threats in Section VI.C.","major_comments":[{"comment":"The paper's central claim that the redesign improved usability and that Helveg enables codebase insight not achievable with a typical API reference rests on the second user study, which used the same five participants who had previously criticized the tool and an interviewer who is one of KAFE's developers and a co-creator of that project. Because the second study was explicitly intended to validate fixes to the participants' own complaints, the Likert gains and supportive quotes in Fig. 5 are vulnerable to demand characteristics and confirmation bias. Section VI.C acknowledges the lack of a direct API-reference comparison and the small sample, but the conclusion in Section VIII still states the comparative claim as established. Please either soften the conclusion to describe exploratory evidence, or add an independent evaluation, such as a between-subjects or within-subjects comparison with a conventional API reference on equivalent comprehension tasks.","section":"Section VI.B and Fig. 5"},{"comment":"The improvement from 68.52% to 93.33% in icon recognition is not a clean measure of the new icon set, because the two tests differed in participants, icons, and task context: the first tested recall of Visual Studio icons that participants had seen for years, while the second tested recognition of Helveg's new icons immediately after a session in which participants had used those icons. The authors themselves note this difference in Section VI.C, yet the text still states that 'this result shows a great improvement in the glyphs' readability.' Please present this as a descriptive, non-comparative result, or conduct a matched test with a pre-exposure baseline.","section":"Section VI.B, glyph tests"},{"comment":"The discussion claims that Helveg 'can be used to gain knowledge about a codebase that a typical API reference can hardly provide,' supported by examples such as the oversized ProjectService node and the visualization of compiler diagnostics. These are plausible illustrations of what the diagram can show, but they are anecdotal observations from five participants, with no task-based or controlled comparison against a conventional API reference. The claims about relative difficulty ('only with great difficulty,' 'would be hard to achieve') are therefore not empirically established. Please reframe these statements as design rationale or hypotheses, or provide comparative task data.","section":"Section VII.A and Section VIII"}],"minor_comments":[{"comment":"The phrase 'color perception deficiencies' should be 'color vision deficiencies' for consistency with standard terminology.","section":"Section V.A"},{"comment":"The sentence 'The testing sessions took between 58 to 96 minutes' should read 'between 58 and 96 minutes.'","section":"Section VI.A"},{"comment":"The sentence 'They have 7-12 years of professional C# experience at a software company' is vague; please specify whether this is the same software company for all participants.","section":"Section VI.A"},{"comment":"The phrase 'one tester (E3) decided to give it a lower score' has an ambiguous antecedent for 'it'; consider 'gave readability a lower score.'","section":"Section VI.B"},{"comment":"The description of the JavaScript mode says 'the JavaScript (JS) mode allowing the user to write custom search logic with a JS filter'; 'with a JS filter' is redundant and could be simplified to 'with custom JavaScript logic.'","section":"Section V.D"},{"comment":"The phrase 'Through repeated evaluation with professional software developers' could be read as implying independent samples; consider 'Through two rounds of evaluation with the same group of professional software developers' to be more precise.","section":"Section VIII"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent and honest design-and-evaluation report, but the evaluation is at proof-of-concept level. If the journal expects strong empirical evidence for usability and comparative insight claims, the current study design is not sufficient; if it accepts systems/design contributions with transparent preliminary evaluation, the manuscript is within scope. The authors' openness in Section VI.C is a positive signal, and the open-source prototype and supplementary material are valuable assets. The main revision need is to bring the strength of the conclusions in line with the evidence, either by tempering the claims or by adding a more controlled comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is an honest, well-written design paper about a tool (Helveg) that turns API documentation into an interactive node-link diagram. The core idea appeared in their VISSOFT 2024 paper; this extended version contributes a redesigned glyph set, a filter builder, a tutorial, a tree view, and a second user study with the same five developers. What's genuinely good: the prototype is open source, the redesign decisions are grounded in the earlier study's feedback, and the new glyph recognition result (93% correct) is a concrete, objective improvement over the 68.5% in the first survey. The qualitative comments give a real sense of how the tool is used, and the authors' discussion of lessons learned is specific and useful.\n\nThe soft spots are the ones you'd expect. The second study reuses the same five participants, the interviewer is one of KAFE's developers (a project the first author co-created), and there is no control condition or direct comparison with a conventional API reference. The participants' own earlier complaints are the ones being fixed, so some of the positive Likert movement in Figure 5 is plausibly the pleasure of seeing your feedback land, not a clean measure of usability. The icon comparison is not clean either: the first survey tested Visual Studio icons with different participants, while the second tested Helveg's custom icons after a session of use. The authors do acknowledge most of this in Section VI.C, which earns them credit.\n\nThe central claim—that Helveg offers insight 'hard to achieve' with a typical API reference—is the weakest part. It rests on anecdote and inference. As a qualitative design study, the paper is solid; as a claim of comparative advantage, it's under-supported. That should be fixed before publication, not by killing the paper.\n\nWho's this for? People working on software visualization, documentation tools, and glyph design. It's a legitimate extension of an ongoing research project, not a breakthrough. I'd send it to peer review, with a clear request for either a more rigorous evaluation or a more careful wording of the comparative claim. I'd cite it if I were doing follow-up work on glyph readability or interactive documentation.","headline":"A genuinely useful design and evaluation of an interactive diagram tool for API docs, held back by a same-participant evaluation that cannot yet support the comparative claim.","tokens_in":20226,"tokens_out":1780,"would_cite":true,"duration_ms":20072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An interactive diagram can give developers a high-level view of an unfamiliar C# codebase that a typical API reference cannot.","keywords":["software visualization","software documentation","API reference","interactive diagram","glyph design","code navigation","C# codebases","user study"],"falsifier":"A controlled study with fresh developers who have never used Helveg, comparing time and accuracy on orientation tasks (identifying project purposes, key dependencies, and refactoring candidates) against a conventional API reference for the same unfamiliar codebase, would settle whether Helveg truly provides insight a typical API reference cannot; a null or negative result would refute the paper's central claim.","tokens_in":19311,"feed_emoji":"🧭","tokens_out":8328,"duration_ms":88749,"temperature":0.7,"pith_summary":"This paper claims that an automatically generated, interactive node-link diagram can serve as or complement an API reference for exploring an unfamiliar codebase, and that a carefully redesigned glyph set makes such a diagram readable enough for daily documentation use. The authors built Helveg, a prototype that mines a C# solution through compiler analysis APIs and renders the codebase as a web-hosted graph whose nodes carry type, member, accessibility, and diagnostic information. After a first user study found the interface, filtering, and glyphs confusing, the extended version replaces JavaScript-based filtering with a visual filter builder, adds a tutorial, cheat sheet, and tree view, and redesigns every icon and node encoding. A second study with the same five professional C# developers reported strong improvements in intuitiveness, interactivity, and readability, and the authors conclude the tool is functional and can provide insight that would be hard to get from a typical API reference.","feed_headline":"Interactive diagrams give developers code insight API pages cannot","feed_subtitle":"Helveg's glyph nodes encode types, inheritance, and compiler errors, and five developers rated the redesign clearer.","key_machinery":"The load-bearing mechanism is the glyph-node: a circular node that fuses entity kind, static/instance status, member-count ratio, accessibility, abstract/sealed modifiers, collapse state, and compiler diagnostics into a single recognizable visual mark. Glyph-nodes are the unit the reader must parse, so their redesign is what moves the second study's satisfaction and readability scores; without them, the diagram degenerates into an unreadable hairball and the paper's claim loses its evidence base. Supporting machinery includes the declares, inheritsFrom, typeOf, and dependsOn edge relations; the filter builder that replaces JavaScript search; and the TidyTree-then-ForceAtlas2 layout pipeline that arranges the graph after each expansion.","core_discovery":"On the paper's own terms, the discovery is that a codebase's documentation can be organised around a single interactive diagram instead of a tree of text pages, and that this diagram can be made legible by encoding C# semantics directly into compound circular glyphs. Each glyph's icon marks the entity or type kind, a filled versus stroked icon marks static versus instance, a surrounding donut shows the static-to-instance member ratio and total member count, corner locks mark accessibility, hexagon and octagon contours mark abstract and sealed, fire and smoke mark compiler errors and warnings, and shadows mark collapsed expandable nodes. Edges separately render declaration containment, type inheritance, member types, and project dependencies, each with its own color and toggleable visibility. The user studies are offered as evidence that, after the redesign, testers recognised 93.33% of the icons on average, answered 95% of glyph-reading questions correctly, and rated the tool as a considerable improvement, supporting the conclusion that Helveg offers insight into a codebase that would be hard to achieve with a typical API reference.","pith_inferences":["Editorial extension: the strongest test of the paper's conclusion would be a controlled comparison in which fresh participants, not the five who criticised the first version, solve identical orientation tasks with Helveg versus a conventional API reference while measuring time and correctness; the current studies do not contain that direct comparison.","Editorial extension: the finding that programmers preferred a structured filter builder over writing JavaScript is a general design lesson: even expert developers may reject flexible scripting interfaces in tools they use for browsing, favouring simpler query construction.","Editorial extension: because the second study used the same five developers and an interviewer familiar with the sample project, the positive ratings should be treated as a signal to investigate rather than a settled measure of usability; a replication with strangers would raise confidence.","Editorial extension: if layout performance were improved, for example through GPU-accelerated rendering, the same diagram-first documentation model could scale to solutions with hundreds of projects, where the current prototype slows and the readability gains would be most valuable."],"forward_implications":["A newcomer to an unfamiliar C# codebase can start from a diagram that shows the whole solution and drill down to a single method, getting both orientation and details without reading a sequence of documentation pages.","Because filtering no longer requires writing JavaScript, a wider range of developers can isolate relevant nodes, so the documentation's usefulness no longer depends on scripting comfort.","Compiler errors and warnings become part of the documentation itself, surfacing code health alongside structure in a way a conventional API reference does not offer.","The diagram can be generated automatically and embedded in a website, making it a practical drop-in for or supplement to a web-hosted API reference rather than a manual drawing task.","If the redesigned glyphs are as readable as the second study suggests, the same encoding template could be reused for other object-oriented languages with comparable type systems."],"supporting_citations":[{"why":"supplies the two-mode account of API reference use (building a mental model versus looking up details) that motivates the diagram's dual overview/detail design.","marker":"[9]"},{"why":"the prior study and glyph survey; its negative feedback establishes the baseline that the redesign must fix and provides the comparison for the second study.","marker":"[13]"},{"why":"the supplementary material holding the full questionnaire, session summaries, testers' answers, and the sample visualization that the study results report from.","marker":"[32]"},{"why":"the survey of programmers, tutors, and students that grounds the initial requirements for familiarity, automation, interactivity, and web output.","marker":"[48]"},{"why":"the cited compiler analysis API used to mine the C# semantic model and obtain diagnostics automatically.","marker":"[49]"},{"why":"the cited visualization design framework used to iterate the design and reject unjustified 3D.","marker":"[53]"},{"why":"the cited glyph design guidelines used to map entity kind to color for pop-out and to avoid mapping variables onto circle radius.","marker":"[59]"},{"why":"the cited taxonomy-based glyph design template that Helveg's icon design follows.","marker":"[60]"},{"why":"the open-source sample codebase used in both user studies, whose developer served as interviewer and could evaluate participant observations.","marker":"[64]"}],"fun_headline_variants":["See C# code structure with Helveg's interactive diagrams","From text docs to interactive glyph maps for code","Helveg: visual API docs with semantic node glyphs","Codebase insight through interactive diagrams, not pages","Glyph-encoded diagrams for exploring C# codebases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's positive results assume that the five developers, who had critiqued the earlier version and were interviewed by someone involved in the sample project, rated the redesign on its merits rather than out of a wish to confirm their earlier complaints or please the interviewer.","fun_headline_variants_meta":{"raw":{"variants":["See C# code structure with Helveg's interactive diagrams","From text docs to interactive glyph maps for code","Helveg: visual API docs with semantic node glyphs","Codebase insight through interactive diagrams, not pages","Glyph-encoded diagrams for exploring C# codebases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1919,"prompt_tokens":1006,"completion_tokens":913,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":835}},"tokens_in":622,"tokens_out":913,"duration_ms":10071,"temperature":1.0,"reasoning_tokens":835,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:35:55.794372+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study with fresh developers who have never used Helveg, comparing time and accuracy on orientation tasks (identifying project purposes, key dependencies, and refactoring candidates) against a conventional API reference for the same unfamiliar codebase, would settle whether Helveg truly provides insight a typical API reference cannot; a null or negative result would refute the paper's central claim.","supporting_citations":[{"cited_title":"How developers use API documentation: an observation study,","cited_arxiv_id":null,"evidence_quote":"supplies the two-mode account of API reference use (building a mental model versus looking up details) that motivates the diagram's dual overview/detail design."},{"cited_title":"Interac- tive diagrams for software documentation,","cited_arxiv_id":null,"evidence_quote":"the prior study and glyph survey; its negative feedback establishes the baseline that the redesign must fix and provides the comparison for the second study."},{"cited_title":"Helveg: Diagrams for Software Documentation — Supplementary Material,","cited_arxiv_id":null,"evidence_quote":"the supplementary material holding the full questionnaire, session summaries, testers' answers, and the sample visualization that the study results report from."},{"cited_title":"Extensible Visualization of C# Codebases,","cited_arxiv_id":null,"evidence_quote":"the survey of programmers, tutors, and students that grounds the initial requirements for familiarity, automation, interactivity, and web output."},{"cited_title":"The Roslyn Project: Exposing the C# and VB compiler’s code analysis,","cited_arxiv_id":null,"evidence_quote":"the cited compiler analysis API used to mine the C# semantic model and obtain diagnostics automatically."},{"cited_title":"Munzner, Visualization Analysis and Design","cited_arxiv_id":null,"evidence_quote":"the cited visualization design framework used to iterate the design and reject unjustified 3D."},{"cited_title":"Taxonomy-Based Glyph Design—with a Case Study on Visualizing Workflows of Biological Experiments,","cited_arxiv_id":null,"evidence_quote":"the cited taxonomy-based glyph design template that Helveg's icon design follows."},{"cited_title":"Roseck ´y and A","cited_arxiv_id":null,"evidence_quote":"the open-source sample codebase used in both user studies, whose developer served as interviewer and could evaluate participant observations."}],"review_version":1}