{"id":"e2842f61-350b-45cb-b87a-87a4c2c22c0f","arxiv_id":"2504.18238","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SecCityVR visualizes static analysis security findings as a VR code city with call graph context and collaboration, and its user study reports lower frustration and temporal demand than a dashboard despite longer task times.","lead":"This paper introduces SecCityVR, a virtual reality environment that displays software security vulnerabilities as colored floors of code city buildings, with call graph arcs and multi-user collaboration. A user study with 17 developers found VR users reported lower frustration and temporal demand than dashboard users, though tasks took longer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VR advantage may rest on a weak baseline: the comparison used a raw table with browser search disabled, not the actual find-sec-bugs dashboard, so lower frustration and temporal demand may not be due to VR.","rationale":"I agree with the reader's weakest_assumption: the most load-bearing condition is whether the baseline dashboard is a fair proxy for find-sec-bugs, and the paper itself acknowledges that it is not. This is a direct threat to the central empirical comparison, not a side issue. There is also an internal inconsistency in Section 4.2's discussion: the text first says hypothesis H2 'could be accepted' based on the higher SUS score, then immediately says the SUS result is not statistically significant and therefore H2 is not accepted. The abstract's 'higher usability' claim thus overstates the evidence. The proposed re-run with the real find-sec-bugs dashboard would settle whether VR's reported experiential advantages survive a fair baseline. Since this concern is fixable and the reader already issued a CONDITIONAL verdict, I do not recommend changing the verdict.","tokens_in":15870,"tokens_out":3659,"duration_ms":37329,"concrete_test":"Re-run the same task set (T1–T4 and the collaborative prioritization task) with the actual find-sec-bugs dashboard UI as the baseline, or with a comparable 2D graphical aggregation (e.g., a SonarQube/SoftVis3D-style code city) with normal search enabled, while keeping the same analyzed JAR, tasks, and questionnaires (SUS, TLX). If VR no longer shows lower frustration and temporal demand (or a SUS advantage), the headline conclusions are baseline artifacts. At minimum, report TLX frustration and temporal demand medians for VR versus the real-tool baseline with effect sizes and confidence intervals, not only p-values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that SecCityVR provides higher usability, lower temporal demand, and significantly lower frustration than find-sec-bugs—depends on the baseline being a fair proxy for the find-sec-bugs dashboard. It is not. Section 4 states that the baseline dashboard is created from the same analysis XML and that participants are not allowed to use the browser's search function; the call-graph data is supplied as a raw JSON text file. This is a generic tabular listing of raw SAST results, not the find-sec-bugs GUI that practitioners would use. The authors concede this in the threats to validity: the baseline displays the raw results in tabular form, and they note that another comparison of VR versus a 2D graphical aggregation should be done in the future to rule out this threat. Because the baseline disables a core feature (search) and lacks any aggregation or graphical encoding, the observed differences in frustration (TLX 17.5 vs. 52.5, p = 0.03) and temporal demand (10 vs. 42.5, p = 0.061 in the main study) may reflect the baseline's handicap rather than VR's merits. The SUS difference (70 vs. 58.75, p = 0.14) is not statistically significant, so the abstract's unqualified claim of higher usability is also not supported by the reported evidence. The load-bearing condition—baseline representativeness—is therefore not met by the current study.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SecCityVR, a virtual reality environment that visualizes static analysis security testing (SAST) results using the code city metaphor, augments the visualization with call-graph arcs, and supports multi-user collaboration. The authors describe the system architecture, the mapping from Java artifacts to city elements, the interaction and collaboration features, and an evaluation consisting of a between-subjects main study (17 participants) and a smaller collaboration study (8 participants). The VR condition is compared with a tabular baseline derived from the same SAST XML and call-graph JSON files. Reported results include longer task completion times in VR, higher median SUS score (70 vs. 58.75, p=0.14), lower TLX temporal demand (10 vs. 42.5 in the main study, p=0.061; 25 vs. 77.5 in the collaboration study, p=0.029), and lower TLX frustration (17.5 vs. 52.5, p=0.03 in the main study). The abstract and conclusion claim that the VR approach provides higher usability, lower temporal demand, and significantly lower frustration, and the authors accept hypotheses H1 and, with caveats, discuss H2.","tokens_in":16143,"tokens_out":4002,"duration_ms":39888,"significance":"If the empirical claims were fully supported, the paper would make a useful contribution to software visualization and security auditing: it applies VR code cities to a security context, integrates call-graph context, and provides an open-source Unity implementation plus a multi-user mode. The use of established instruments (SUS, TLX, VRSQ, IPQ) and the availability of raw data are strengths. However, the central comparative claims are weakened by two load-bearing problems: the baseline is not the find-sec-bugs dashboard named in the abstract but a raw tabular listing with browser search disabled, and several headline benefits (usability, temporal demand in the main study) are not statistically significant at the chosen alpha level. The paper's scientific value currently lies more in the system design and exploratory findings than in the evidence for a decisive advantage of VR over a traditional dashboard.","major_comments":[{"comment":"The comparison baseline is not a fair proxy for the 'traditional dashboard find-sec-bugs' named in the abstract. Section 4 states that the baseline is a tabular display of the same SAST XML and that participants are not allowed to use the browser's search function; the call graph is provided as a raw JSON text file. This is a generic table with a core feature disabled, not the find-sec-bugs GUI that practitioners would use. The authors themselves acknowledge in the threats to validity that 'the kind of information displayed on the baseline dashboard could be criticized' and that the raw tabular form may be an unsuitable comparison for visualization properties. Because the central claim of lower frustration and lower temporal demand is a comparison against this weakened baseline, the observed differences may reflect the baseline's handicap rather than a benefit of VR. To make the central claim load-bearing, the study would need to use the actual find-sec-bugs dashboard (or an equivalent 2D graphical aggregation) as the baseline, or the claims must be explicitly limited to a comparison against raw tabular results.","section":"Section 4 (Evaluation) and Section 4.2, Threats to validity"},{"comment":"The abstract's claim of 'higher usability' is not supported by the reported statistics: the SUS difference (70 vs. 58.75) has p=0.14, which is not significant at alpha=0.05. Likewise, 'lower temporal demand' in the main study has p=0.061, which is also not significant. Section 4.2 correctly states that H2 cannot be accepted because the SUS result is not significant, yet the conclusion and abstract present improved usability and reduced cognitive load as established benefits. This internal inconsistency must be resolved by either reporting only the significant findings or providing additional evidence (e.g., effect sizes and a preregistered analysis) that justifies the stronger wording.","section":"Abstract and Section 4.2 (Usability, Task Load)"},{"comment":"The statistical analysis suffers from multiple comparisons and a small sample. The TLX questionnaire includes six dimensions, and the authors perform separate Mann-Whitney-U tests for each dimension in both the main and collaboration studies without any correction for multiple comparisons (e.g., Bonferroni). With approximately 9 VR and 8 dashboard participants in the main study and 4 per group in the collaboration study, the single significant frustration result (p=0.03) would not survive a conservative correction across six dimensions. The paper should report effect sizes, confidence intervals, or adjusted p-values, and should label the TLX dimension comparisons as exploratory rather than definitive.","section":"Section 4.2 (Results) and Section 4.1 (Procedure)"},{"comment":"The VRSQ analysis reports p-values (p_ocu=0.046, p_dis=0.012, p_vrsq=0.033) described as 'significant result,' but VRSQ is collected only in the VR group and there is no described comparison condition or null hypothesis. It is unclear what test produced these p-values (e.g., comparison against a zero-symptom baseline? pre-post comparison?) and what the explicit hypotheses were. Without this information, the p-values are uninterpretable. The authors should specify the test, the comparison value, and the direction of the effect.","section":"Section 4.2 (Motion Sickness)"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, e.g., 'Additionnally' in Section 4, 'groundlay' in Section 3, and 'differ notable' in Section 4.2. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"In the collaboration study results, the performance dimension is reported as '72.5 Dashboard, 52.5 VR, p=0.2*' with an asterisk despite p=0.2 being non-significant. The asterisk convention should be consistent and only applied to p<0.05.","section":"Section 4.2 (Task Load)"},{"comment":"The tables of job titles, experience, and security experience are not numbered and have incomplete captions (e.g., Table 3 is captioned 'Security Experience' but the column header is 'Experience Count'). Numbering and clearer captions would improve readability.","section":"Section 4.2 (Demographic Data)"},{"comment":"The paper reports on the collaboration study with 8 participants but does not clearly state how many were in each group (VR vs. dashboard) or whether these participants are a subset of the 17 from the main study. Clarify the participant flow and group sizes.","section":"Section 4 (Procedure and Results)"},{"comment":"Figure 4 shows task times but does not include error bars or descriptive statistics such as standard deviations. Adding these would help readers assess the variance behind the reported medians.","section":"Section 4.2 (Efficiency)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid system contribution and a transparently reported study, but the central comparative claims are currently overstated relative to the evidence. The most important fix is to either re-run the comparison against the actual find-sec-bugs dashboard (or another 2D graphical tool) or explicitly reframe the contribution as a proof-of-concept with exploratory results. The authors should also correct the internal contradiction between the non-significant SUS/temporal-demand results and the abstract/conclusion wording. With these changes, the paper could become a defensible contribution; without them, the headline results are not reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, SecCityVR is a real, novel artifact: a VR code city that layers SAST findings onto building floors, draws call-graph arcs between vulnerable methods, and supports synchronized multi-user exploration with guided review. I looked for prior VR work on security-vulnerability visualization in the related work and didn't find any, so the combination appears new. The authors ship the Unity project, a demo video, and the raw questionnaire data, which makes the work independently checkable. Second, the evaluation is a pilot that is being oversold. The comparison baseline is not the find-sec-bugs dashboard; it is the raw XML results in a table with browser search disabled, plus the call graph as a JSON text file. The authors acknowledge this in the threats-to-validity section. That makes the 'lower frustration' (p=0.03) and 'lower temporal demand' (p=0.061 in the main study, p=0.029 in the collaboration study) results suggestive but not clearly attributable to VR—they may reflect how bad the raw table is. The SUS difference (70 vs 58.75, p=0.14) is not statistically significant, so the abstract's 'higher usability' and the conclusion's 'improved usability' overclaim. The 'reduced cognitive load' in the conclusion also lacks support: the main study shows no significant difference in mental demand. And in the collaboration study, physical demand was significantly higher in VR (p=0.029), which the summary omits. Sample size is 17 unique participants, with no correction for multiple comparisons. Still, the paper is honest at the level of reporting p-values and limitations; the problem is the spin in the abstract and conclusion. The system design is thoughtful, the call-graph visualization addresses a real gap (P2), and the collaborative guided-review feature is a plausible answer to P3. Who should read it: people working on software visualization, VR for security, or code-city metaphors. It is a useful pointer and a citable proof-of-concept, not a demonstration of VR superiority. If this came to me for review, I would not desk-reject it; I would send it to expert reviewers and ask them to push for tempered claims, and ideally a follow-up comparison against the actual find-sec-bugs GUI or a 2D aggregated view.","headline":"Novel VR code city for vulnerability exploration, but the evaluation overreaches: the baseline is a raw table and the headline usability claim is not significant.","tokens_in":16650,"tokens_out":2988,"would_cite":true,"duration_ms":28099,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a user study, a VR code city rendered software vulnerabilities with significantly less frustration and higher usability than a dashboard, though task times were longer.","keywords":["Virtual Reality","Security Vulnerabilities","Static Analysis Security Testing","Code City Metaphor","Software Visualization","Security Audits","Collaborative Software Engineering","Call Graph Visualization"],"falsifier":"A follow-up study with a larger sample that compares SecCityVR against a current 2D graphical aggregation, such as the real find-sec-bugs or SonarQube dashboard with search enabled, and finds no significant difference in frustration or temporal demand would overturn the central claim; showing that the time gap cannot be reduced by adding search and teleport-to-target would further weaken the practical case.","tokens_in":15664,"feed_emoji":"🏙️","tokens_out":8168,"duration_ms":74747,"temperature":0.7,"pith_summary":"The paper builds SecCityVR, a virtual-reality environment that turns a Java codebase into a code city: classes become buildings, methods become floors colored by vulnerability priority, and call-graph edges become arcs running above the city. The claim is that this spatial, collaborative presentation lets security auditors find and understand vulnerabilities as well as they would with a tabular dashboard, while feeling less rushed and less frustrated. A between-subjects study with 17 participants found higher SUS usability (70 vs 58.75), lower temporal demand, and significantly lower frustration for VR, but also significantly longer task completion times. The authors argue this trade-off is worth it: VR shifts the experience from stressful triage to exploratory review and enables real-time multi-user discussion.","feed_headline":"VR code cities cut security-audit frustration vs dashboards","feed_subtitle":"In a 17-participant study, the VR tool scored higher usability and lower workload, though tasks took longer.","key_machinery":"The central object is the code city metaphor transplanted into VR: a squarified treemap lays out packages as districts on a baseplate sized by total lines of code, classes are extruded as buildings, methods appear as floors only when they contain a vulnerability (or are connected to one), and method floors are colored by issue priority and widened for emphasis. Call-graph edges are drawn as color-gradient arcs between methods, making possible propagation paths visible. This mapping carries the argument because it preserves the package-to-class hierarchy while adding spatial locality, and the on-demand rendering of arcs plus the multi-user networking (via Photon Fusion) is what supports collaborative exploration of vulnerability impact.","core_discovery":"SecCityVR is claimed to be the first VR-based approach for visualizing and interactively exploring software security vulnerabilities. It maps each class to a building, each method to a color-coded floor (red, orange, green, or blue matching the priority scale of the find-sec-bugs analysis), packages to districts via a squarified treemap, and call-graph edges to arcs that users can render on demand; dependency packages are distinguished in magenta. Multi-user networking lets two auditors walk or fly through the city, teleport to each other, and start a guided review where one user's position is synchronized to the other's. In an evaluation against a tabular baseline built from the same find-sec-bugs XML and call-graph JSON, VR users solved the same tasks with comparable correctness and the results support accepting H1, the claim that the same tasks can be solved. VR scored higher on the System Usability Scale and markedly lower on temporal demand and frustration, although the SUS difference was not statistically significant, so H2 was not formally accepted.","pith_inferences":["If the lower-frustration effect survives a stronger baseline, VR may be most valuable for junior developers: the embodied layout plus built-in explanations and mitigations could support learning how vulnerabilities spread, not just finding them.","The city could be extended into a complete triage tool by visualizing concurrent vulnerabilities within one method and letting users mark issues as reviewed or deferred, features the paper names only as future work.","A testable extension is to measure whether the arc-based view specifically speeds up root-cause reasoning about call chains compared with 2D graph views, since the arcs are the main differentiator of this design.","The practical case would be stronger if the comparison were repeated against a proper 2D graphical aggregation such as the real find-sec-bugs or SonarQube dashboard with search enabled; the authors themselves call for this follow-up."],"forward_implications":["VR code cities are a viable medium for security audits: users completed the same four tasks as the dashboard group with comparable correctness, supporting the paper's H1.","Security reviews in VR are experienced as less rushed and less frustrating, which could improve developer engagement and willingness to participate in audits.","Multi-user VR collaboration supports distinct work strategies, such as splitting up to explore and then using guided review, or teleporting to a partner's findings for immediate discussion.","Call-graph arcs give an at-a-glance view of how vulnerabilities propagate across components, directly addressing the paper's problem statement P2.","Adding search, teleportation to distant targets, and source-code integration could close the significant time gap between VR and dashboard usage."],"supporting_citations":[{"why":"Introduces the code city metaphor (classes as buildings, packages as districts) that SecCityVR extends into VR for vulnerabilities.","marker":"[29]"},{"why":"Defines squarified treemaps, the layout algorithm used to subdivide the city baseplate into package districts.","marker":"[6]"},{"why":"Experimental evidence that VR representation of code cities is viable compared with on-screen representation, motivating the VR choice.","marker":"[25]"},{"why":"Shows that code cities give users a sense of locality, which supports the exploration-oriented design of SecCityVR.","marker":"[22]"},{"why":"Lists developers' demand for analysis output that shows what else could be affected, grounding the call-graph and info-panel features.","marker":"[14]"},{"why":"Supplies the controlled-experiment procedure comparing code cities against tables and the significance-testing approach used in the user study.","marker":"[9]"},{"why":"Describes the SEE-Tool, the closest prior VR collaborative code-city environment, which the paper contrasts with SecCityVR's vulnerability focus.","marker":"[17]"},{"why":"Provides the System Usability Scale used to measure the usability difference between VR and dashboard.","marker":"[5]"},{"why":"Provides the NASA-TLX workload instrument that yielded the temporal-demand and frustration findings.","marker":"[12]"},{"why":"Shows developer engagement and curiosity in VR code cities, supporting the collaboration and gamification aspect of SecCityVR.","marker":"[20]"}],"fun_headline_variants":["First VR code city maps security flaws, cuts frustration","VR security audit: lower frustration, longer tasks","SecCityVR: virtual city visualizes software vulnerabilities","Collaborative VR security visualization eases audit frustration","VR code city for security flaws: better usability, less frustration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison dashboard was a raw tabular listing of the same XML and JSON results, not the actual find-sec-bugs dashboard interface; if that baseline is unfairly weak, the reported VR advantages in frustration and temporal demand may shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["First VR code city maps security flaws, cuts frustration","VR security audit: lower frustration, longer tasks","SecCityVR: virtual city visualizes software vulnerabilities","Collaborative VR security visualization eases audit frustration","VR code city for security flaws: better usability, less frustration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3878,"prompt_tokens":1033,"completion_tokens":2845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":2769}},"tokens_in":649,"tokens_out":2845,"duration_ms":22061,"temperature":1.0,"reasoning_tokens":2769,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:20:34.736126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A follow-up study with a larger sample that compares SecCityVR against a current 2D graphical aggregation, such as the real find-sec-bugs or SonarQube dashboard with search enabled, and finds no significant difference in frustration or temporal demand would overturn the central claim; showing that the time gap cannot be reduced by adding search and teleport-to-target would further weaken the practical case.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the code city metaphor (classes as buildings, packages as districts) that SecCityVR extends into VR for vulnerabilities."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines squarified treemaps, the layout algorithm used to subdivide the city baseplate into package districts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Experimental evidence that VR representation of code cities is viable compared with on-screen representation, motivating the VR choice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the SEE-Tool, the closest prior VR collaborative code-city environment, which the paper contrasts with SecCityVR's vulnerability focus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows developer engagement and curiosity in VR code cities, supporting the collaboration and gamification aspect of SecCityVR."}],"review_version":1}