Pith. sign in

REVIEW 3 major objections 5 minor 112 references

Software Testing for Extended Reality Applications: A Systematic Mapping Study

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read XR software testing mapped: 34 studies, clear gaps

desk verdict Useful first map of XR testing with a real tools/datasets contribution, but the automated search string is internally inconsistent and undermines the completeness claim. read the letter →

arxiv 2501.08909 v2 pith:KEURYGFN submitted 2025-01-15 cs.SE

Ruizhen Gu , José Miguel Rojas , Donghwan Shin This is my paper · ORCID
classification cs.SE
keywords softwaretestingextendedrealitysystematicmappingstudyvirtualaugmentedtestautomationoracleXRapplications
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims to be the first systematic mapping study of software testing for extended reality (XR) applications. It selects 34 primary studies from more than a thousand candidate works, classifies them by research topic, test activity, test concern, test technique, and evaluation method, and publishes the extracted data plus a repository of tools and datasets. The point is to give researchers and practitioners a structured, auditable picture of what has been tried in XR testing and where the open problems are. If the mapping is accepted, the field gains a shared baseline against which future work can be positioned.

What carries the argument

The machinery is the systematic mapping methodology itself: a search string built from PICOC facets and synonym sets, selection criteria piloted on a test set, classification via systematic keywording, a data extraction form, and three rounds of backward snowballing. The load-bearing mechanism is the structured classification of each primary study along dimensions such as test activity, objective, target, level, type, technique, evaluation metric, and environment, which is what converts a list of papers into the claimed landscape and gap analysis.

What would settle it

Run the paper's search string in IEEE Xplore, ACM Digital Library, and Scopus, apply the same inclusion and exclusion criteria, and compare the resulting set with the 34 primary studies. If the multi-database search yields additional primary studies that change the reported distributions (for example, more AR or integration-testing studies), the paper's 'first mapping' claim and its gap analysis would be shown to be incomplete.

Watch

Extended reading notes

Core claim

The central claim is that XR software testing is an emerging but underdeveloped field, and that the evidence supports a specific characterization: research grew from two studies in 2017 to ten in 2023; VR dominates (59% of studies) over AR (26%); automated testing is the most common topic; functionality is the leading test objective; system testing is the dominant level; machine learning is the most used technique; and most proposed solutions receive only preliminary validation. The paper further claims that test generation with automated oracles is the least explored activity, integration testing is nearly absent, and software-centric usability testing is underdeveloped.

Load-bearing premise

The whole map depends on the assumption that searching OpenAlex plus three rounds of backward snowballing captured enough of the relevant literature, so the reported counts and gaps reflect the real field rather than the coverage of one database.

Editorial extensions

If this is right

  • A newcomer can use the 34-study map and the public repository as a starting point instead of re-searching the literature from scratch.
  • The dominance of machine-learning-based testing and image datasets suggests that future XR test tools will likely be oracle predictors trained on screenshots.
  • The near absence of integration testing (one study) and the rarity of unit testing point to concrete opportunities for new methods.
  • Because 60% of proposed solutions are validated only in controlled settings, industrial deployment remains largely unproven.
  • The growth since 2017 and the arrival of real-world evaluations in 2023 indicate a transition toward practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to replicate the search across IEEE Xplore, ACM Digital Library, and Scopus; if those databases surface many additional primary studies, the reported counts and gaps would need revision.
  • The gap analysis implies that oracle automation, especially visual oracles for six-degree-of-freedom interactions, is the bottleneck most likely to reward investment.
  • As XR development shifts toward new headsets and platforms, the paper's VR-centric evidence base may under-represent testing problems specific to mobile and web AR.
  • The mapping's emphasis on software-centric testing suggests a research program of converting user-study findings, such as cybersickness factors, into automated oracles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents a systematic mapping study (SMS) of software testing for Extended Reality (XR) applications, following the Petersen et al. (2015) guidelines. The authors define three research questions (with sub-questions) covering research status, test facets, and evaluation methodologies. They search OpenAlex with a structured query, screen candidates with three independent authors, perform three iterations of backward snowballing, and arrive at 34 primary studies. The selected studies are classified by venue, topic, research type, immersive technology, test activity, test objective, test target, test level, test type, test technique, evaluation metrics, and evaluation environment. The paper also catalogs datasets and tools referenced in the primary studies and proposes future research directions such as interaction formalisation, oracle automation, XR-specific testing, software-centric usability testing, and AI for XR testing.

Significance. If the search and selection process is complete, this is a useful contribution: it provides the first structured overview of XR software testing, an auditable classification scheme, a public repository of extracted data and tools, and a set of research directions grounded in the primary studies. The methodology has notable strengths: explicit inclusion/exclusion criteria, piloting of the selection criteria, independent screening by three authors, a data extraction form, and a dedicated threats-to-validity section. The paper also ships a public repository, which supports reproducibility. However, the significance is conditional on the completeness of the literature search; the search-string construction described in §4.1.2 has a structural flaw that directly affects which studies could be retrieved, and the paper does not report the provenance of the final 34 studies between the automated search and snowballing. This weakens confidence in the 'first comprehensive mapping study' claim and in the gap analysis built on the corpus.

major comments (3)
  1. [§4.1.2 and §4.1.3] The reported search string ($XR AND $XRacr) AND ($T OR $B), with $XRacr restricted to titles and $XR allowed in title or full text, requires every retrieved study to have both a full XR phrase and its acronym, and the acronym must appear in the title. This systematically excludes any relevant study whose title uses only the full phrase or only the acronym. This contradicts the search evaluation in §4.1.3: the test set includes Bierbaum et al. (PS9) with title 'Automated testing of virtual reality application interfaces' and Rafi et al. (PS23) with title 'PredART: Towards Automatic Oracle Prediction of Object Placements in Augmented Reality Testing', neither of which contains 'VR' or 'AR' in the title, so they cannot be matched by the stated query. Several final primary studies also have acronym-free titles (PS9, PS10, PS12, PS19), meaning they could only have been recovered by snowballing. The paper must either correct the reported search string, re-run the search with a disjunctive structure (full-term OR acronym), or provide a per-study provenance table and a recall analysis that justifies the current corpus.
  2. [§5 and Figure 5] The flow from 1167 initial studies to 135 after selection and then to 34 final primary studies is not fully auditable. The text states that backward snowballing identified 53 additional studies, but it does not report how many of the final 34 primary studies came from the automated search versus snowballing, whether the 53 were added before or after full-text screening, or how many of the 53 were ultimately included. Because snowballing (Stage 3) starts from the 135 studies retained after the automated search, it cannot recover relevant studies that are not cited by those retained studies. A PRISMA-style flow diagram with per-path counts and a per-study provenance table is needed to support the completeness claim and to allow replication.
  3. [§4.4] The threats-to-validity section acknowledges the single-database risk and the temporal bias of OpenAlex, but it does not acknowledge the structural limitation of the ANDed acronym requirement in the search string, which is a stronger and more specific threat to completeness. The statement that three iterations of snowballing 'significantly reduced the likelihood of missing important contributions' is only as strong as the starting set; if the automated search under-covers a sub-community whose papers are not co-cited with the retained studies, snowballing cannot compensate. The threats discussion should be revised to include this search-string limitation, and the 'first comprehensive' claim should be tempered to 'to the best of our knowledge' with an explicit statement of the residual completeness risk.
minor comments (5)
  1. [§4.1.2] Please specify the exact OpenAlex query fields used (e.g., title.search, fulltext.search) and the full query string, since OpenAlex's coverage of full text varies by source and this affects reproducibility.
  2. [§6.2.2] 'We want to know that multiple studies...' should read 'We note that multiple studies...'.
  3. [§5.1.1] Footnote 18 contains a typo: 'in their tile' should be 'in their title'.
  4. [§5.2.1] The word cloud in Figure 9 is difficult to audit and is not reproducible; a frequency table of test activities would be preferable for a mapping study.
  5. [§6.2.2] The statement about the Unity List dataset being no longer accessible is useful, but it would be clearer to state the access date and the URL checked, as done for other resources.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the mapping study is an external literature synthesis; no fitted parameter, self-referential derivation, or load-bearing self-citation appears.

full rationale

This paper makes no quantitative derivation from fitted parameters. Its central claim—being the first systematic mapping study of XR software testing—rests on a literature search and selection process applied to an external corpus of 34 primary studies, not on any result produced by the authors. The classification and data-extraction schemes (Sections 4.2.2 and 4.2.3) are qualitative synthesis procedures following published guidelines, and the findings in Section 5 are descriptive aggregations of the selected studies. The only self-citation in the reference list (Gu and Rojas 2023, cited in Section 2.2.1 as an example of scripted Android GUI testing frameworks) is illustrative and non-load-bearing. The search-string structure criticized in the skeptical note—requiring both an XR phrase and its acronym in titles—is a completeness threat acknowledged in Section 4.4, not a circularity: it affects whether relevant studies were missed, but it does not make the conclusions equivalent to the inputs. No step reduces to its own definition or to a citation by the same authors, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or invented entities appear in this review. The central claims rest on methodological assumptions about literature coverage, search operationalization, and classification reliability, all of which are disclosed in Sections 4.1 through 4.4.

assumptions (3)
  • domain assumption The OpenAlex database, together with three iterations of backward snowballing, provides sufficiently complete coverage of the XR software testing literature.
    Invoked in Section 4.1.4 and Section 4.4, where the authors acknowledge incomplete indexing and temporal bias. If this assumption fails, the 34-study corpus and the 'first mapping study' claim may be incomplete.
  • domain assumption The search string and PICOC facets operationalize the research questions so that relevant primary studies are retrieved.
    Stated in Section 4.1.2; the search was evaluated against a test set of eight papers, but this is a judgment about terminology coverage rather than a formal guarantee.
  • domain assumption Manual classification and data extraction by the authors, with consensus-based conflict resolution, reliably assign studies to topics, test facets, and research types.
    Described in Sections 4.2.2 and 4.2.3. The authors note that inter-rater agreement was not formally measured, so reliability rests on the authors' shared judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Software Testing for Extended Reality Applications: A Systematic Mapping Study." pith.science (2026). https://pith.science/paper/KEURYGFN

@misc{pith2026250108909,
  author       = {Pith},
  title        = {Pith review of: Software Testing for Extended Reality Applications: A Systematic Mapping Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KEURYGFN}},
  note         = {Machine review of arXiv:2501.08909}
}
read the original abstract

Extended Reality (XR) is an emerging technology spanning diverse application domains and offering immersive user experiences. However, its unique characteristics, such as six degrees of freedom interactions, present significant testing challenges distinct from traditional 2D GUI applications, demanding novel testing techniques to build high-quality XR applications. This paper presents the first systematic mapping study on software testing for XR applications. We selected 34 studies focusing on techniques and empirical approaches in XR software testing for detailed examination. The studies are classified and reviewed to address the current research landscape, test facets, and evaluation methodologies in the XR testing domain. Additionally, we provide a repository summarising the mapping study, including datasets and tools referenced in the selected studies, to support future research and practical applications. Our study highlights open challenges in XR testing and proposes actionable future research directions to address the gaps and advance the field of XR software testing.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

112 extracted references · 44 canonical work pages

  1. [1]

    write newline

    " write newline " cite write " FUNCTION editor.postfix editor num.names #1 > "( )" "( )" if FUNCTION editor.trans.postfix editor num.names #1 > "( )" "( )" if FUNCTION trans.postfix translator num.names #1 > "( )" "( )" if FUNCTION authors.editors.reflist.apa5 'field := 'dot := field num.names 'numnames := numnames 'format.num.names := format.num.names na...

  2. [2]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year eprint archive archivePrefix primaryClass adsurl adsnote version label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.sta...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap duplicate "7" = swap duplicate "8" = swap "9" = or or or or or or or or or FUNCTION n.separate 't := "" #0 'numnames := t empty not t #-1 #1 subs...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION output.doi doi empty skip "doi:" doi * "" * output if FUNCTION format.archive archivePrefix empty "" archivePrefix ":" * if FUNCTION format.primaryClass primaryClass empty "" " [" primaryClass * "] " * if FUNCTION format.eprint eprint empty "" archive empty " https://arxiv.org/abs/" eprint * " " * " ...

  5. [5]

    write newline

    " write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTION find.integer 't := #0 'int := int not t empty not and t #1 #1 substring 's :=...

  6. [6]

    write newline

    " write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTION find.integer 't := #0 'int := int not t empty not and t #1 #1 substring 's :=...

  7. [7]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter edition editor eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url doi volume year archivePrefix primaryClass eid adsurl adsnote version label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.sta...

  8. [8]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

Show all 112 references
  1. [9]

    Available from:

    ENTRY address assignee author booktitle chapter cartographer day edition editor howpublished institution inventor journal key keywords month note number organization pages part publisher school series title type volume word year eprint doi url lastchecked updated archive archi...

  2. [10]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  3. [11]

    Cambridge University Press, doi:10.1017/CBO9780511809163

    Ammann P, Offutt J (2008) Introduction to Software Testing. Cambridge University Press, doi:10.1017/CBO9780511809163

    2008 doi

  4. [12]

    In: 2019 21st Symposium on Virtual and Augmented Reality (SVR), pp 196--205, doi:10.1109/SVR.2019.00044

    Andrade SA, Nunes FLS, Delamaro ME (2019) Towards the systematic testing of virtual reality programs. In: 2019 21st Symposium on Virtual and Augmented Reality (SVR), pp 196--205, doi:10.1109/SVR.2019.00044

    2019

  5. [13]

    In: Symposium on Virtual and Augmented Reality (SVR), pp 57--66, doi:10.1109/SVR51698.2020.00024

    Andrade SA, Quevedo AJU, Nunes FLS, et al (2020) Understanding vr software testing needs from stakeholders’ points of view. In: Symposium on Virtual and Augmented Reality (SVR), pp 57--66, doi:10.1109/SVR51698.2020.00024

    2020

  6. [14]

    Software Testing, Verification and Reliability 33(8)

    Andrade SA, Nunes FLS, Delamaro ME (2023) Exploiting deep reinforcement learning and metamorphic testing to automatically test virtual reality applications. Software Testing, Verification and Reliability 33(8). doi:10.1002/stvr.1863

    2023 doi

  7. [15]

    eyeautomate: An experiment for the comparison of two generations of android gui testing

    Ardito L, Coppola R, Morisio M, et al (2019) Espresso vs. eyeautomate: An experiment for the comparison of two generations of android gui testing. In: Proc. of the 23rd Intl. Conf. on Evaluation and Assessment in Software Engineering. ACM, EASE '19, p 13–22, doi:10.1145/331900...

    2019

  8. [16]

    Journal of Cultural Heritage 26:101--108

    Barbieri L, Bruno F, Muzzupappa M (2017) Virtual museum system evaluation through user studies. Journal of Cultural Heritage 26:101--108. doi:https://doi.org/10.1016/j.culher.2017.02.005

    2017 doi

  9. [17]

    IEEE Transactions on Software Engineering 41(5):507--525

    Barr ET, Harman M, McMinn P, et al (2015) The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering 41(5):507--525. doi:10.1109/TSE.2014.2372785

    2015

  10. [19]

    Proc ACM Hum-Comput Interact 6

    B\" o rsting I, Heikamp M, Hesenius M, et al (2022) Software engineering for augmented reality - a research agenda. Proc ACM Hum-Comput Interact 6. doi:10.1145/3532205

    2022 doi

  11. [20]

    In: Intl

    Bouvier P, De Sorbier F, Chaudeyrac P, et al (2008) Cross benefits between virtual reality and games. In: Intl. Conf. and Industry Symposium on Computer Games, Animation, Multimedia, IPTV, Edutainment and Security (CGAT'08), doi:10.5176/978-981-08-8227-3_cgat08-26

    2008 doi

  12. [21]

    Proc of Eurographics Poitiers, France

    Breen D, Rose E, Whitaker R (2000) Interactive occlusion and collision of real and virtual objects in augmented reality. Proc of Eurographics Poitiers, France

    2000

  13. [22]

    Patient Education and Counseling 98

    Brown-Johnson C, Berrean B, Cataldo J (2015) Development and usability evaluation of the mhealth tool for lung cancer (mhealth tlc): A virtual world health game for lung cancer patients. Patient Education and Counseling 98. doi:10.1016/j.pec.2014.12.006

    2015 doi

  14. [23]

    IEEE Transactions on Dependable and Secure Computing 18(2):550--562

    Casey P, Baggili I, Yarramreddy A (2021) Immersive virtual reality attacks and the human joystick. IEEE Transactions on Dependable and Secure Computing 18(2):550--562. doi:10.1109/TDSC.2019.2907942

    2021

  15. [24]

    Computers in Biology and Medicine 95:43--54

    Charron O, Lallement A, Jarnet D, et al (2018) Automatic detection and segmentation of brain metastases on multimodal mr images with a deep convolutional neural network. Computers in Biology and Medicine 95:43--54. doi:10.1016/j.compbiomed.2018.02.004

    2018 doi

  16. [25]

    Procedia - Social and Behavioral Sciences 97:691--699

    Chen CJ, Lau SY, Chuah KM, et al (2013) Group usability testing of virtual reality-based learning environments: A modified approach. Procedia - Social and Behavioral Sciences 97:691--699. doi:10.1016/j.sbspro.2013.10.289

    2013 doi

  17. [26]

    Intl Journal of Frontiers in Engineering Technology 3(5)

    Cheng S, Qu H (2021) Key Issues of Real-time Collision Detection in Virtual Reality . Intl Journal of Frontiers in Engineering Technology 3(5). doi:10.25236/IJFET.2021.030505

    2021

  18. [27]

    The Computer Journal 64(5):826--841

    Corrêa CG, Delamaro ME, Chaim ML, et al (2020) Software testing automation of vr-based systems with haptic interfaces. The Computer Journal 64(5):826--841. doi:10.1093/comjnl/bxaa054

    2020 doi

  19. [28]

    Software Testing, Verification and Reliability 28(8)

    Corrêa Souza AC, Nunes FLS, Delamaro ME (2018) An automated functional testing approach for virtual reality applications. Software Testing, Verification and Reliability 28(8). doi:10.1002/stvr.1690

    2018 doi

  20. [29]

    In: Proc

    Davis S, Nesbitt K, Nalivaiko E (2014) A systematic review of cybersickness. In: Proc. of the 2014 Conference on Interactive Entertainment. ACM, IE2014, p 1–9, doi:10.1145/2677758.2677780

    2014

  21. [30]

    Frontiers in Robotics and AI 5

    Dey A, Billinghurst M, Lindeman RW, et al (2018) A Systematic Review of 10 Years of Augmented Reality Usability Studies : 2005 to 2014. Frontiers in Robotics and AI 5. doi:10.3389/frobt.2018.00037

    2018

  22. [31]

    Springer Intl

    Doerner R, Broll W, Grimm P, et al (eds) (2022) Virtual and Augmented Reality (VR/AR): Foundations and Methods of Extended Realities (XR) . Springer Intl. Publishing, doi:10.1007/978-3-030-79062-2

    2022 doi

  23. [32]

    In: Proc

    Emery V, Jacko J, Kongnakorn T, et al (2001) Identifying critical interaction scenarios for innovative user modeling. In: Proc. of the 1st Intl. Conf. on Universal Access in Human-Computer Interaction, pp 481--485

    2001

  24. [33]

    In: HCI Intl

    Figueira T, Gil A (2022) Youkai: A cross-platform framework for testing vr/ar apps. In: HCI Intl. 2022 -- Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence. Springer Nature Switzerland, pp 3--12, doi:10.1007/978-3-031-21707-4_1

    2022 doi

  25. [34]

    Information and Software Technology 55(8):1374--1396

    Garousi V, Mesbah A, Betin-Can A, et al (2013) A systematic mapping study of web application testing. Information and Software Technology 55(8):1374--1396. doi:10.1016/j.infsof.2013.02.006

    2013 doi

  26. [35]

    The Intl Journal of Robotics Research 32(11):1231--1237

    Geiger A, Lenz P, Stiller C, et al (2013) Vision meets robotics: The kitti dataset. The Intl Journal of Robotics Research 32(11):1231--1237. doi:10.1177/0278364913491297

    2013 doi

  27. [36]

    ACM Trans Comput-Hum Interact 26(3)

    Harms P (2019) Automated usability evaluation of virtual reality applications. ACM Trans Comput-Hum Interact 26(3). doi:10.1145/3301423

    2019 doi

  28. [37]

    In: 2013 IEEE Sixth Intl

    Herbold S, Harms P (2013) Autoquest -- automated quality engineering of event-driven software. In: 2013 IEEE Sixth Intl. Conf. on Software Testing, Verification and Validation Workshops, pp 134--139, doi:10.1109/ICSTW.2013.23

    2013 doi

  29. [38]

    Synthesis Lectures on Human-Centered Informatics 1:i--105

    Hertzum M (2020) Usability testing: A practitioner's guide to evaluating the user experience. Synthesis Lectures on Human-Centered Informatics 1:i--105. doi:10.2200/S00987ED1V01Y202001HCI045

    2020 doi

  30. [39]

    In: Proc

    Hesenius M, Griebe T, Gries S, et al (2014) Automating ui tests for mobile applications with formal gesture descriptions. In: Proc. of the 16th Intl. Conf. on Human-Computer Interaction with Mobile Devices & Services. ACM , MobileHCI '14, p 213–222, doi:10.1145/2628363.2628391

    2014

  31. [40]

    In: Intl

    Izuazu UU, Kim DS, Lee JM (2023) Unravelling the black box: Enhancing virtual reality network security with interpretable deep learning-based intrusion detection system. In: Intl. Conf. on Information and Communication Technology Convergence (ICTC), pp 928--931, doi:10.1109/IC...

    2023

  32. [41]

    Assembly Automation 41(1):89--105

    Jin Y, Geng J, He Z, et al (2021) A capsule-based collision detection approach of irregular objects in virtual maintenance. Assembly Automation 41(1):89--105. doi:10.1108/AA-12-2019-0224

    2021 doi

  33. [42]

    In: 2017 Intl

    Jung Sm, Oh SH, Whangbo Tk (2017) 360° stereo image based vr motion sickness testing system. In: 2017 Intl. Conf. on Emerging Trends & Innovation in ICT (ICEI), pp 150--153, doi:10.1109/ETIICT.2017.7977027

    2017

  34. [43]

    Sensors 22(4)

    Kamińska D, Zwoliński G, Laska-Leśniewicz A (2022) Usability testing of virtual reality applications—the pilot study. Sensors 22(4). doi:10.3390/s22041342

    2022 doi

  35. [44]

    Themes in Science and Technology Education 10(2):85--119

    Kavanagh S, Luxton-Reilly A, Wuensche B, et al (2017) A systematic review of virtual reality in education. Themes in Science and Technology Education 10(2):85--119

    2017

  36. [45]

    Applied Ergonomics 113:104107

    Kia K, Hwang J, Kim JH (2023) Effects of error rates and target sizes on neck and shoulder biomechanical loads during augmented reality interactions. Applied Ergonomics 113:104107. doi:10.1016/j.apergo.2023.104107

    2023

  37. [46]

    In: 2021 IEEE 4th Intl

    Kilger F, Kabil A, Tippmann V, et al (2021) Detecting and preventing faked mixed reality. In: 2021 IEEE 4th Intl. Conf. on Multimedia Information Processing and Retrieval (MIPR), pp 399--405, doi:10.1109/MIPR51284.2021.00074

    2021

  38. [47]

    In: Proc

    Kim HG, Baddar WJ, Lim Ht, et al (2017 a ) Measurement of exceptional motion in vr video contents for vr sickness assessment using deep convolutional autoencoder. In: Proc. of the 23rd ACM Symposium on Virtual Reality Software and Technology. ACM, VRST '17, doi:10.1145/3139131.3139137

    2017

  39. [48]

    Archives of Plastic Surgery 44(3):179--187

    Kim Y, Kim H, Kim YO (2017 b ) Virtual Reality and Augmented Reality in Plastic Surgery : A Review . Archives of Plastic Surgery 44(3):179--187. doi:10.5999/aps.2017.44.3.179

    2017 doi

  40. [49]

    Intl Journal of Human–Computer Interaction 36(10):893--910

    Kim YM, Rhiu I, Yun MH (2020) A systematic review of a virtual reality system from the perspective of user experience. Intl Journal of Human–Computer Interaction 36(10):893--910. doi:10.1080/10447318.2019.1699746

    2020

  41. [50]

    In: 2023 IEEE XVI Intl

    Kirayeva RR, Khafizov MR, Turdiev TT, et al (2023) Automated testing of functional requirements for virtual reality applications. In: 2023 IEEE XVI Intl. Scientific and Technical Conference Actual Problems of Electronic Instrument Engineering (APEIE), pp 1760--1764, doi:10.110...

    2023

  42. [51]

    Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep. EBSE-2007-01, Keele University

    2007

  43. [52]

    IEEE Transactions on Reliability 68(1):45--66

    Kong P, Li L, Gao J, et al (2019) Automated testing of android apps: A systematic literature review. IEEE Transactions on Reliability 68(1):45--66. doi:10.1109/TR.2018.2865733

    2019

  44. [53]

    In: Proc

    Kundu RK, Elsaid OY, Calyam P, et al (2023) Vr-lens: Super learning-based cybersickness detection and explainable ai-guided deployment in virtual reality. In: Proc. of the 28th Intl. Conf. on Intelligent User Interfaces. ACM, IUI '23, p 819–834, doi:10.1145/3581641.3584044

    2023

  45. [54]

    In: Proc

    Kuri M, Karre SA, Reddy YR (2021) Understanding software quality metrics for virtual reality products - a mapping study. In: Proc. of the Innovations in Software Engineering Conference (Formerly Known as India Software Engineering Conference). ACM , ISEC '21, doi:10.1145/34523...

    2021

  46. [55]

    ACM Trans Priv Secur 25(4)

    Lehman SM, Alrumayh AS, Kolhe K, et al (2022) Hidden in plain sight: Exploring privacy risks of mobile augmented reality applications. ACM Trans Priv Secur 25(4). doi:10.1145/3524020

    2022 doi

  47. [56]

    IEEE Transactions on Visualization and Computer Graphics 29(4):2102--2116

    Lehman SM, Elezovikj S, Ling H, et al (2023) Archie++ : A cloud-enabled framework for conducting ar system testing in the wild. IEEE Transactions on Visualization and Computer Graphics 29(4):2102--2116. doi:10.1109/TVCG.2022.3141029

    2023

  48. [57]

    Journal of Ambient Intelligence and Humanized Computing 4(1):17--26

    Lele A (2013) Virtual reality and its military utility. Journal of Ambient Intelligence and Humanized Computing 4(1):17--26. doi:10.1007/s12652-011-0052-4

    2013 doi

  49. [58]

    In: IEEE/ACM Intl

    Leykin A, Tuceryan M (2004) Automatic determination of text readability over textured backgrounds for augmented reality systems. In: IEEE/ACM Intl. Symposium on Mixed and Augmented Reality, pp 224--230, doi:10.1109/ISMAR.2004.22

    2004 doi

  50. [59]

    In: 2020 IEEE 31st Intl

    Li S, Wu Y, Liu Y, et al (2020) An exploratory study of bugs in extended reality applications on the web. In: 2020 IEEE 31st Intl. Symposium on Software Reliability Engineering (ISSRE), pp 172--183, doi:10.1109/ISSRE5003.2020.00025

    2020

  51. [60]

    Proc ACM Softw Eng 1(FSE)

    Li S, Gao C, Zhang J, et al (2024) Less cybersickness, please: Demystifying and detecting stereoscopic visual inconsistencies in virtual reality apps. Proc ACM Softw Eng 1(FSE). doi:10.1145/3660803

    2024 doi

  52. [61]

    ://arxiv.org/abs/2407.03037

    Liu Z, Li C, Chen C, et al (2024) Vision-driven automated mobile gui testing via multimodal large language model. ://arxiv.org/abs/2407.03037

    2024 arXiv

  53. [62]

    In: IEEE Intl

    Lu C, Shi J, Jia J (2013) Abnormal event detection at 150 fps in matlab. In: IEEE Intl. Conf. on Computer Vision, pp 2720--2727, doi:10.1109/ICCV.2013.338

    2013 doi

  54. [63]

    In: IEEE Computer Society Conf

    Mahadevan V, Li W, Bhalodia V, et al (2010) Anomaly detection in crowded scenes. In: IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, pp 1975--1981, doi:10.1109/CVPR.2010.5539872

    2010

  55. [64]

    Telemanipulator and Telepresence Technologies 2351

    Milgram P, Takemura H, Utsumi A, et al (1994) Augmented reality: A class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies 2351. doi:10.1117/12.197321

    1994 doi

  56. [65]

    i-com 22(3):175--192

    Minor S, Ketoma VK, Meixner G (2023) Test automation for augmented reality applications: a development process model and case study. i-com 22(3):175--192. doi:10.1515/icom-2023-0029

    2023 doi

  57. [66]

    In: 1st Intl

    Odeleye B, Loukas G, Heartfield R, et al (2021) Detecting framerate-oriented cyber attacks on user experience in virtual reality. In: 1st Intl. Workshop on Security for XR and XR for Security

    2021

  58. [67]

    In: Proc

    Paduraru C, Stefanescu A, Jianu A (2024) Unit test generation using large language models for unity game development. In: Proc. of the 1st ACM Intl. Workshop on Foundations of Applied Software Engineering for Games. ACM , FaSE4Games 2024, p 7–13, doi:10.1145/3663532.3664466

    2024

  59. [68]

    In: Research Challenges in Information Science

    Pastor Ric \'o s F (2022) Scriptless testing for extended reality systems. In: Research Challenges in Information Science. Springer Intl. Publishing, pp 786--794, doi:10.1007/978-3-031-05760-1_56

    2022 doi

  60. [69]

    Pastore F, Mariani L, Fraser G (2013) Crowdoracles: Can the crowd solve the oracle problem? In: 2013 IEEE Sixth Intl. Conf. on Software Testing, Verification and Validation, pp 342--351, doi:10.1109/ICST.2013.13

    2013 doi

  61. [70]

    In: Proc

    Petersen K, Feldt R, Mujtaba S, et al (2008) Systematic mapping studies in software engineering. In: Proc. of the 12th Intl. Conf. on Evaluation and Assessment in Software Engineering. BCS Learning & Development Ltd., EASE'08, p 68–77

    2008

  62. [71]

    Information and Software Technology 64:1--18

    Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology 64:1--18. doi:10.1016/j.infsof.2015.03.007

    2015 doi

  63. [72]

    In: Proc

    Politowski C, Gu\' e h\' e neuc YG, Petrillo F (2022) Towards automated video game testing: still a long way to go. In: Proc. of the 6th Intl. ICSE Workshop on Games and Software Engineering: Engineering Fun, Inspiration, and Motivation. ACM , GAS '22, p 37–43, doi:10.1145/352...

    2022

  64. [73]

    In: Intl

    Prasetya ISWB, Shirzadehhajimahmood S, Ansari SG, et al (2021) An agent-based architecture for ai-enhanced automated testing for xr systems, a short paper. In: Intl. Conf. on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, pp 213--217, doi:10.1109/ICSTW5...

    2021

  65. [74]

    ://arxiv.org/abs/2205.01833

    Priem J, Piwowar H, Orr R (2022) Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts. ://arxiv.org/abs/2205.01833

    2022 arXiv

  66. [75]

    Proc of the IEEE 107(4):651--666

    Qiao X, Ren P, Dustdar S, et al (2019) Web ar: A promising future for mobile augmented reality—state of the art, challenges, and insights. Proc of the IEEE 107(4):651--666. doi:10.1109/JPROC.2019.2895105

    2019

  67. [77]

    Qiu Z, Liu W, Feng H, et al (2024) Can large language models understand symbolic graphics programs? ://arxiv.org/abs/2408.08313

    2024 arXiv

  68. [78]

    CCF Transactions on Pervasive Computing and Interaction 4

    Qu C, Che X, Ma S, et al (2022) Bio-physiological-signals-based vr cybersickness detection. CCF Transactions on Pervasive Computing and Interaction 4. doi:10.1007/s42486-022-00103-8

    2022 doi

  69. [79]

    In: Proc

    Rafi T, Zhang X, Wang X (2023) Predart: Towards automatic oracle prediction of object placements in augmented reality testing. In: Proc. of the 37th IEEE/ACM Intl. Conf. on Automated Software Engineering. ACM , doi:10.1145/3551349.3561160

    2023

  70. [80]

    In: 2019 Intl

    Ramaseri Chandra AN, El Jamiy F, Reza H (2019) A review on usability and performance evaluation in virtual reality systems. In: 2019 Intl. Conf. on Computational Science and Computational Intelligence (CSCI), pp 1107--1114, doi:10.1109/CSCI49370.2019.00210

    2019

  71. [81]

    Jurnal Buana Informatika 14(01):11--19

    Richard Gunawan , Yohanes Priadi Wibisono , Clara Hetty Primasari , et al (2023) Blackbox Testing on Virtual Reality Gamelan Saron Using Equivalence Partition Method . Jurnal Buana Informatika 14(01):11--19. doi:10.24002/jbi.v14i01.6606

    2023 doi

  72. [82]

    Kennedy KSBNorman E

    Robert S. Kennedy KSBNorman E. Lane, Lilienthal MG (1993) Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The Intl Journal of Aviation Psychology 3(3):203--220. doi:10.1207/s15327108ijap0303\_3

    1993 doi

  73. [83]

    ://doi.org/10.48550/arXiv.2305.07842

    Roberts J (2023) The ar/vr technology stack: A central repository of software development libraries, platforms, and tools. ://doi.org/10.48550/arXiv.2305.07842

    2023 doi arXiv

  74. [84]

    In: 2017 ACM/IEEE Intl

    Rodriguez I, Wang X (2017) An empirical study of open source virtual reality software projects. In: 2017 ACM/IEEE Intl. Symposium on Empirical Software Engineering and Measurement (ESEM), pp 474--475, doi:10.1109/ESEM.2017.65

    2017 doi

  75. [85]

    In: Proc

    Rzig DE, Iqbal N, Attisano I, et al (2023) Virtual reality (vr) automated testing in the wild: A case study on unity-based vr applications. In: Proc. of the 32nd ACM SIGSOFT Intl. Symposium on Software Testing and Analysis. ACM , ISSTA 2023, p 1269–1281, doi:10.1145/3597926.3598134

    2023

  76. [86]

    In: 2018 10th Intl

    Sarupuri B, Hoermann S, Whitton MC, et al (2018) Lute: A locomotion usability test environmentfor virtual reality. In: 2018 10th Intl. Conf. on Virtual Worlds and Games for Serious Applications (VS-Games), pp 1--4, doi:10.1109/VS-Games.2018.8493432

    2018

  77. [87]

    In: 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp 0219--0226, doi:10.1109/UEMCON47517.2019.8992974

    Scheibmeir J, Malaiya YK (2019) Quality model for testing augmented reality applications. In: 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp 0219--0226, doi:10.1109/UEMCON47517.2019.8992974

    2019

  78. [88]

    In: 2020 4th Intl

    Sendari S, Firmansah A, Aripriharta (2020) Performance analysis of augmented reality based on vuforia using 3d marker detection. In: 2020 4th Intl. Conf. on Vocational Education and Training (ICOVET), pp 294--298, doi:10.1109/ICOVET50258.2020.9230276

    2020

  79. [89]

    In: Proc

    Su T, Meng G, Chen Y, et al (2017) Guided, stochastic model-based gui testing of android apps. In: Proc. of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM , p 245–256, doi:10.1145/3106237.3106298

    2017

  80. [90]

    Proc ACM Program Lang 5(OOPSLA)

    Su T, Yan Y, Wang J, et al (2021) Fully automated functional fuzzing of android apps for detecting non-crashing logic bugs. Proc ACM Program Lang 5(OOPSLA). doi:10.1145/3485533

    2021 doi

  81. [91]

    The Aeronautical Journal 124(1280):1615–1635

    Tadeja S, Seshadri P, Kristensson P (2020) Aerovr: An immersive visualisation system for aerospace design and digital twinning in virtual reality. The Aeronautical Journal 124(1280):1615–1635. doi:10.1017/aer.2020.49

    2020 doi

  82. [92]

    ://docs.unity3d.com/Manual/class-GameObject.html, accessed: 2024-08-13

    Technologies U (2024) Unity - manual: Gameobject. ://docs.unity3d.com/Manual/class-GameObject.html, accessed: 2024-08-13

    2024

  83. [93]

    In: Proc

    Thomas P, Spielman S, Craswell N, et al (2024) Large language models can accurately predict searcher preferences. In: Proc. of the 47th Intl. ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , SIGIR '24, p 1930–1940, doi:10.1145/3626772.3657707

    2024

  84. [94]

    Software Quality Journal 27(1):149--201

    Tramontana P, Amalfitano D, Amatucci N, et al (2019) Automated functional testing of mobile applications: A systematic mapping study. Software Quality Journal 27(1):149--201. doi:10.1007/s11219-018-9418-6

    2019 doi

  85. [95]

    In: RCIS Workshops

    Tramontana P, Luca MD, Fasolino AR (2022) An approach for model based testing of augmented reality applications. In: RCIS Workshops

    2022

  86. [96]

    IEEE Transactions on Services Computing 16(4):2559--2574

    Valluripally S, Frailey B, Kruse B, et al (2023) Detection of security and privacy attacks disrupting user immersive experience in virtual reality learning environments. IEEE Transactions on Services Computing 16(4):2559--2574. doi:10.1109/TSC.2022.3216539

    2023

  87. [97]

    Software Testing, Verification and Reliability 31(3):e1771

    Vos TEJ, Aho P, Pastor Ricos F, et al (2021) testar – scriptless testing through graphical user interface. Software Testing, Verification and Reliability 31(3):e1771. doi:10.1002/stvr.1771

    2021 doi

  88. [98]

    DrDobb's Journal 27(7):17--26

    Walsh AE (2002) Understanding scene graphs. DrDobb's Journal 27(7):17--26

    2002

  89. [99]

    In: Intl

    Wang X (2022) VRTest : An extensible framework for automatic testing of virtual reality scenes. In: Intl. Conf. on Software Engineering: Companion Proceedings (ICSE-Companion). ACM, pp 232--236, doi:10.1145/3510454.3516870

    2022

  90. [100]

    In: 2023 38th IEEE/ACM Intl

    Wang X, Rafi T, Meng N (2023) Vrguide: Efficient testing of virtual reality scenes via dynamic cut coverage. In: 2023 38th IEEE/ACM Intl. Conf. on Automated Software Engineering (ASE). IEEE Computer Society, pp 951--962, doi:10.1109/ASE56229.2023.00197

    2023

  91. [101]

    IEEE Computer Society, ://www.swebok.org

    Washizaki H (ed) (2024) Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. IEEE Computer Society, ://www.swebok.org

    2024

  92. [102]

    In: 2012 Intl

    Wei Z, Xinxin G (2012) The collision detection algorithm in virtual reality. In: 2012 Intl. Conf. on Computer Science and Electronics Engineering, pp 538--541, doi:10.1109/ICCSEE.2012.412

    2012 doi

  93. [103]

    Requir Eng 11:102--107

    Wieringa R, Maiden N, Mead N, et al (2006) Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. Requir Eng 11:102--107. doi:10.1007/s00766-005-0021-6

    2006 doi

  94. [104]

    In: Proc

    Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proc. of the 18th Intl. Conf. on Evaluation and Assessment in Software Engineering. ACM , EASE '14, doi:10.1145/2601248.2601268

    2014

  95. [105]

    Applied Sciences 13(11)

    Xu P, Sun Q (2023) Virtual reality collision detection based on improved ant colony algorithm. Applied Sciences 13(11). doi:10.3390/app13116366

    2023 doi

  96. [106]

    Brain Informatics 9(1):24

    Yang AHX, Kasabov N, Cakmak YO (2022) Machine learning methods for the study of cybersickness: A systematic review. Brain Informatics 9(1):24. doi:10.1186/s40708-022-00172-6

    2022 doi

  97. [107]

    In: Proc

    Yang X, Zhang X (2023) A study of user privacy in android mobile ar apps. In: Proc. of the 37th IEEE/ACM Intl. Conf. on Automated Software Engineering (ASE'22). ACM, doi:10.1145/3551349.3560512

    2023

  98. [108]

    In: Proc

    Yang X, Wang Y, Rafi T, et al (2024) Towards automatic oracle prediction for ar testing: Assessing virtual object placement quality under real-world scenes. In: Proc. of the 33rd ACM SIGSOFT Intl. Symposium on Software Testing and Analysis. ACM, ISSTA 2024, p 717–729, doi:10.1...

    2024

  99. [109]

    Journal of Systems and Software 117:334--356

    Zein S, Salleh N, Grundy J (2016) A systematic mapping study of mobile application testing techniques. Journal of Systems and Software 117:334--356. doi:10.1016/j.jss.2016.03.065

    2016 doi

  100. [110]

    In: Intl

    Zhang M, Zhou D, Lv C, et al (2014) Collision detection technology based on capsule model in virtual maintenance. In: Intl. Conf. on Reliability, Maintainability and Safety (ICRMS), pp 1150--1155, doi:10.1109/ICRMS.2014.7107384

    2014

  101. [111]

    IEEE Transactions on Software Engineering 49(3):991--1026

    Zhang X, Tao J, Tan K, et al (2023) Finding critical scenarios for automated driving systems: A systematic mapping study. IEEE Transactions on Software Engineering 49(3):991--1026. doi:10.1109/TSE.2022.3170122

    2023

  102. [112]

    In: 2019 34th IEEE/ACM Intl

    Zheng Y, Xie X, Su T, et al (2019) Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In: 2019 34th IEEE/ACM Intl. Conf. on Automated Software Engineering (ASE), pp 772--784, doi:10.1109/ASE.2019.00077

    2019

  103. [113]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year archivePrefix primaryClass adsurl adsnote version label extra.labe...

  104. [114]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.