Pith. sign in

REVIEW 3 major objections 4 minor 28 references

TESTQUEST: A Web Gamification Tool to Improve Locators and Page Objects Quality

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TESTQUEST applies gamification to locator robustness and Page Object implementation, claiming to be the first tool to do so, and rewards only changes that pass test execution.

desk verdict A novel, concrete gamification plugin for web test locators, undermined by an unvalidated fragility heuristic that drives all rewards. read the letter →

arxiv 2505.24756 v1 pith:PASMM7PX submitted 2025-05-30 cs.SE

Dario Olianas , Diego Clerissi , Maurizio Leotta , Filippo Ricca This is my paper
classification cs.SE
keywords WebtestingGamificationLocatorrobustnessPageObjectpatternTestfragilityIDEpluginAchievementsDailies
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents TESTQUEST, an IDE plugin that adds game mechanics to web testing to push developers toward more robust locator choices and better-structured Page Objects. The authors claim this is the first time gamification has been applied specifically to locator robustness or to implementing the Page Object pattern. TESTQUEST scans the test suite, computes a fragility score for each locator from a set of literature-derived best practices, assigns daily tasks and achievements, and rewards users only when the improved locators or Page Objects pass execution. If the approach works, it would turn the tedious maintenance of web test artifacts into an ongoing, rewarded activity and reduce the manual repair effort that locator breakage currently causes.

What carries the argument

The machinery is a fragility-scoring algorithm combined with a task-driven gamification loop. Each locator is assigned a coefficient from 0 to 1: different locator types receive different base coefficients, and XPath locators receive an additional penalty computed from the kinds of predicates present and the locator's length, following the same philosophy as an existing XPath-generation algorithm. These scores feed a Fragility window that ranks all locators by predicted breakage risk, and they drive the targeted assignment of dailies, so that tasks point at actual weaknesses in the suite. The loop closes with execution listeners: a rewarded change only counts when the test that exercises the modified locator or Page Object passes. This ensures that gamified progress is tied to validated, not just syntactic, improvements.

What would settle it

A controlled study where locators scored as low-fragility and high-fragility are tracked across real page revisions: if high-scoring locators break no more often than low-scoring ones, or if users who complete TESTQUEST's dailies show no reduction in breakage or repair effort compared with a control group, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that embedding a gamification framework directly into the web testing environment can motivate testers to produce more resilient locators and cleaner Page Objects, and that no prior tool has applied gamification to either of these two targets. TESTQUEST operationalises this by scoring every locator in the project with a fragility value from 0 to 1 based on best practices such as preferring ID and XPath locators with meaningful predicates, avoiding absolute XPaths, and keeping locators short and readable; Page Objects are checked against practices such as not exposing locator details, not placing assertions in Page Object methods, and using inheritance and return types to model navigation. The plugin then assigns fifty dailies and twenty-nine achievements, in random, targeted, or inclusive modes, and only counts changes as progress once the containing test executes successfully. The intended consequence is that testers are guided, through immediate feedback and rewards, to the same best practices that automated locator generation and repair tools try to enforce.

Load-bearing premise

The load-bearing premise is that the fragility score computed from static features of a locator reliably predicts whether that locator will break when the web page changes.

Editorial extensions

If this is right

  • If TESTQUEST works as intended, web testers will be guided, task by task, toward locators that survive page changes, reducing the maintenance burden of test suites.
  • The targeted daily mode ties each assigned task to a detected weakness in the project, so rewards correspond to fixing actual locator or Page Object issues.
  • The requirement that changes pass test execution before earning rewards means gamified progress cannot be gained by cosmetic edits that leave tests broken.
  • The fragility window gives an at-a-glance, continuously updated ranking of locator risk, which itself may change how testers decide what to refactor.
  • This positions locator robustness and Page Object design as the next frontier for gamification in software testing, after unit and GUI testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step would be to calibrate the fragility weights against observed breakage data from real web applications, since the paper does not disclose or validate those weights.
  • A controlled experiment comparing test breakage rates and repair effort between testers using TESTQUEST and testers following the same best-practice checklist without the game layer would isolate the effect of gamification.
  • Because the inclusive mode already inspects the DOM for untested elements, TESTQUEST could grow into a coverage-driven tool that suggests new tests for uncovered page elements, not just improvements to existing locators.
  • The same task-driver pattern could be extended to other test-quality smells, such as overly long test workflows, which the paper names as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces TESTQUEST, an IntelliJ IDEA plugin that applies gamification to web testing with the aim of improving locator robustness and Page Object quality. The tool extracts locators and Page Objects from a test suite, evaluates them against a set of literature-derived best practices (Table I), computes a fragility score for each locator, and assigns daily tasks and achievements intended to guide users toward those best practices. The authors claim that this is the first application of gamification to locator robustness and Page Object implementation. The paper describes the tool's architecture, its two main GUI windows, the daily assignment modes (RANDOM, TARGETED, INCLUSIVE), and gives two usage examples. No empirical evaluation is presented; the conclusion lists a future experiment with students and professionals as next steps.

Significance. If the central claims were substantiated, TESTQUEST would be a novel contribution: existing gamified web testing tools (e.g., GIPGUT, GERRY) do not target locator robustness or Page Object structure, so the niche is real. The paper also has strengths in presentation: the architecture is clearly described, the set of 50 dailies and 29 achievements is concrete, and the authors provide a public repository link, which supports reproducibility. However, the significance is currently limited by the absence of any empirical validation of the tool's effectiveness or of its underlying fragility model, and the manuscript's claims in the Abstract and Section III go beyond what is demonstrated.

major comments (3)
  1. [Abstract and Section III] The central effectiveness claim—that TESTQUEST 'improves test robustness' (Abstract) and 'motivates users to develop more resilient locators' (Section III)—is not supported by any empirical data in the manuscript. There is no user study, no comparison against a non-gamified workflow, and no before-and-after measurement of locator fragility or breakage. Section V explicitly defers such an experiment to future work. As it stands, the claims in the Abstract and Section III should be tempered to describe a proposed tool, or the paper should include at least a pilot evaluation.
  2. [Section IV] The fragility score is the primary reward signal in TARGETED mode and in the Fragility window, yet its computation is undisclosed and unvalidated. The text states that the algorithm 'assigns a different fragility coefficient to different types of locators' and, for XPath, computes a coefficient 'based on the types of predicates used in the XPath, and its length,' inspired by Sidereal [7], but no weights are given. Without the weights or a validation against observed breakage, the ordering induced by the score might not correspond to the empirical probability that a locator breaks under DOM evolution. If the ordering is wrong, completing dailies and achievements could actively reward changes that do not make tests more robust. The authors should release the coefficients and validate the scoring against a real breakage benchmark, or at least against the Sidereal/ROBULA+ datasets.
  3. [Sections III and IV] There is a circularity concern: the dailies, achievements, and the fragility score are all derived from the same set of best practices in Table I. The tool therefore measures success by its own definition of quality rather than by an external criterion. No independent evidence is provided that locators satisfying L1–L6 break less often in practice, nor that Page Objects conforming to P1–P6 reduce maintenance effort beyond what is already known from the cited literature. The paper should either cite direct empirical evidence for each Table I rule or add a validation showing that the fragility score predicts actual breakage.
minor comments (4)
  1. [Title and Abstract] The name is written inconsistently: 'TEST QUEST' in the title and 'TESTQUEST' in the abstract and body; please unify the spelling.
  2. [Section IV] The sentence 'Although it has several differences, the algorithm was inspired by the XPath generation algorithm used by Sidereal [7]' is grammatically awkward; consider rephrasing, e.g., 'The algorithm was inspired by Sidereal's XPath generation algorithm [7], though with several differences.'
  3. [Section IV] The example daily 'Convert the following absolute XPath locator to relative' should read '...to a relative one' for clarity, though the meaning is still clear.
  4. [Figure 2] Figure 2's label 'User Info panel (label 1 in Figure 2)' is referenced in the text, but the reproduction quality of the figures is not verifiable; please ensure the labels are legible in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TESTQUEST's fragility score is an explicit, literature-based heuristic, and no empirical prediction or derivation is claimed that would reduce to its own inputs.

full rationale

The paper does not derive a result or make an empirically validated prediction. The fragility coefficient is an explicitly stated heuristic ('strictly based on the best practices listed in Table I') and is used to generate dailies and achievements from the same best-practice list. While this means the tool rewards compliance with its own quality definition, the paper makes no claim that this constitutes evidence of external effectiveness; instead, it explicitly defers user experiments to future work ('conduct an experiment involving students and professionals to evaluate the effectiveness'). The Sidereal reference is a design inspiration from prior same-group work, but it is not load-bearing: no conclusion in this paper rests on Sidereal's correctness. Therefore, the central concern is an unvalidated heuristic and a validation gap, which is a threat to validity rather than circularity under the stated criteria.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central claim rests on three unvalidated links: the best-practice list defines quality, the fragility score operationalizes it from static features, and gamification motivates developers to act on it. The fragility coefficient weights are the main free parameter: they are not disclosed, not calibrated to breakage data, and they drive every score and reward. No new physical entities are introduced; the fragility index is a new metric whose validity is not independently established.

free parameters (1)
  • Fragility coefficient weights for locator types and XPath predicates = Unspecified
    Section IV states that the algorithm assigns different fragility coefficients to locator types and, for XPath, to predicate classes and length. These values are not disclosed and are not calibrated against actual breakage data, yet they drive every fragility score and gamification reward.
assumptions (3)
  • domain assumption Compliance with the L1-L6 locator practices and P1-P6 Page Object practices predicts test robustness.
    Section III's Table I treats these practices as ground truth for quality. They are drawn from cited literature but are not independently validated in this paper.
  • domain assumption Static fragility features, namely locator type, XPath predicates, and length, predict the probability of breakage during web application evolution.
    Section IV computes the fragility coefficient from these static features, but no empirical link to real breakage is shown.
  • domain assumption Gamification elements such as points, dailies, and achievements increase developer engagement and motivate refactoring behavior.
    Section II presents this as a general background premise, and the paper assumes it transfers to locator and Page Object maintenance without local evidence.
invented entities (1)
  • Fragility index (0 to 1)
    purpose: Quantifies the likelihood that a locator will break during software evolution and serves as the core reward and ranking signal.
    The index is defined entirely by the paper's own heuristic rules and is not validated against observed locator breakage. It is both the target metric and the basis for gamified incentives, so its validity is critical and unproven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TESTQUEST: A Web Gamification Tool to Improve Locators and Page Objects Quality." pith.science (2026). https://pith.science/paper/PASMM7PX

@misc{pith2026250524756,
  author       = {Pith},
  title        = {Pith review of: TESTQUEST: A Web Gamification Tool to Improve Locators and Page Objects Quality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PASMM7PX}},
  note         = {Machine review of arXiv:2505.24756}
}
read the original abstract

Web applications play a crucial role in our daily lives, making it essential to employ testing methods that ensure their quality. Typically, Web testing automation frameworks rely on locators to interact with the graphical user interface, acting as connection points to the elements on a Web page. Nevertheless, locators are widely recognized as a major vulnerability in Web testing, as they are highly sensitive to the frequent changes in Web page structures caused by rapid software evolution. The adoption of the Page Object pattern to separate test logic from structural layout - supporting code reuse and maintainability - has generally led to more robust test cases. However, their implementation is a manually intensive task, and even automated support may require manual realignment efforts. Although gamification strategies have recently been integrated into the Web testing process to boost user engagement, using tasks and rewards aligned with testing activities, they have not yet been employed to enhance the robustness of locators and support the implementation of Page Objects. In this paper, we introduce TESTQUEST, a tool designed to improve test robustness by applying gamification to locators and Page Objects, boosting user engagement while guiding them toward the adoption of best practices.

Figures

Figures reproduced from arXiv: 2505.24756 by the authors.

Figure 1
Figure 1. The TESTQUEST workflow and architecture. action, which adds the option to activate the plugin in the target test project IDE2 . Initially, any test-related file within the target project is retrieved, relying on extractors to extract data about locators and Page Objects. From a programmatic perspective, in TESTQUEST the concept of locator is strongly inspired by the structure supported by Selenium WebDriver [3], whi… view at source ↗
Figure 2
Figure 2. TESTQUEST Gamification Window [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. TESTQUEST Fragility Window. scores of modified locators, or collecting the Page Objects failing the best practices of Table I) and the outcomes of test executions, eventually reactivating the whole process sketched in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 23 canonical work pages

  1. [7]

    SIDEREAL: Statistical adaptive generation of robust locators for Web testing,

    M. Leotta, F. Ricca, and P. Tonella, “SIDEREAL: Statistical adaptive generation of robust locators for Web testing,” Journal of Software: Testing, Verification and Reliability (STVR) , vol. 31, 2021. [Online]. Available: https://doi.org/10.1002/stvr.1767

    doi:10.1002/stvr.1767 2021

  2. [1]

    Comparing the maintainability of Selenium WebDriver test suites employing different locators: A Case Study,

    M. Leotta, D. Clerissi, F. Ricca, and C. Spadaro, “Comparing the maintainability of Selenium WebDriver test suites employing different locators: A Case Study,” in Proceedings of 1st International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation (JAMAICA). ACM, 2013, pp. 53–58. [Online]. Available: https://doi.org/10.1145/24892...

    work page arXiv 2013

  3. [2]

    Challenges of End- to-End Testing with Selenium WebDriver and How to Face Them: A Survey,

    M. Leotta, B. García, F. Ricca, and J. Whitehead, “Challenges of End- to-End Testing with Selenium WebDriver and How to Face Them: A Survey,” in Proceedings of 16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2023) . IEEE, 2023, pp. 339–350

    work page 2023

  4. [3]

    [Online]

    Selenium WebDriver Documentation. [Online]. Available: https: //www.selenium.dev/documentation/webdriver/

    work page

  5. [4]

    Guess the state: Exploiting determinism to improve GUI exploration efficiency,

    D. Clerissi, G. Denaro, M. Mobilio, and L. Mariani, “Guess the state: Exploiting determinism to improve GUI exploration efficiency,” IEEE Transactions on Software Engineering (TSE) , no. 01, pp. 1–18, 2024

    work page 2024

  6. [5]

    Why do record/replay tests of Web applications break?

    M. Hammoudi, G. Rothermel, and P. Tonella, “Why do record/replay tests of Web applications break?” in 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST) . IEEE, 2016, pp. 180–190

    work page 2016

  7. [6]

    ROBULA+: An algorithm for generating robust XPath locators for Web testing,

    M. Leotta, A. Stocco, F. Ricca, and P. Tonella, “ROBULA+: An algorithm for generating robust XPath locators for Web testing,” Journal of Software: Evolution and Process (JSEP) , vol. 28, no. 3, pp. 177–204,

    work page

  8. [8]

    Generating and selecting resilient and maintainable locators for Web automated testing,

    V . Nguyen, T. To, and G.-H. Diep, “Generating and selecting resilient and maintainable locators for Web automated testing,” Software Testing, Verification and Reliability (STVR) , vol. 31, no. 3, p. e1760, 2021

    work page 2021

Show all 28 references
  1. [9]

    M. Fowler. (2013) Page Object. [Online]. Available: https: //martinfowler.com/bliki/PageObject.html

    2013

  2. [10]

    A family of experiments to assess the impact of Page Object pattern in Web test suite development,

    M. Leotta, M. Biagiola, F. Ricca, M. Ceccato, and P. Tonella, “A family of experiments to assess the impact of Page Object pattern in Web test suite development,” in Proceedings of 13th IEEE International Conference on Software Testing, Verification and Validation (ICST) . IEE...

    2020

  3. [11]

    Gamification in software testing: A characterization study,

    G. M. de Jesus, F. C. Ferrari, D. de Paula Porto, and S. C. P. F. Fabbri, “Gamification in software testing: A characterization study,” in Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAST), 2018, pp. 39–48

    2018

  4. [12]

    Can gamification help in software testing education? Findings from an empirical study,

    R. Blanco, M. Trinidad, M. J. Suarez-Cabal, A. Calderón, M. Ruiz, and J. Tuya, “Can gamification help in software testing education? Findings from an empirical study,” Journal of Systems and Software (JSS) , vol. 200, p. 111647, 2023

    2023

  5. [13]

    A review on tools, mechanics, benefits, and challenges of gamified software testing,

    T. Fulcini, R. Coppola, L. Ardito, and M. Torchiano, “A review on tools, mechanics, benefits, and challenges of gamified software testing,” ACM Computing Surveys, vol. 55, no. 14s, pp. 1–37, 2023

    2023

  6. [14]

    A gaming quest to improve Web locators robustness,

    D. Clerissi, M. Leotta, and F. Ricca, “A gaming quest to improve Web locators robustness,” in Proceedings of the 3rd ACM International Workshop on Gamification in Software Development, Verification, and Validation (GAMIFY), 2024, pp. 10–17

    2024

  7. [15]

    Gamifica- tion: Using game-design elements in non-gaming contexts,

    S. Deterding, M. Sicart, L. Nacke, K. O’Hara, and D. Dixon, “Gamifica- tion: Using game-design elements in non-gaming contexts,” in CHI’11 Extended Abstracts on Human Factors in Computing Systems , 2011, pp. 2425–2428

    2011

  8. [16]

    COLOR: Correct locator recommender for broken test scripts using various clues in Web application,

    H. Kirinuki, H. Tanno, and K. Natsukawa, “COLOR: Correct locator recommender for broken test scripts using various clues in Web application,” in Proceedings of the 26th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2019, pp. 310–320

    2019

  9. [17]

    Towards the generation of robust E2E test cases in template-based Web applications,

    A. R. Fasolino and P. Tramontana, “Towards the generation of robust E2E test cases in template-based Web applications,” in 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) . IEEE, 2022, pp. 104–111

    2022

  10. [18]

    Incremental Web application testing using Page Object,

    B. Yu, L. Ma, and C. Zhang, “Incremental Web application testing using Page Object,” in 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb). IEEE, 2015, pp. 1–6

    2015

  11. [19]

    APOGEN: Automatic Page Object generator for Web testing,

    A. Stocco, M. Leotta, F. Ricca, and P. Tonella, “APOGEN: Automatic Page Object generator for Web testing,” Software Quality Journal (SQJ), vol. 25, no. 3, pp. 1007–1039, 2017. [Online]. Available: https://doi.org/10.1007/s11219-016-9331-9

    2017 doi

  12. [20]

    ASSESSOR: A PO-based WebDriver test suites generator from Selenium IDE recordings,

    M. Leotta, A. Molinari, and F. Ricca, “ASSESSOR: A PO-based WebDriver test suites generator from Selenium IDE recordings,” in Proceedings of 15th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2022, pp. 389–399. [Online]. Available: htt...

    2022

  13. [21]

    On effectiveness and efficiency of gamified exploratory GUI testing,

    R. Coppola, T. Fulcini, L. Ardito, M. Torchiano, and E. Alègroth, “On effectiveness and efficiency of gamified exploratory GUI testing,” IEEE Transactions on Software Engineering (TSE) , 2023

    2023

  14. [22]

    GERRY: A gamified browser tool for GUI testing,

    G. Garaccione, T. Fulcini, and M. Torchiano, “GERRY: A gamified browser tool for GUI testing,” in Proceedings of the 1st International Workshop on Gamification of Software Development, Verification, and Validation (GAMIFY), 2022, pp. 2–9

    2022

  15. [23]

    Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin,

    G. Garaccione, T. Fulcini, P. Stefanut Bodnarescul, R. Coppola, and L. Ardito, “Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin,” in Proceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments (IDE) , 2024, pp. 76–80

    2024

  16. [24]

    Towards predicting fragility in End- to-End Web tests,

    S. Di Meglio and L. L. L. Starace, “Towards predicting fragility in End- to-End Web tests,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE) , 2024, pp. 387–392

    2024

  17. [25]

    Guidelines for GUI testing maintenance: A linter for test smell detection,

    T. Fulcini, G. Garaccione, R. Coppola, L. Ardito, and M. Torchiano, “Guidelines for GUI testing maintenance: A linter for test smell detection,” in Proceedings of the 13th International Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST) , 2022, pp. 17–24

    2022

  18. [26]

    [Online]

    Page Object Models. [Online]. Available: https://www .selenium.dev/ documentation/test_practices/encouraged/page_object_models/

  19. [27]

    Testing Web applications with state objects,

    A. Van Deursen, “Testing Web applications with state objects,” Commu- nications of the ACM , vol. 58, no. 8, pp. 36–43, 2015

    2015

  20. [2016]

    Available: https://doi .org/10.1002/smr.1771

    [Online]. Available: https://doi .org/10.1002/smr.1771

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.