REVIEW 3 major objections 5 minor 36 references
WebXAII: an open-source web framework to study human-XAI interaction
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read WebXAII lets researchers define complete human-XAI experiments in JSON configuration files that can be shared for replication.
desk verdict A useful, honestly-scoped open-source framework for human-XAI experiments; the 'full protocols' claim outruns the linear, precomputed architecture, but the artifact and the published-protocol reproduction are real. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a composite, view-driven architecture defined entirely in JSON. Four modules—Connection, Protocol, Experiment, and Task—manage successive stages of a participant's journey, while four views—Questionnaire, Instruction, Instance decision, and Feedback—handle the concrete interactions, with the Instance decision view as the fundamental unit where participants act on a model prediction and its explanations. The architecture works because a protocol is a declarative document, not a program: nesting modules, randomizing instance order within tasks, and binding participant groups to different protocol configurations all happen through the JSON description. Its boundary condition is equally important: because AI and XAI outputs are precomputed externally, the framework never needs to run a model during the experiment.
What would settle it
Try to encode the reproduced study's fifth experiment in WebXAII: participants choose whether to see an explanation, and that choice alters both the task and the reward. If no JSON configuration can represent this conditional flow—as the paper states in Section 5—then the framework's claim of embodying full protocols does not extend to choice-dependent designs; a successful implementation after a tree-based protocol extension would be the decisive countertest.
Extended reading notes
Core claim
The paper's central claim is that the visible surface of human-XAI studies decomposes into a small set of reusable building blocks: modules for connection, protocol, experiment, and task, plus views for questionnaires, instructions, instance decisions, and feedback. Arranging these blocks in the nested order Protocol > Experiment > Task > Instance decision is sufficient to represent a representative published protocol, including a between-group design with four experimental arms, which the implementation captures as four distinct protocols sharing the same structure. The framework deliberately keeps all machine-learning predictions and XAI explanations precomputed, presenting them as text or images rather than calling live models. The authors state this as an explicit design assumption in Section 3.1 and show through the reproduction that a rigorous modern study fits inside that boundary.
Load-bearing premise
The load-bearing assumption is that every implementable human-XAI protocol fits a linear Protocol > Experiment > Task > Instance decision structure with all AI and XAI outputs precomputed, which excludes protocols in which a participant's earlier choice changes what comes later, such as the reproduced study's fifth experiment that the paper concedes it cannot implement.
Editorial extensions
If this is right
- A study's complete experimental interface can be published as one JSON file alongside its results, allowing others to replay the exact participant experience.
- Multi-arm studies are handled by defining several protocol configurations and associating each participant group to one through the connection module, with no per-arm code.
- Since all AI and XAI outputs are precomputed, deploying a study does not require hosting live machine-learning models during data collection.
- Adding a new kind of question or display can be done as a new generic view without redesigning the protocol layer, a path the paper lists as planned future work.
- Protocols where a participant's choice changes what comes later, such as the fifth study of the reproduced work, currently fall outside the framework's capacities.
Reading between the lines
- Editorial inference: if WebXAII is adopted by multiple groups, shared JSON protocols could become a de facto standard for reporting human-XAI experiments, making replication cheap enough to become routine rather than rare.
- Editorial inference: the framework's modules are not intrinsically tied to XAI, so the same JSON-protocol pattern could generalize to any human-subject study that presents precomputed stimuli and records choices, such as perception or judgment experiments.
- Editorial inference: the paper demonstrates one representative protocol, not coverage across the field; a natural next test would be to encode a sample of recently published human-XAI protocols and measure what fraction fit the linear, precomputed structure without extension.
- Editorial inference: if the list-based protocol structure is replaced by the tree-based structure the paper proposes, branching designs now excluded would become representable, and the framework would cover choice-dependent experiments such as the unreproduced fifth study.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces WebXAII, an open-source web framework for implementing interfaces for human-XAI interaction studies. The framework defines experimental protocols as JSON configurations composed of hierarchical modules (Connection, Protocol, Experiment, Task) and views (Questionnaire, Instruction, Instance decision, Feedback). The authors claim that full experimental protocols can be embodied without bespoke programming, and they validate this by re-implementing the protocol of Study 1 of Vasconcelos et al. (2023), a maze-based human-XAI study, using configuration files and screenshots. The paper also reviews related tools such as z-Tree, oTree, LimeSurvey, and ARDAS, and discusses planned extensions including dynamic model interactions and tree-structured protocols.
Significance. If validated, WebXAII addresses a real reproducibility gap: human-XAI study interfaces are typically bespoke and rarely shared. The contribution is concrete and checkable: the code and configuration files are publicly available, the architecture is clearly described, and the case study shows that at least one published protocol's structure can be represented with generic views. The JSON-configurable design is a useful step toward shareable, reproducible study interfaces. The main weakness is that the demonstration stops at static screenshots and generated stimuli rather than an end-to-end participant session, so the central capability claim is only partially evidenced. The paper honestly acknowledges the main architectural limitation, but that limitation conflicts with the unqualified 'full experimental protocols' claim in the abstract.
major comments (3)
- [Abstract; Section 3.1; Section 5] The abstract and Section 6 assert that WebXAII 'can embody full experimental protocols,' but Section 3.1 constrains protocols to a linear Protocol > Experiment > Task > Instance-decision structure with all AI and XAI outputs precomputed (assumptions 1-4), and Section 5 explicitly concedes that protocols in which participant choices affect later stages, such as Vasconcelos et al.'s Study 5, cannot be implemented. The single case study reproduces only Study 1, which lies entirely inside the supported class, so the boundary of the claimed generality is never tested. The 'full' claim should be qualified to the supported class, or additional cases that exercise the boundary should be demonstrated.
- [Section 4.2] The validation does not demonstrate an end-to-end participant-facing run. Section 4.2 states that the authors did not have access to the original maze stimuli and 'only use a small sample of maze images which we generated,' and the evidence presented consists of screenshots (Figures 3 and 4) and configuration files. No deployed session with participants is reported, and there is no verification that responses are logged, that timing constraints fire, or that randomization works as specified. Since the central claim is that the framework can effectively embody protocols, the authors should either report a deployed test session (even a pilot with a few users) or explicitly narrow the claim to structural reproduction.
- [Section 4.1; Section 3.1, Connection module] The implementation of Vasconcelos et al.'s Study 1 requires assigning participants to four experimental conditions (Protocols A-D), but Section 3.1 says only that 'each user identifier is associated with a specific protocol beforehand' and the case study does not describe how random assignment to conditions is realized. If participant-to-condition randomization is performed outside WebXAII, this should be stated; if WebXAII is expected to support it, this is a missing feature that affects the claim of embodying the protocol.
minor comments (5)
- [Throughout] The manuscript contains small language and formatting issues: 'javascript' should be 'JavaScript', 'configurations files' should be 'configuration files', and 'open-source software' should be 'open-source software' where it appears with the indefinite article.
- [Figure 2 caption] The caption uses 'Instructions View' while the body text and Figure 1 use 'Instruction view'; please standardize the terminology.
- [Section 4.2] The sentence 'we implement the structure of the study, but we only use a small sample of maze images which we generated' is an important limitation and should be reflected in the abstract or conclusion, not only inside the case-study section.
- [Section 3.1] The phrase 'which can then be transparent to the end-participant' is ambiguous; consider replacing 'transparent' with 'invisible' or 'not directly visible.'
- [References] Several DOIs have inconsistent spacing or formatting (e.g., references [15], [20], [21], and [23]); these should be normalized according to the journal style.
Circularity Check
No circularity found: WebXAII's central claim is validated against an external published protocol, and its acknowledged Study 5 limitation is a scope concern rather than a circular derivation.
full rationale
WebXAII is a software systems paper rather than a paper that derives a quantitative prediction from fitted inputs. Its central claim is that the framework can embody full experimental protocols, and this is demonstrated by reproducing the protocol of an external published study, Vasconcelos et al. [35]. The framework's architecture is defined in Section 3.1 as a Protocol > Experiment > Task > Instance-decision hierarchy with precomputed AI and XAI outputs, and Section 4 instantiates that architecture for Vasconcelos et al.'s Study 1. The benchmark is external, the configuration files and source code are publicly available on GitHub, and no load-bearing claim is justified by a self-citation chain. The only self-referential aspect is the generic one shared by all framework papers: the framework's own component structure is used to encode the target protocol. That is an implementation choice, not a derivation of the conclusion from itself. The paper explicitly concedes in Section 5 that Vasconcelos et al.'s Study 5, which requires participant choices to influence later protocol stages, cannot currently be implemented. This weakens the unqualified wording 'full experimental protocols' in the abstract, but that is an overclaim or scope issue, not circularity. No specific reduction of a claimed capability to an input, no fitted parameter renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled in via citation could be identified. Under the hard rule requiring a quotable reduction to establish circularity, the appropriate finding is no significant circularity, with score 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The Protocol > Experiment > Task > Instance decision hierarchy is sufficient to represent human-XAI experimental protocols.
- domain assumption All AI and XAI computations can be performed before the experiment runs.
- ad hoc to paper Vasconcelos et al.'s first study is representative of state-of-the-art human-XAI protocols.
invented entities (1)
-
WebXAII framework
independent evidence
Cite this review
Pith. "Pith review of WebXAII: an open-source web framework to study human-XAI interaction." pith.science (2026). https://pith.science/paper/NP6X4HJR
@misc{pith2026250614777,
author = {Pith},
title = {Pith review of: WebXAII: an open-source web framework to study human-XAI interaction},
year = {2026},
howpublished = {\url{https://pith.science/paper/NP6X4HJR}},
note = {Machine review of arXiv:2506.14777}
}
read the original abstract
This article introduces WebXAII, an open-source web framework designed to facilitate research on human interaction with eXplainable Artificial Intelligence (XAI) systems. The field of XAI is rapidly expanding, driven by the growing societal implications of the widespread adoption of AI (and in particular machine learning) across diverse applications. Researchers who study the interaction between humans and XAI techniques typically develop ad hoc interfaces in order to conduct their studies. These interfaces are usually not shared alongside the results of the studies, which limits their reusability and the reproducibility of experiments. In response, we design and implement WebXAII, a web-based platform that can embody full experimental protocols, meaning that it can present all aspects of the experiment to human participants and record their responses. The experimental protocols are translated into a composite architecture of generic views and modules, which offers a lot of flexibility. The architecture is defined in a structured configuration file, so that protocols can be implemented with minimal programming skills. We demonstrate that WebXAII can effectively embody relevant protocols, by reproducing the protocol of a state-of-the-art study of the literature.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
V. Hassija, V. Chamola, A. Mahapatra, A. Singal, D. Goel, K. Huang, S. Scardapane, I. Spinelli, M. Mahmud, A. Hussain, Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence, Cognitive Computation (2024). doi: 10.1007/s12559-023-10179-8
-
[2]
D. Minh, H. X. Wang, Y. F. Li, T. N. Nguyen, Explainable artificial intelligence: a comprehensive review, Artificial Intelligence Review (2022). doi:10.1007/s10462-021-10088-y
-
[3]
Z. Bylinskii, L. Herman, A. Hertzmann, S. Hutka, Y. Zhang, Towards Better User Studies in Computer Graphics and Vision, Foundations and Trends in Computer Graphics and Vision (2023). doi:10.1561/0600000106
-
[4]
R. Fok, D. S. Weld, In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making, AI Magazine (2024). doi: 10.1002/aaai.12182
-
[5]
A. Šarčević, D. Pintar, M. Vranić, A. Krajna, Cybersecurity Knowledge Extraction Using XAI, Applied Sciences (2022). doi:10.3390/app12178669
-
[6]
B. Fresz, E. Dubovitskaya, D. Brajovic, M. F. Huber, C. Horz, How Should AI Decisions Be Explained? Requirements for Explanations from the Perspective of European Law, Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2024). doi:10.1609/aies.v7i1.31648
- [7]
-
[8]
R. Müller, How Explainable AI Affects Human Performance: A Systematic Review of the Be- havioural Consequences of Saliency Maps, International Journal of Human–Computer Interaction (2025). doi:10.1080/10447318.2024.2381929
arXiv 2025
Show all 36 references
- [9]
-
[10]
Zhang, P
Y. Zhang, P. Tiňo, A. Leonardis, K. Tang, A survey on neural network interpretability, IEEE Transactions on Emerging Topics in Computational Intelligence (2021) 726–742. doi:10.1109/ TETCI.2021.3100641
2021
- [11]
- [12]
- [13]
- [14]
-
[15]
S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, W. Samek, On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation, PloS ONE (2015). doi: 10. 1371/journal.pone.0130140
2015
-
[16]
Achtibat, M
R. Achtibat, M. Dreyer, I. Eisenbraun, S. Bosse, T. Wiegand, W. Samek, S. Lapuschkin, From attribution maps to human-understandable explanations through concept relevance propagation, Nature Machine Intelligence (2023). doi:10.1038/S42256-023-00711-8
2023 doi
- [17]
-
[18]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization, International Journal of Computer Vision (2020). doi:10.1007/s11263-019-01228-7
2020 doi
-
[19]
Chattopadhay, A
A. Chattopadhay, A. Sarkar, P. Howlader, V. N. Balasubramanian, Grad-CAM++: Generalized gradient-based visual explanations for deep convolutional networks, in: WACV, 2018. doi: 10. 1109/WACV.2018.00097
2018
- [20]
-
[21]
Zhang, F
H. Zhang, F. Torres, R. Sicre, Y. Avrithis, S. Ayache, Opti-CAM: Optimizing saliency maps for interpretability, Computer Vision and Image Understanding (2024). doi:10.1016/j.cviu.2024. 104101
2024 doi
- [22]
- [23]
- [24]
- [25]
- [26]
- [27]
-
[28]
Chromik, M
M. Chromik, M. Schuessler, A Taxonomy for Human Subject Evaluation of Black-Box Explanations in XAI., ExSS-ATEC@IUI (2020). URL: https://ceur-ws.org/Vol-2582/paper9.pdf
2020
-
[29]
van der Waa, E
J. van der Waa, E. Nieuwburg, A. Cremers, M. Neerincx, Evaluating XAI: A comparison of rule- based and example-based explanations, Artificial Intelligence (2021). doi: 10.1016/j.artint. 2020.103404
2021
- [30]
-
[31]
Fischbacher, z-Tree: Zurich toolbox for ready-made economic experiments, Experimental Economics (2007)
U. Fischbacher, z-Tree: Zurich toolbox for ready-made economic experiments, Experimental Economics (2007). doi:10.1007/s10683-006-9159-4
2007 doi
-
[32]
D. L. Chen, M. Schonger, C. Wickens, oTree—An open-source platform for laboratory, online, and field experiments, Journal of Behavioral and Experimental Finance (2016). doi: 10.1016/j.jbef. 2015.12.001
2016 doi
-
[33]
D. A. Quispe G, F. Rajabiyazdi, G. A. Jamieson, A Machine Learning-Based Micro-World Platform for Condition-Based Maintenance, in: 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2020. doi:10.1109/SMC42975.2020.9283448
2020
-
[34]
Gentile, B
D. Gentile, B. Donmez, G. A. Jamieson, Human performance consequences of normative and contrastive explanations: An experiment in machine learning for reliability maintenance, Artificial Intelligence (2023). doi:10.1016/j.artint.2023.103945
2023
- [35]
-
[36]
Buçinca, M
Z. Buçinca, M. B. Malaya, K. Z. Gajos, To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making, Proceedings of the ACM on Human- Computer Interaction (2021). doi:10.1145/3449287
2021 doi
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.