pith. sign in

arxiv: 2606.27506 · v1 · pith:HPMBO4OCnew · submitted 2026-06-25 · 💻 cs.SE

Contextual Associations Between Webpage Elements for Web Accessibility: An Empirical Study

Kishan Rakesh , Shiyi Wei This is my paper

Pith reviewed 2026-06-29 01:08 UTC · model grok-4.3

classification 💻 cs.SE
keywords web accessibilityaccessibility treelink predictionscreen readerscontextual associationsgraph neural networksempirical study
0
0 comments X

The pith

Human-perceived contextual associations between webpage elements can be recovered from the accessibility tree using link prediction and generalize across websites.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether machine learning can identify which surrounding webpage elements provide meaningful context for ambiguous accessible names such as 'Read more'. It frames the task as link prediction on graphs constructed from accessibility trees, enriched with spatial and semantic features drawn from the DOM and CSS. A dataset of human annotations is built across 35 websites with three annotators per page, and four models are compared to heuristic baselines under leave-one-site-out cross-validation. The work tests whether associations learned on some sites transfer to unseen sites. If the models succeed, screen readers could automatically surface relevant context without site-specific rules or manual markup.

Core claim

The study claims that contextual associations between webpage elements, as perceived by humans, can be recovered from the accessibility tree using link prediction techniques, with models that generalize across different websites when evaluated under leave-one-site-out cross-validation on a dataset of 35 sites.

What carries the argument

The accessibility tree represented as a graph augmented with spatial and semantic features from the DOM and CSS, on which link prediction models (MLP, GCN, GAT, SEAL) operate to recover human-annotated contextual associations.

If this is right

  • Models trained on a subset of websites will predict associations on unseen websites at rates above the heuristic baselines.
  • The approach will identify surrounding elements that make ambiguous accessible names interpretable without requiring changes to existing web markup.
  • A human-annotated dataset of contextual associations will be released for use in accessibility research and tool development.
  • Graph-based link prediction provides a scalable alternative to manual or rule-based selection of contextual elements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the associations generalize, browser extensions or screen reader software could apply the models in real time to improve navigation on arbitrary pages.
  • The method might extend to other accessibility scenarios where element relationships matter, such as voice navigation or ARIA label suggestions.
  • Cross-site performance would imply that common web design patterns produce detectable regularities in accessibility trees.

Load-bearing premise

The human annotations collected from three independent annotators per page accurately reflect the contextual associations that screen-reader users would perceive as meaningful when navigating by element list.

What would settle it

If the machine learning models fail to outperform the two heuristic baselines by a statistically significant margin on Hit@K and MRR under leave-one-site-out cross-validation in the full 35-site study, the claim that associations can be recovered and generalized would not hold.

read the original abstract

[Context] Screen reader users navigating webpages by element list often encounter accessible names such as "Read more" that are valid under the W3C Accessible Name and Description Computation specification but uninterpretable in isolation. The surrounding elements that would make these names meaningful exist in the page but are not linked to the target by any mechanism. No prior work has empirically studied how to select which surrounding elements are contextually relevant to a given target. [Objective] This registered report investigates whether human-perceived contextual associations between webpage elements can be recovered from the accessibility tree using link prediction, and whether the learned associations generalize across websites. [Method] We will construct a dataset of human-annotated contextual associations on 35 websites, stratified across the Tranco top-million list, with three independent annotators per page. Each page is represented as a graph derived from its accessibility tree, augmented with spatial and semantic features from the DOM and CSS. We compare four machine learning models (MLP, GCN, GAT, and SEAL) against two heuristic baselines under leave-one-site-out cross-validation with a pre-registered statistical framework, using Hit@K and MRR. [Results] We have conducted a five-site author-annotated pilot study to establish the pipelines and parameterize the power simulation, with pilot Hit@10 ranging from 0.16 to 0.85 across four learned models and 0.08 to 0.30 across two heuristic baselines. The final results will be reported after the planned experiments and analyses are completed. [Conclusion] The study contributes a human-annotated dataset of contextual associations on webpages, an empirical evaluation of link prediction for context selection on accessibility-tree graphs, and a cross-site generalization analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. This registered report describes a planned study to test whether human-perceived contextual associations between webpage elements can be recovered via link prediction on graphs constructed from accessibility trees (augmented with DOM/CSS features) and whether the resulting models generalize across sites. The design calls for a 35-site dataset with three independent annotators per page, comparison of MLP/GCN/GAT/SEAL against two heuristics under leave-one-site-out CV, and evaluation by Hit@K/MRR; a five-site author-annotated pilot has already been run to validate pipelines and inform power analysis.

Significance. If the pre-registered analyses confirm the hypotheses, the work would supply a publicly useful annotated dataset together with the first systematic evidence that standard graph link-prediction methods can identify contextually relevant elements for screen-reader navigation, directly addressing an accessibility gap not covered by existing W3C specifications.

major comments (1)
  1. [Method] Method section on annotation protocol: the central claim that the collected labels represent 'human-perceived contextual associations' for screen-reader users rests on the untested assumption that annotations from three independent (non-screen-reader) annotators will match the associations that actual screen-reader users would find meaningful; this assumption is load-bearing for both the recovery and generalization claims yet receives no validation plan or inter-rater reliability criterion tied to accessibility expertise.
minor comments (1)
  1. [Abstract] Abstract: the reported pilot Hit@10 ranges (0.16–0.85 for learned models) are useful but would benefit from per-model values so readers can assess variability before the full results appear.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the detailed and constructive comment on the annotation protocol. We address it point-by-point below and agree that the manuscript requires revision to better justify and qualify the use of non-expert annotators.

read point-by-point responses

  1. Referee: [Method] Method section on annotation protocol: the central claim that the collected labels represent 'human-perceived contextual associations' for screen-reader users rests on the untested assumption that annotations from three independent (non-screen-reader) annotators will match the associations that actual screen-reader users would find meaningful; this assumption is load-bearing for both the recovery and generalization claims yet receives no validation plan or inter-rater reliability criterion tied to accessibility expertise.

    Authors: We agree that the assumption is load-bearing and currently lacks explicit validation against screen-reader users. The registered protocol defines the labels as human-perceived contextual associations (with screen-reader navigation as the motivating use case), and the three-annotator design follows standard practice for establishing reliable human judgments in similar annotation tasks. To strengthen the manuscript we will (1) add inter-rater reliability reporting (Fleiss' kappa) as a pre-registered analysis, (2) explicitly discuss the assumption and its implications as a limitation in the revised Limitations section, and (3) outline a feasible post-hoc validation step (e.g., expert review of a small subset) that could be performed without altering the core data-collection plan. These changes will be incorporated in the next version. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The manuscript is a registered report for a planned empirical study. It describes graph construction from accessibility trees, augmentation with DOM/CSS features, and application of standard off-the-shelf link-prediction models (MLP, GCN, GAT, SEAL) versus heuristics under leave-one-site-out CV. No equations, derivations, or fitted parameters are defined in terms of the target metrics (Hit@K, MRR). The pilot is reported separately and does not serve as the basis for any claimed generalization. No self-citation chains, ansatzes, or renamings appear. The central claim remains an empirical question to be tested on the full dataset.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim depends on the assumption that accessibility-tree graphs plus spatial and semantic features are sufficient to encode human contextual judgments, and that the planned human annotations constitute reliable ground truth.

free parameters (1)
  • model hyperparameters
    Hyperparameters of the four ML models will be chosen or tuned during the study.
axioms (2)
  • domain assumption The accessibility tree derived from the DOM and CSS captures the structural information relevant to screen-reader users.
    This underpins the construction of the input graph for link prediction.
  • domain assumption Human annotations from three independent raters per page provide a stable ground-truth signal for contextual relevance.
    The entire evaluation treats these annotations as the target that models must recover.

pith-pipeline@v0.9.1-grok · 5841 in / 1380 out tokens · 41218 ms · 2026-06-29T01:08:04.635113+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 7 canonical work pages

  1. [1]

    2026 , howpublished =

    The. 2026 , howpublished =

    2026

  2. [2]

    Screen Reader User Survey \#10 Results , year =

  3. [3]

    2401.16450 , archivePrefix=

    Calista Huang and Alyssa Ma and Suchir Vyasamudri and Eugenie Puype and Sayem Kamal and Juan Belza Garcia and Salar Cheema and Michael Lutz , year=. 2401.16450 , archivePrefix=

    work page arXiv

  4. [4]

    Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility , articleno =

    Fathallah, Nadeen and Hern\'. Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility , articleno =. 2025 , isbn =. doi:10.1145/3663547.3746360 , abstract =

    work page doi:10.1145/3663547.3746360 2025

  5. [5]

    Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility , articleno =

    Gubbi Mohanbabu, Ananya and Pavel, Amy , title =. Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility , articleno =. 2024 , isbn =. doi:10.1145/3663548.3675658 , abstract =

    work page doi:10.1145/3663548.3675658 2024

  6. [6]

    Sunkara, Srinivas and Wang, Maria and Liu, Lijuan and Baechler, Gilles and Hsiao, Yu-Chung and Chen, Jindong and Sharma, Abhanshu and Stout, James W. W. Towards Better Semantic Understanding of Mobile Interfaces. Proceedings of the 29th International Conference on Computational Linguistics. 2022

    2022

  7. [7]

    2019 , volume=

    Jaume, Guillaume and Kemal Ekenel, Hazim and Thiran, Jean-Philippe , booktitle=. 2019 , volume=

    2019

  8. [8]

    2023 , month = sep, howpublished =

    Barcik, Gabriel and Tran, Duc-Hieu , title =. 2023 , month = sep, howpublished =

    2023

  9. [9]

    Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , booktitle=

    Shuyan Zhou and Frank F. Xu and Hao Zhu and Xuhui Zhou and Robert Lo and Abishek Sridhar and Xianyi Cheng and Tianyue Ou and Yonatan Bisk and Daniel Fried and Uri Alon and Graham Neubig , booktitle=. 2024 , url=

    2024

  10. [10]

    Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =

    Deng, Xiang and Gu, Yu and Zheng, Boyuan and Chen, Shijie and Stevens, Samuel and Wang, Boshi and Sun, Huan and Su, Yu , title =. Proceedings of the 37th International Conference on Neural Information Processing Systems , articleno =. 2023 , publisher =

    2023

  11. [11]

    2003 , month = nov, url =

    Cai, Deng and Yu, Shipeng and Wen, Ji-Rong and Ma, Wei-Ying , institution =. 2003 , month = nov, url =

    2003

  12. [12]

    Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages =

    Xie, Mulong and Xing, Zhenchang and Feng, Sidong and Xu, Xiwei and Zhu, Liming and Chen, Chunyang , title =. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages =. 2022 , isbn =. doi:10.1145/3540250.3549138 , abstract =

    work page doi:10.1145/3540250.3549138 2022

  13. [13]

    Visual Intelligence , year =

    Shi, Danqing and Cui, Weiwei and Huang, Danqing and Zhang, Haidong and Cao, Nan , title =. Visual Intelligence , year =. doi:10.1007/s44267-023-00010-1 , url =

    work page doi:10.1007/s44267-023-00010-1

  14. [14]

    AI , VOLUME =

    Prazina, Irfan and Pozderac, Damir and Okanović, Vensada , TITLE =. AI , VOLUME =. 2025 , NUMBER =

    2025

  15. [15]

    Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =

    Jiang, Yue and Zhou, Changkong and Garg, Vikas and Oulasvirta, Antti , title =. Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , articleno =. 2024 , isbn =. doi:10.1145/3613904.3642822 , abstract =

    work page doi:10.1145/3613904.3642822 2024

  16. [16]

    ACM Trans

    Ang, Gary and Lim, Ee-Peng , title =. ACM Trans. Interact. Intell. Syst. , month = sep, articleno =. 2023 , issue_date =. doi:10.1145/3578522 , abstract =

    work page doi:10.1145/3578522 2023

  17. [17]

    Enhancing UI Tests Robustness With Graph Convolutional Networks

    Ayli, Maroun and Bakouny, Youssef and Seifeddine, Hani and Jalloul, Nader and Kilany, Rima. Enhancing UI Tests Robustness With Graph Convolutional Networks. Management of Digital EcoSystems. 2025

    2025

  18. [18]

    Web Image Context Extraction with Graph Neural Networks and Sentence Embeddings on the DOM Tree

    Dang, Chen and Randrianarivo, Hicham and Fournier-S'niehotta, Rapha \"e l and Audebert, Nicolas. Web Image Context Extraction with Graph Neural Networks and Sentence Embeddings on the DOM Tree. Machine Learning and Principles and Practice of Knowledge Discovery in Databases. 2021

    2021

  19. [19]

    Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages =

    Zhang, Muhan and Chen, Yixin , title =. Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages =. 2018 , publisher =

    2018

  20. [20]

    and Ying, Rex and Leskovec, Jure , title =

    Hamilton, William L. and Ying, Rex and Leskovec, Jure , title =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =. 2017 , isbn =

    2017

  21. [21]

    International Conference on Learning Representations , year=

    Graph Attention Networks , author=. International Conference on Learning Representations , year=

  22. [22]

    2024 , month = dec, type =

    Charles Adams and Alastair Campbell and Rachael Bradley Montgomery and Michael Cooper and Andrew Kirkpatrick , title =. 2024 , month = dec, type =

    2024

  23. [23]

    , title =

    Apple Inc. , title =

  24. [24]

    2018 , month = dec, type =

    Joanmarie Diggs and Bryan Garaventa and Michael Cooper , title =. 2018 , month = dec, type =

    2018

  25. [25]

    Playwright: Fast and Reliable End-to-End Testing for Modern Web Apps , howpublished =

  26. [26]

    International Conference on Learning Representations , year=

    Semi-Supervised Classification with Graph Convolutional Networks , author=. International Conference on Learning Representations , year=

  27. [27]

    and Ba, Jimmy , title =

    Kingma, Diederik P. and Ba, Jimmy , title =. International Conference on Learning Representations (ICLR) , year =

  28. [28]

    Journal of Machine Learning Research , year =

    Asela Gunawardana and Guy Shani , title =. Journal of Machine Learning Research , year =

  29. [29]

    2023 , month = jun, type =

    Accessible Rich Internet Applications (. 2023 , month = jun, type =

    2023

  30. [30]

    Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation , booktitle =

    Le Pochat, Victor and Van Goethem, Tom and Tajalizadehkhoob, Samaneh and Korczy. Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation , booktitle =. 2019 , publisher =

    2019

  31. [31]

    and Welling, Max , title =

    Kipf, Thomas N. and Welling, Max , title =. NIPS Workshop on Bayesian Deep Learning , year =

  32. [32]

    Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , year =

    Hu, Weihua and Fey, Matthias and Zitnik, Marinka and Dong, Yuxiao and Ren, Hongyu and Liu, Bowen and Catasta, Michele and Leskovec, Jure , title =. Advances in Neural Information Processing Systems 33 (NeurIPS 2020) , year =

    2020

  33. [33]

    Time for a Change: A Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis , journal =

    Benavoli, Alessio and Corani, Giorgio and Dem. Time for a Change: A Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis , journal =

  34. [34]

    and Krippendorff, Klaus , title =

    Hayes, Andrew F. and Krippendorff, Klaus , title =. Communication Methods and Measures , volume =