REVIEW 3 major objections 4 minor 40 references
Integrating Human Feedback into a Reinforcement Learning-Based Framework for Adaptive User Interfaces
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Tying UI adaptation to a per-user reinforcement learning agent improves satisfaction and engagement.
desk verdict Genuinely useful empirical work on per-user RLHF for UI adaptation, undermined by an abstract that claims more than the design can show. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a dual-source reward model: the reinforcement learning agent's reward is the sum of a baseline predictive HCI model (general engagement prediction) and a personalized reward modifier, a fully connected multilayer perceptron trained on the participant's pairwise preferences over 64 adaptation video clips. This per-user modifier shifts the policy so each agent is unique; agents are trained with GA3C, an asynchronous actor-critic method. The evaluation machinery is a balanced two-treatment, two-domain crossover design analyzed with a linear mixed model that includes period, sequence, and carryover-with-sequence, treating subject as a random effect.
What would settle it
Run the same experiment with a third condition: an adaptive interface using the same RL pipeline but with only the predictive HCI reward and no personalized preference modifier; if participants rate this no-feedback adaptive arm as high as the personalized-feedback arm, the central claim about human feedback is falsified.
Extended reading notes
Core claim
The paper's claim is that a UI adaptation system that learns a unique reinforcement-learning policy for each individual, using that person's stated preferences as a reward modifier, produces a significantly better user experience than a fixed, non-adaptive interface. The empirical evidence is two linear mixed models in which the non-adaptive condition is estimated 0.97 points lower on satisfaction (95% CI [-1.20, -0.74], p < 0.001) and 0.23 points lower on engagement (95% CI [-0.41, -0.04], p = 0.016); domain is not significant and the design shows no carryover. The paper interprets this as a step beyond its earlier general human-feedback model, which failed to reach significance, and as evidence that personalization at the level of individual reward models matters.
Load-bearing premise
The claim that human feedback itself drives the UX gains assumes that the adaptive and non-adaptive conditions differ only in the personalized feedback, yet no adaptive-without-feedback arm was run and group is confounded with carryover.
Editorial extensions
If this is right
- Adaptive interfaces trained with a per-user reward modifier can be expected to outperform static interfaces on satisfaction and engagement, at least in short controlled tasks.
- The absence of a significant domain effect suggests the benefit transfers between e-learning and trip-planning, at least for the adaptation actions studied.
- Individual differences dominate the model (ICC 0.76 for satisfaction and 0.55 for engagement), implying that user-specific random variation carries much of the observed variance.
- The technique effect on satisfaction (-0.97 on a 10-point scale) is larger relative to its scale than the effect on engagement (-0.23 on a 5-point scale), so personalized adaptation may shape perceived satisfaction more strongly than engagement.
Reading between the lines
- Editorial inference: because the design has no adaptive-without-feedback control arm, the strongest statement the data license is that RL-based personalized adaptation beats static; isolating the contribution of human feedback itself would require a third arm with the same RL adaptation but no personalized reward modifier.
- Editorial inference: the per-user training cost is substantial (roughly one million steps per user over three days on a high-end GPU), so practical deployment would likely need shared or cluster-based preference models; the paper itself lists cluster-based models as future work.
- Editorial inference: the pairwise video-comparison setup is an offline preference elicitation; a natural next test is whether feedback collected during live interaction, rather than from pre-generated clips, strengthens or changes the observed effects.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper extends an existing RL-based framework for adaptive user interfaces (AUIs) by adding a personalized human-feedback module: users provide offline pairwise preference judgments on UI adaptation clips, a per-user reward-modifier MLP is trained on these judgments, and a GA3C RL agent is trained for each user with a dual reward combining a predictive HCI model and the personalized modifier. The authors then conduct a within-subject crossover experiment with 33 participants across two domains (trip planning and e-learning), comparing an adaptive UI with human feedback against a non-adaptive UI. Linear mixed models on QUIS satisfaction and UES engagement scores show significantly higher satisfaction and engagement for the adaptive condition, with no significant domain or group effects. The abstract and conclusions claim that incorporating human feedback into RL-driven adaptations significantly enhances user experience.
Significance. If the stated causal claim were supported, this would be a meaningful contribution: it would be one of the first empirical demonstrations of RLHF applied to adaptive user interfaces, with per-user agents trained from individual preference data and evaluated in a real user study. The manuscript has several strengths: it provides replication links for the framework and experimental materials, uses standardized questionnaires with reported Cronbach's alpha (QUIS 0.97, UES 0.90), checks LMM assumptions (Shapiro-Wilk and Levene's tests), and uses a balanced crossover design. It is also not circular: the outcome questionnaires were not used as training signals for the reward models. However, the central causal claim about the specific contribution of human feedback is not supported by the experimental design, because the design lacks an adaptive-without-feedback control arm. This is a load-bearing issue for the paper's main contribution.
major comments (3)
- [Abstract; Sections 4.4 and 5.2] The central claim, stated in the abstract and repeated in Sections 5.3 and 7, is that 'incorporating human feedback into RL-driven adaptations significantly enhances UX.' The experiment, however, compares only two conditions: Adaptive-with-human-feedback and Non-Adaptive. There is no Adaptive-without-feedback arm, nor an Adaptive-with-generic-feedback arm. Therefore the significant LMM estimates (satisfaction: -0.97, p < 0.001; engagement: -0.23, p = 0.016) can only establish that the combined feedback-augmented adaptive system outperforms a static interface. They cannot isolate the contribution of human feedback, which could plausibly arise from RL adaptation itself, from the offline preference session, or from novelty effects. The paper's own hypotheses H_n1/H_n2 are framed as comparing 'AUIs that use predictive HCI models with personalized human feedback' against non-adaptive UIs, so the statistical test matches the narrower claim, not the abstract's causal one. The prior study [15], which compared adaptive systems with and without a general feedback model, is a separate experiment with different procedures, tasks, and participant populations, and its feedback contrast was not statistically significant; it is not a valid control. To support the abstract claim, the design would need a third condition (e.g., adaptive RL without human feedback, or adaptive with a non-personalized feedback model) under identical conditions in the same experiment. Absent that, the manuscript must be revised to state the supported claim: the proposed feedback-augmented adaptive framework improves UX relative to non-adaptive UIs, with the specific contribution of the feedback mechanism remaining an open question.
- [Section 5.2 and Section 4.4.3] The paper's conclusion that 'there is no carryover effect' is too strong. The LMM includes group as a fixed effect, but group is completely confounded with period, treatment order, and the treatment-period interaction. The non-significance of the group coefficients does not constitute a test for carryover; it only indicates no detectable difference between the four sequence groups. A proper carryover analysis would either model the treatment-by-period interaction explicitly or analyze first-period data only. With only 33 participants, the test for group effects is likely underpowered, so the absence of significance provides little evidence for the absence of carryover. This does not necessarily invalidate the main technique effect, but the claim that carryover was ruled out is not established by the reported analysis.
- [Section 5.3] The discussion attributes the significant improvements in this study to the individualized preference models, contrasting it with the non-significant results in prior work [15]. This cross-experiment comparison is not supported by the data: the two studies differ in multiple respects (participant recruitment, experimental procedure, task domains, and the exact system version), and the current study lacks the control condition needed to separate the effect of individualization from the effect of having feedback at all, or from RL adaptation itself. The statement in Section 5.3 that the improvements 'emphasize the effectiveness of personalized adaptation strategies' therefore goes beyond what the experiment can establish. The discussion should be reworded to present the individualized-feedback mechanism as a design choice whose incremental benefit is not directly tested.
minor comments (4)
- [Throughout] There are multiple typos and grammatical issues, including 'leaning process' (Abstract), 'adaption' (Abstract), 'continuosly' (Section 1), and 'a most 120 minutes' (Section 4.6.3). A thorough proofreading pass is needed.
- [Section 4.4 vs Section 4.6] Section 4.4 describes three sessions (User Feedback, RL Agent Training, User Interaction), but Section 4.6.3 says 'each participant attended two sessions in total.' This is inconsistent: the RL agent training is evidently an offline process that does not require the participant, but the text should clarify that the participant attends two sessions while the overall methodology has three phases.
- [Table 4] Table 4 labels the residual variance as 'α²', which is an unusual notation; the standard symbol for residual variance in LMM notation is σ². Please correct this for clarity.
- [Section 4.7] The analysis plan states that 'group or sequence (which was confounded with carryover and treatment*period interaction)' was included as a fixed factor, but it does not explain how the authors would distinguish these effects if the group term had been significant. This is related to the major comment on carryover; the text should be clearer about the limitations of the design.
Circularity Check
No significant circularity: the empirical UX comparison is self-contained and not forced by the reward-model inputs.
full rationale
The paper's central claim is an empirical comparison between an adaptive UI trained with personalized human feedback and a non-adaptive UI, with UX measured by post-task QUIS and UES questionnaires. The outcome questionnaires were not used to fit the reward models: the preference model was trained offline on pairwise video comparisons (Sections 4.4.1 and 4.6.1), and the per-user RL agents were trained separately (Sections 4.4.2 and 4.6.2). The LMM results in Section 5.2 do not reduce by construction to the reward model, the HCI model, or the preference model, since no equation maps those model weights to the questionnaire scores. The self-citations to [16] and [15] supply the base framework and a baseline predictive HCI model, but they do not assert the empirical UX improvement, and the [15] model is a separate prior model trained on data from 25 users external to this study's 33 participants. The absence of an adaptive-without-feedback control is a potential internal-validity or confounding concern about the abstract's causal attribution, not a circularity of definition or a fitted-input-called-prediction step. No specific circular reduction can be exhibited from the paper's text, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- Per-user reward-modifier MLP weights =
Trained per participant on 8,356 total pairwise feedback responses
- Predictive HCI model parameters =
Trained on interaction data from 25 e-commerce users in [15]
- Dual-reward combination rule or weight =
Not reported
- GA3C training hyperparameters and network architecture =
Not reported
assumptions (4)
- domain assumption QUIS and UES self-reports capture the user experience construct as defined.
- domain assumption Pairwise preference judgments on four-second video clips transfer to satisfaction with live adapted interfaces.
- domain assumption The predictive HCI model trained on 25 e-commerce users is a valid general engagement signal for trip-planning and e-learning apps.
- domain assumption Non-significant group effects imply the absence of carryover and order effects in the crossover design.
Cite this review
Pith. "Pith review of Integrating Human Feedback into a Reinforcement Learning-Based Framework for Adaptive User Interfaces." pith.science (2026). https://pith.science/paper/NHNTR74P
@misc{pith2026250420782,
author = {Pith},
title = {Pith review of: Integrating Human Feedback into a Reinforcement Learning-Based Framework for Adaptive User Interfaces},
year = {2026},
howpublished = {\url{https://pith.science/paper/NHNTR74P}},
note = {Machine review of arXiv:2504.20782}
}
read the original abstract
Adaptive User Interfaces (AUI) play a crucial role in modern software applications by dynamically adjusting interface elements to accommodate users' diverse and evolving needs. However, existing adaptation strategies often lack real-time responsiveness. Reinforcement Learning (RL) has emerged as a promising approach for addressing complex, sequential adaptation challenges, enabling adaptive systems to learn optimal policies based on previous adaptation experiences. Although RL has been applied to AUIs,integrating RL agents effectively within user interactions remains a challenge. In this paper, we enhance a RL-based Adaptive User Interface adaption framework by incorporating personalized human feedback directly into the leaning process. Unlike prior approaches that rely on a single pre-trained RL model, our approach trains a unique RL agent for each user, allowing individuals to actively shape their personal RL agent's policy, potentially leading to more personalized and responsive UI adaptations. To evaluate this approach, we conducted an empirical study to assess the impact of integrating human feedback into the RL-based Adaptive User Interface adaption framework and its effect on User Experience (UX). The study involved 33 participants interacting with AUIs incorporating human feedback and non-adaptive user interfaces in two domains: an e-learning platform and a trip-planning application. The results suggest that incorporating human feedback into RL-driven adaptations significantly enhances UX, offering promising directions for advancing adaptive capabilities and user-centered design in AUIs.
Figures
Reference graph
Works this paper leans on
-
[15]
Daniel Gaspar-Figueiredo, Marta Fernández-Diego, Silvia Abrahão, and Emilio Insfrán. 2025. A Comparative Study on Reward Models for UI Adaptation with Re- inforcement Learning. Accepted in Empirical Software Engineering Journal(EMSE) (2025). Available at: https://figshare.com/s/da9556a26a047fb44e4b
work page 2025
-
[1]
Silvia Abrahão, Emilio Insfrán, Arthur Sluÿters, and Jean Vanderdonckt. 2021. Model-based intelligent user interface adaptation: challenges and future direc- tions. Software and Systems Modeling 20, 5 (2021), 1335–1349. doi:10.1007/s10270- 021-00909-7
doi:10.1007/s10270- 2021
-
[2]
John G Adair. 1984. The Hawthorne effect: a reconsideration of the methodologi- cal artifact. Journal of applied psychology 69, 2 (1984), 334
work page 1984
-
[3]
Pierre A. Akiki, Arosha K. Bandara, and Yijun Yu. 2014. Adaptive Model-Driven User Interface Development Systems. ACM Comput. Surv. 47, 1, Article 9 (may 2014), 33 pages. doi:10.1145/2597999
doi:10.1145/2597999 2014
-
[4]
Mohammad Babaeizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. 2017. Reinforcement Learning through Asynchronous Advantage Actor- Critic on a GPU. arXiv:1611.06256 [cs.LG] https://arxiv.org/abs/1611.06256
arXiv 2017
-
[5]
Tita Alissa Bach, Amna Khan, Harry Hallock, Gabriela Beltrão, and Sonia Sousa
-
[6]
Sebastian Baltes and Paul Ralph. 2022. Sampling in software engineering research: a critical review and guidelines. Empirical Software Engineering 27, 4 (28 Apr 2022), 94. doi:10.1007/s10664-021-10072-8
-
[7]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:arXiv:1606.01540
arXiv 2016
Show all 40 references
-
[8]
Browne, Edward Powley, Daniel Whitehouse, Simon M
Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Pe- ter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligenc...
2012
-
[9]
Brumby and Susan Zhuang
Duncan P. Brumby and Susan Zhuang. 2015. Visual Grouping in Menu Interfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Com- puting Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 4203–4206. doi:10...
2015
-
[10]
Victor R Basili1 Gianluigi Caldiera and H Dieter Rombach. 1994. The goal question metric approach. Encyclopedia of software engineering 1 (1994), 528–532
1994
-
[11]
Gaëlle Calvary, Joëlle Coutaz, David Thevenin, Quentin Limbourg, Laurent Bouil- lon, and Jean Vanderdonckt. 2003. A Unifying Reference Framework for multi- target user interfaces. Interacting with Computers 15, 3 (06 2003), 289–308
2003
-
[12]
Brown, Miljan Martic, Shane Legg, and Dario Amodei
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences. arXiv:1706.03741 [stat.ML] https://arxiv.org/abs/1706.03741
2023 arXiv
-
[13]
Kevin Doherty and Gavin Doherty. 2018. Engagement in HCI: conception, theory and measurement. ACM Computing Surveys (CSUR) 51, 5 (2018), 1–39
2018
-
[14]
Amr Elmasry, Mostafa Kahla, Fady Ahdy, and Mahmoud Hashem. 2019. Red– black trees with constant update time. Acta Informatica 56, 5 (01 Jul 2019), 391–404. doi:10.1007/s00236-019-00335-9
2019 doi
-
[16]
Daniel Gaspar-Figueiredo, Marta Fernández-Diego, Ruben Nuredini, Silvia Abra- hao, and Emilio Insfran. 2024. Reinforcement Learning-Based Framework for the Intelligent Adaptation of User Interfaces. InSymposium on Engineering Interactive Computing Systems (EICS ’24) . 40–48. d...
2024
-
[17]
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. 2023. trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback. In Conference on Empirical Methods in Natural Language Pro...
2023 doi
-
[18]
Wiard Jorritsma, Fokie Cnossen, and Peter MA van Ooijen. 2015. Adaptive support for user interface customization: a study in radiology. International Journal of Human-Computer Studies 77 (2015), 1–9
2015
-
[19]
Maxwell K.D. 2002. Applied Statistics for Software Managers. Applied Statistics for Software Managers (2002), . https://cir.nii.ac.jp/crid/1573668924056253312
2002
-
[20]
Koo and Mae Y
Terry K. Koo and Mae Y. Li. 2016. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.Journal of Chiropractic Medicine 15, 2 (2016), 155–163. doi:10.1016/j.jcm.2016.02.012
2016 doi
-
[21]
Brockhoff, and Rune H
Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest Package: Tests in Linear Mixed Effects Models. Journal of Statistical Software 82, 13 (2017), 1–26. doi:10.18637/jss.v082.i13
2017 doi
-
[22]
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. In Conference on Empirical Methods in Natural Language...
2023 doi
-
[23]
Thomas Langerak, Sammy Christen, Mert Albaba, Christoph Gebhardt, and Otmar Hilliges. 2022. MARLUI: Multi-Agent Reinforcement Learning for Goal- Agnostic Adaptive UIs. arXiv preprint arXiv:2209.12660 , (2022),
2022 arXiv
-
[24]
Thomas Langerak, Sammy Christen, Mert Albaba, Christoph Gebhardt, Christian Holz, and Otmar Hilliges. 2024. MARLUI: Multi-Agent Reinforcement Learning for Adaptive Point-and-Click UIs. Proc. ACM Hum.-Comput. Interact. 8, EICS, Article 253 (June 2024), 27 pages. doi:10.1145/3661147
2024 doi
-
[25]
Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of User Engagement. In User Modeling, Adaptation, and Personalization , Judith Masthoff, Bamshad Mobasher, Michel C. Desmarais, and Roger Nkambou (Eds.). Springer Berlin Heidelberg, Berlin, Heidelbe...
2012
-
[26]
Kent L Norman, Ben Shneiderman, B Harper, and L Slaughter. 1998. Question- naire for user interaction satisfaction. Available at: https://site.unibo.it/hfrs/en/ questionnaire-and-scales-2/quis
1998
-
[27]
Heather L O’Brien and Elaine G Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American society for Information Science and Technology 59, 6 (2008), 938–955
2008
-
[28]
O’Brien, Paul Cairns, and Mark Hall
Heather L. O’Brien, Paul Cairns, and Mark Hall. 2018. A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form. International Journal of Human-Computer Studies 112 (2018), 28–39. doi:10.1016/j.ijhcs.2018.01.004
2018 doi
-
[29]
Gaëlle Calvary Sara Bouzit and Denis Chêne et al. 2016. A Comparison of Shortcut and Step-by-Step Adaptive Menus for Smartphones.European Workshop on Imagery and Cognition (2016), 1–12. doi:10.14236/ewic/HCI2016.26
2016 doi
-
[30]
R Core Team. 2013. R: A language and environment for statistical computing. R Foundation for Statistical Computing. (No Title) (2013),
2013
-
[31]
Subcommittee SC 4
Ergonomics of human-system interaction Technical Committee ISO/TC 159. Subcommittee SC 4. 2018. ISO 9241-11:2018 Ergonomics of Human-system Interaction: Egonomie de L’interaction Homme-système. Usability : definitions and concepts. Utilisabilité : définitions et concepts. Part...
2018
-
[32]
Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting User Interfaces with Model-Based Reinforcement Learning. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machine...
2021
-
[33]
Sira Vegas, Cecilia Apa, and Natalia Juristo. 2015. Cross-Over Designs in Software Engineering Experiments: Benefits and Perils. IEEE Transactions on Software Engineering 42 (01 2015), 1–1. doi:10.1109/TSE.2015.2467378
2015
-
[34]
Gianni Viano, Andrea Parodi, James Alty, Chris Khalil, Inaki Angulo, Daniele Biglino, Michel Crampes, Christophe Vaudry, Veronique Daurensan, and Philippe Lachaud. 2000. Adaptive User Interface for Process Control Based on Multi- Agent Approach. In Proc. of the Working Confere...
-
[35]
Dmitry Vidmanov and Alexander Alfimtsev. 2024. Mobile User Interface Adapta- tion Based on Usability Reward Model and Multi-Agent Reinforcement Learning. Multimodal Technologies and Interaction 8, 4 (2024), . doi:10.3390/mti8040026
2024 doi
-
[36]
Wei Wang, Hourieh Khalajzadeh, John Grundy, Anuradha Madugalla, and Humphrey O. Obie. 2024. Adaptive User Interfaces for Software Supporting Chronic Disease. In International Conference on Software Engineering (Lisbon, Portugal) (ICSE-SEIS’24). Association for Computing Machin...
2024
-
[37]
Ohlsson, Björn Regnell, and Anders Wesslén
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Empirical Strategies. Springer Berlin Heidelberg, Berlin, Heidelberg, 9–36. doi:10.1007/978-3-642-29044-2_2
2012 doi
-
[38]
Yifu Yuan, Jianye Hao, Yi Ma, Zibin Dong, Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and Yan Zheng. 2024. Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback. arXiv preprint arXiv:2402.02423 (2024),
2024 arXiv
-
[39]
Lamia Zouhaier, Yousra BenDalyHlaoui, and Leila Ben Ayed. 2023. Adaptive user interface based on accessibility context. Multimedia Tools and Applications 82, 23 (01 Sep 2023), 35621–35650. doi:10.1007/s11042-023-14390-5
2023 doi
-
[2024]
A systematic literature review of user trust in AI-enabled systems: An HCI perspective. Int. J. Hum. Comput. Interact. 40, 5 (2024), 1251–1266
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.