Human-Computer Interaction · arXiv:2611.16287 v1 · submitted 12 Nov 2026 · 34 references
RiskLens: Human-Informed Incident Transfer for AI Risk Evaluation
The paper says
“As AI capabilities advance, early-stage teams need better ways to judge whether historical incidents transfer to new products, demanding support beyond static risk checklists or one-off ideation tools.”
more of the abstract
We present RiskLens, a human-AI workflow with an interactive system built around a live crawl, refreshable ranker, and extensible, human-informed schema. RiskLens analyzes 2,034 AI incidents and 10,443 AI products to rank transferable incidents using a feature schema spanning technical, operational, and user-facing attributes and supports an interface that presents ranked incident evidence, generated scenarios, risk metrics, and mitigation suggestions. In a study with 21 AI-familiar participants, users surfaced evaluation criteria that we added as 13 human-informed dimensions, including harm categories, failure mechanisms, and vulnerable groups. The feature-based model outperformed a generic baseline with an 86.2% win rate, and remained stable on a twenty-month future holdout; after human-informed schema expansion, the updated model outperformed the same baseline on 91.4% of pairs.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
A solid HCI systems paper whose headline win rates are internally consistent but rest on GPT-4o as both teacher and judge, so the human-value claim needs broader validation before the numbers are taken at face value. This line is wrong
the referee pass did not run
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 34 references say.
What would settle it
Recruit 10–20 independent AI-risk practitioners (not the authors) and, on the same 50 held-out products from §5.1.2, show each the top-5 lists from both the feature-based model and the Sentence-BERT baseline; have them pick the more useful incident for their own risk-assessment purposes using their own criteria (no shared rubric). Compare the human preference rate against GPT-4o's 86.2%. As a secondary arm, have a subset of raters score roughly 100 product–incident pairs on the six eq. (1) dimensions and compute agreement with GPT-4o's labels, since that labeler is currently unvalidated. If independent humans prefer the feature-based model at ≥80% and agree with the eq-1 labels at κ≥0.6, the headline win rates are credible; if human preference is near chance or diverges from GPT-4o, the 86.2%/85.3%/91.4% figures are inflated by the labeler–judge coupling.
from the skeptic's reading
How the review went
3 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
Not run on this paper: the referee, the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
I read RiskLens as a genuine systems contribution, not a rehash. The new artifact is specific: a feature-match ranking model for product-to-incident transfer, evaluated against a text-similarity baseline, plus a 21-participant study that produces 13 human-informed schema dimensions. The ablation and coefficient tables give real evidence about what the model learns, and the temporal holdout is a reasonable robustness check. The authors also do something I appreciate: they state the necessary condition for their evaluation (b206) and then list the limits of their own design in the limitations section, including the missing control condition (b233), the single holdout, and the narrow baseline (b379, b380). Their RQ2 claims are framed as self-reported change, not causal effects, which is the right level of modesty.
The soft spot is the one the reader flagged, and it is load-bearing rather than cosmetic. The same model family supplies the eq. 1 training labels, extracts the features, and then judges the pairwise comparisons. The baseline was never trained on the judge's preferences. So an 86.2% win rate largely demonstrates that a model fitted to GPT-4o's rubric beats an unfitted baseline at predicting that rubric. That is informative only insofar as the rubric matches what practitioners need, and the human check is thin: 50 pairs, one author as rater, second author as verifier, kappa 0.71, using the same structured rubric given to the LLM. It validates that GPT-4o can apply the authors' rubric consistently; it does not validate the labeler used in training, which is a different prompt and rubric. I would not call the paper fraudulent or even sloppy on this point, because the authors flag the necessary condition themselves. But the central scientific claim is conditional, and the condition is not yet met.
The Phase 3 result is softer still. The 13 new dimensions and the revised judging rationales come from the same 21 transcripts, so the expanded model is evaluated by a rubric that encodes the very considerations the model was built to satisfy. The jump from 86.2% to 91.4% changes model and criteria together and is partly self-fulfilling. The authors are upfront that this is a "human-informed evaluation criteria" comparison, but the number will be quoted out of context if it is not accompanied by an independent human evaluation.
Minor points: the temporal holdout compares only top-1 recommendations, which is a narrower test than the main protocol; there are no release artifacts, which makes replication harder. Neither is disqualifying for a CHI-style systems paper.
Who gets value: HCI and responsible-AI readers working on incident-based risk assessment or LLM-as-judge methodology. The paper deserves a serious referee; I would send it out, then ask the authors for a non-author human validation of the training labels or a practitioner-facing study, and ideally code and data.
The strongest objection
The central claim (Phase 1, §5.1.2 and §5.1.3) rests entirely on GPT-4o as the arbiter of relevance. The hybrid model is trained to fit GPT-4o-assigned labels from the six-dimension rubric in eq. (1); product/incident features are extracted by GPT-4o (§3.1.2); and the pairwise evaluation judge is GPT-4o (§5.1.2). The Sentence-BERT baseline was trained on nothing. An 86.2% win rate over a baseline that was never fitted to the judge's preferences is, to first order, 'the model trained on judge J beats an untrained baseline at predicting J's ratings.' That is informative only insofar as J's relevance matches what practitioners need. The paper states this necessary condition (sentence b206) and checks it on 50 pairs with one author as human rater and a second as verifier (κ=0.71). Two gaps remain. First, that check validates the pairwise comparator only; the eq. (1) six-dimension training labeler — the source of the learned signal producing the model's edge — receives no human validation at all, and it is a different prompt/rubric than the pairwise judge. Second, the human rater applied 'the same structured rubric presented to the LLM judge,' and the raters are the system's own authors; this tests whether GPT-4o applies the authors' rubric consistently, not whether that rubric captures practitioners' actual assessment needs — the very thing §6.1.2 claims users 'revise.' For Phase 3 (§5.3.3) a compounding leak exists: the 13 new features and the revised judging rationales both derive from the same 21-participant transcripts, so the judge's rubric encodes exactly the considerations the expanded model was built to satisfy; the 91.4% (vs 86.2%) changes model and rubric simultaneously and is partly self-fulfilling.
The reader's weakest assumption — GPT-4o's pairwise relevance judgments as a proxy for human judgments — is the right crux, and I agree with the CONDITIONAL verdict. My pass sharpens that concern in two ways the reader underweighted. First, the human-agreement check (κ=0.71, n=50) covers only the pairwise comparator; the eq. (1) training labeler, which is what actually shapes the model's learned signal, is never human-validated, and it uses a different six-dimension rubric than the pairwise judge. The paper's sentence b206 conditions the win-rate interpretation on judge-human alignment, and the authors do validate that endpoint — but the other endpoint, the one most responsible for the model's edge over an untrained baseline, is asserted, not checked. Second, the human rater is a system author applying the same structured rubric handed to the LLM, so the check establishes GPT-4o's consistency with the authors' rubric, not with practitioners' independent notions of usefulness; given §6.1.2's claim that users revise the very meaning of relevance, the 50-pair check is a weaker grounding than it appears. Phase 3 adds a compounding issue: the 13 new features and the revised judge rationales both come out of the same 21-participant transcripts, so the rubric rewards exactly what the expanded model was built to encode, making the 91.4%-vs-86.2% gain partly self-fulfilling; the authors do label this 'under human-informed evaluation criteria,' but the confound between model change and rubric change remains unresolved. None of this makes the paper dishonest or internally inconsistent — the limitations section is candid about the single baseline, the missing control, and the single temporal holdout (sentences b374, b378, b380). It does mean the headline quantitative claims inherit an unquantified dependence on GPT-4o's notion of relevance that the strongest architectural result — the ablation showing categorical features add signal on top of text (Table 4) — does not escape, since the labels are themselves GPT-4o-produced. The concrete test (independent practitioner preference + labeler agreement) would settle whether the concern lands; until then CONDITIONAL, not ACCEPT, is the right call, and my concern does not move the verdict.
Pith's reading of the paper (a machine's summary, not the authors')
This paper introduces RiskLens, a human-AI workflow that ranks historical AI incidents by how relevant they are to a new product's risk assessment. The central claim is that a hybrid model combining explicit categorical feature matches (like technology type, application domain, deployment setting) with learned text similarity can outperform a pure text-similarity baseline at recommending incidents useful for early-stage risk evaluation. The paper reports that this model wins 86.2% of pairwise comparisons against a Sentence-BERT baseline, retains an 85.3% win rate on a temporal holdout of future products, and after incorporating 13 additional human-informed feature dimensions (such as vulnerable groups, jurisdiction, safeguards, and mitigation feasibility) reaches a 91.4% win rate under revised evaluation criteria. A user study with 21 AI-familiar participants suggests that using the system broadens the range of risks considered, increases reliance on incident evidence, and leads to more structured assessment processes. The paper argues that incident transferability is a contextual judgment that depends on shared attributes enabling a failure and on differences altering its likelihood or consequences, and that human feedback can usefully revise the criteria by which relevance is defined.
What carries the argument. The central mechanism is a hybrid ranking model that scores each product–incident pair using a logistic regression over binary categorical feature matches and a learned text similarity component. For each of eight (later 21) categorical dimensions, the model checks whether the product and incident share the same value and assigns a weight learned from training data; the text similarity is computed via Sentence-BERT with a learned projection. The weighted sum is passed through a sigmoid to produce a relevance score. The weights reveal which features matter most: technology type, application domain, scale of operation, and deployment setting dominate the categorical signal, while text similarity contributes independently. The model is trained on 50,000 product–incident pairs labeled by GPT-4o, and evaluated via pairwise comparisons judged by the same LLM, with a human agreement check (kappa = 0.71) on a subset.
Core claim. RiskLens demonstrates that a feature-based hybrid ranking model, which combines eight categorical match signals (technology type, application domain, scale of operation, deployment setting, output type, target user group, model class, and interaction modality) with learned text similarity, can recommend historical AI incidents to a target product more effectively than a pure text-similarity baseline. The paper's key empirical finding is that the full model achieves an 86.2% win rate against a Sentence-BERT baseline in LLM-judged pairwise comparisons on held-out products, and 85.3% on a temporal holdout of products from a later twenty-month period. After the user study identified 13 additional human-informed feature dimensions (including harm categories, failure mechanisms, vulnerable groups, jurisdiction, regulatory regimes, safeguards, mitigation feasibility, data lifecycle attributes, and risk propagation patterns), the expanded model reached a 91.4% win rate under revised evaluation criteria that incorporate those human-informed rationales. The paper argues that this approach supports a more contextual, evidence-based assessment of whether and why a historical failure could transfer to a new product.
The premise it rests on. GPT-4o's pairwise relevance judgments are a valid proxy for what a human practitioner would consider the more useful incident for a product's risk assessment; if GPT-4o's notion of relevance diverges from practitioners' needs, both the training labels and the headline win rates are biased.
If the paper is right (Pith's inferences, not the authors')
If the model's win rates hold up in practice, early-stage AI teams could systematically screen historical incidents for relevance to their product without relying on manual search or static checklists.
The explicit feature schema makes the basis of each recommendation inspectable: a product developer can see which shared attributes (e.g., same technology type or application domain) drove the ranking, and which differences might limit transfer.
The human-informed schema expansion shows that user feedback can be translated into reusable comparison dimensions, making the system's notion of relevance evolve with practitioner needs.
The temporal holdout result suggests that a model trained on older incidents can remain effective on newer products and incidents, at least across a twenty-month window, without retraining.
The paper's reliance on GPT-4o for both training labels and evaluation judge creates a circularity that is only partially addressed by the human agreement check (kappa = 0.71 on 50 pairs). If GPT-4o's notion of 'relevance for risk assessment' systematically differs from what practitioners need, both the training signal and the headline win rates would inherit that bias. A stronger test would invol
The 21-participant study lacks a control condition, so the observed improvements in breadth, depth, and structure of participants' risk assessment plans cannot be cleanly attributed to the recommender alone. The paper acknowledges this limitation but does not quantify the potential confound from the study activities themselves (e.g., the act of drawing mental models, think-aloud protocols, and exi
The paper does not compare RiskLens against complete existing workflows such as Farsight or AHA!; the baseline is a pure text-similarity retriever. A direct comparison with those tools would clarify whether the feature-based ranking adds value beyond the incident retrieval and scenario generation that those systems already provide.
The feature schema expansion was driven by a single study with 21 participants assessing one product each. The generalizability of the 13 new dimensions to other product domains, user groups, or incident types is untested. Some dimensions (e.g., spread mode) had a near-zero learned coefficient, suggesting they may not be broadly useful.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
As AI capabilities advance, early-stage teams need better ways to judge whether historical incidents transfer to new products, demanding support beyond static risk checklists or one-off ideation tools. We present RiskLens, a human-AI workflow with an interactive system built around a live crawl, refreshable ranker, and extensible, human-informed schema. RiskLens analyzes 2,034 AI incidents and 10,443 AI products to rank transferable incidents using a feature schema spanning technical, operational, and user-facing attributes and supports an interface that presents ranked incident evidence, generated scenarios, risk metrics, and mitigation suggestions. In a study with 21 AI-familiar participants, users surfaced evaluation criteria that we added as 13 human-informed dimensions, including harm categories, failure mechanisms, and vulnerable groups. The feature-based model outperformed a generic baseline with an 86.2% win rate, and remained stable on a twenty-month future holdout; after human-informed schema expansion, the updated model outperformed the same baseline on 91.4% of pairs.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 1.00
“The feature-based model outperformed a generic baseline with an 86.2% win rate, and remained stable on a twenty-month future holdout; after human-informed schema expansion, the updated model outperformed the same baseline on 91.4% of pairs.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.10 · human participants 0.96 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.96 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| As AI capabilities advance, early-stage teams need better ways to judge whether historical incidents transfer to new products, demanding support beyond static risk checklists or one-off ideation tools. | background 0.66 | 0.06 | 0.06 | asserted 0.99 |
| We present RiskLens, a human-AI workflow with an interactive system built around a live crawl, refreshable ranker, and extensible, human-informed schema. | contribution 0.87 | 0.92 | 0.07 | asserted 1.00 |
| RiskLens analyzes 2,034 AI incidents and 10,443 AI products to rank transferable incidents using a feature schema spanning technical, operational, and user-facing attributes and supports an interface that presents ranked incident evidence, generated scenarios, risk metrics, and mitigation suggestions. | method 0.84 | 0.95 | 0.13 | asserted 1.00 |
| In a study with 21 AI-familiar participants, users surfaced evaluation criteria that we added as 13 human-informed dimensions, including harm categories, failure mechanisms, and vulnerable groups. | contribution 0.53 | 0.95 | 0.06 | asserted 1.00 |
| The feature-based model outperformed a generic baseline with an 86.2% win rate, and remained stable on a twenty-month future holdout; after human-informed schema expansion, the updated model outperformed the same baseline on 91.4% of pairs. | result 1.00 | 0.96 | 0.41 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.59
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 4,512 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (6)
Figures are the authors', reproduced from the paper.
References (34)
- [1] Gavin Abercrombie, Djalel Benbouzid, Paolo Giudici, Delaram Golpayegani, Julio Hernandez, Pierre Noro, Harshvardhan Pandit, Eva Paraschou, Charlie Pownall, Jyoti Prajapati, Mark A. Sayre, Ushnish Sengupta, Arthit Suriyawongkul, Ruby Thelot, Sofia Vei, and Laura Waltersdorfer. ... on PithPith's reviewarXiv
- [2] Avinash Agarwal and Manisha J. Nene. 2024. Standardised Schema and Taxonomy for AI Incident Databases in Critical Digital Infrastructure. In 2024 IEEE Pune Section International Conference (PuneCon). IEEE, 1–6. doi:10.1109/punecon63413.2024.10895867 on Pithdoi:10.1109/punecon63413.2024.10895867
- [3] Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020. Fairlearn: A toolkit for assessing and improving fairness in AI.Microsoft, Tech. Rep. MSR-TR-2020-32(2020) on Pith
- [4] Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, and Saleema Amershi. 2023. Aha!: Facilitating ai impact assessment by generating examples of harms.arXiv preprint arXiv:2306.03280(2023) on PithPith's reviewarXiv
- [5] Michaela Hardt, Xiaoguang Chen, Xiaoyi Cheng, Michele Donini, Jason Gelman, Satish Gollaprolu, John He, Pedro Larroy, Xinyu Liu, Nick McCarthy, Ashish Rathi, Scott Rees, Ankit Siva, ErhYuan Tsai, Keerthan Vasist, Pinar Yilmaz, Muhammad Bilal Zafar, Sanjiv Das, Kevin Haas, Tyle... on Pith
- [6] Viviane Herdel, Sanja Šćepanović, Edyta Bogucka, and Daniele Quercia. 2024. ExploreGen: Large language models for envisioning the uses and risks of AI technologies. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 584–596 on Pith
- [7] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé, Miro Dudik, and Hanna Wallach. 2019. Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, 1–16... on Pithdoi:10.1145/3290605.3300830
- [8] Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and P. M. Krafft. 2020. Toward situated interventions for algorithmic equity: lessons from the field. InProceedings of the 2020 Conference on Fairness, Accountabi... on Pithdoi:10.1145/3351095.3372874
- [9] Michelle S. Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. 2024. Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(C... on Pithdoi:10.1145/3613904.3642830
- [10] J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.biometrics(1977), 159–174 on Pith
- [11] Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and Merge: Aligning Position Biases in LLM-based Evaluators. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohi... on Pithdoi:10.18653/v1/2024.emnlp-main.621
- [12] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv:2303.16634 [cs.CL] https://arxiv.org/abs/2303.16634 on PithPith's reviewarXiv
- [13] Sean McGregor. 2020. Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. arXiv:2011.08512 [cs.CY] https://arxiv.org/abs/2011.08512 on PithPith's reviewarXiv
- [14] Microsoft. 2022. Microsoft Responsible AI Standard v2: General Requirements. https://msblogs.thesourcemediaassets.com/sites/5/2022/06/Microsoft- Responsible-AI-Standard-v2-General-Requirements-3.pdf on Pith
- [15] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. InProceedings of the Conference on Fairness, Accountability, and Transparency(Atlanta, G... on PithPith's reviewdoi:10.1145/3287560.3287596
- [16] National Institute of Standards and Technology. 2024. AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework on Pith
- [17] DTK Ng, WY Luo, HMY Chan, and SKW Chu. 2022. An examination on primary students’ development in AI literacy through digital story writing. Computers & Education: Artificial Intelligence100054, 3 (2022), 100054 on Pith
- [18] Rock Yuren Pang, Sebastin Santy, René Just, and Katharina Reinecke. 2024. Blip: facilitating the exploration of undesirable consequences of digital technologies. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–18 on Pith
- [19] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076 [cs.CL] https://arxiv.org/abs/2404.13076 on PithPith's reviewarXiv
- [20] Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. InProceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2–4 on Pith
show 14 more
- [21] Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2021. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for Shifting Organizational Practices.Proceedings of the ACM on Human-Computer Interaction5, CSCW1 (April 2021), 1–23. doi:10.... on Pith
- [22] Daniel M Russell, Mark J Stefik, Peter Pirolli, and Stuart K Card. 1993. The cost structure of sensemaking. InProceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. 269–276 on Pith
- [23] Peter Slattery, Alexander K. Saeri, Emily A. C. Grundy, Jess Graham, Michael Noetel, Risto Uuk, James Dao, Soroush Pour, Stephen Casper, and Neil Thompson. 2025. The AI Risk Repository: A Comprehensive Meta-Review, Database, and Taxonomy of Risks From Artificial Intelligence. ... on PithPith's reviewarXiv
- [24] Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the LLM-as-a- Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. InProceedings of the 30th International Conference on Intellige... on Pithdoi:10.1145/3708359.3712091
- [25] Yiliu Tang, Jason Situ, Andrea Yaoyun Cui, Mengke Wu, and Yun Huang. 2025. LLM Integration in Extended Reality: A Comprehensive Review of Current Trends, Challenges, and Future Perspectives. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’2... on Pithdoi:10.1145/3706598.3714224
- [26] Yiliu Tang, Jason Situ, and Yun Huang. 2024. Beyond User Experience: Technical and Contextual Metrics for Large Language Models in Extended Reality. InCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing(Melbourne VIC, Australia)(Ub... on Pithdoi:10.1145/3675094.3678995
- [27] Yiliu Tang, Mengke Wu, Jason Situ, Jingyi Liu, Andrea Yaoyun Cui, and Yun Huang. 2026. Empowered XR through Generative AI: Balancing Superpowers and Risks. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). Association for Computing Machi... on Pithdoi:10.1145/3772318.3791307
- [28] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large Language Models are not Fair Evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo... on Pithdoi:10.18653/v1/2024.acl-long.511
- [29] Zijie J. Wang, Chinmay Kulkarni, Lauren Wilcox, Michael Terry, and Michael Madaio. 2024. Farsight: Fostering Responsible AI Awareness During AI Application Prototyping. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems(Honolulu, HI, USA)(CHI ’24). ... on Pithdoi:10.1145/3613904.3642335
- [30] Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. 2025. Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819 [cs.CL] https://arxiv.org/abs/ 2410.21819 Manuscript submitted to ACM RiskLens: Human-Informed Incident Transfer for AI Risk Evaluation 25 on PithPith's reviewarXiv
- [31] Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. 2025. Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates. arXiv:2408.13006 [cs.CL] https://arxiv.org/abs/2408.13006 on PithPith's reviewarXiv
- [32] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. InProceedings of the 37th Internati... on Pith
- [33] Qingxiao Zheng, Minrui Chen, Pranav Sharma, Yiliu Tang, Mehul Oswal, Yiren Liu, and Yun Huang. 2025. EvAlignUX: Advancing UX Evaluation through LLM-Supported Metrics Exploration. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Associat... on Pithdoi:10.1145/3706598.3714045
- [34] Qingxiao Zheng, Yiliu Tang, Yiren Liu, Weizi Liu, and Yun Huang. 2022. UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems(New Orleans, LA, USA)(CHI ’... on Pithdoi:10.1145/3491102.3501855
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment