Coding benchmarks conflate the model with the harness, environment, and verifier into a single end-to-end score, which is misaligned with agentic software engineering.
ACM Transactions on Computer-Human Interaction, 27(5)
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
citation-role summary
background 1
other 1
citation-polarity summary
representative citing papers
A repeatable worksheet and human-reviewed expansion process turns expert-elicited AI use cases into 107 grounded scenarios to support consistent human-centered evaluations.
Queer artists largely refuse and resist generative AI, seeing it as anti-relational and disruptive to the community-oriented, identity-forming nature of their art practices, with only limited acceptance for surreal image generation.
A statistical framework decomposes human annotation outcomes into four interpretable variation sources and extends classical measurement-error models to handle both shared and individualized notions of truth.
RLHF annotations frequently lack stable underlying preferences; consistency diagnostics on PRISM and PluriHarms show that removing inconsistent annotators flips majority harm classifications for 18.6% of prompts.
Proposes an interdisciplinary framework and taxonomy for responsible evaluation of AI mental health tools based on analysis of 135 publications identifying gaps in metrics, expert involvement, safety, and equity.
An LLM annotation pipeline for Schwartz human values in Russian social media, calibrated against expert labels and transferred to a smaller encoder, changes measured value prevalence but not necessarily overall trend direction.
Industry markets AI agents for orchestration, creation, and insight, but a usability study with 31 participants reveals users face challenges from capability misalignment and lack of meta-cognition in tools like Operator and Manus.
Proposes AI-driven simulations for literary-historical experiments and reports preliminary text-generation results claiming the first limited in-distribution outputs matching human novels.
Context specification is a process that turns diffuse stakeholder perspectives into explicit definitions of properties, behaviors, and outcomes to guide context-aware AI evaluations.
citing papers explorer
-
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Coding benchmarks conflate the model with the harness, environment, and verifier into a single end-to-end score, which is misaligned with agentic software engineering.
-
Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios
A repeatable worksheet and human-reviewed expansion process turns expert-elicited AI use cases into 107 grounded scenarios to support consistent human-centered evaluations.
-
"I Just Don't Want My Work Being Fed Into The AI Blender": Queer Artists on Refusing and Resisting Generative AI
Queer artists largely refuse and resist generative AI, seeing it as anti-relational and disruptive to the community-oriented, identity-forming nature of their art practices, with only limited acceptance for surreal image generation.
-
From Ground Truth to Measurement: A Statistical Framework for Human Labeling
A statistical framework decomposes human annotation outcomes into four interpretable variation sources and extends classical measurement-error models to handle both shared and individualized notions of truth.
-
RLHF May Not Reflect Genuine Preferences
RLHF annotations frequently lack stable underlying preferences; consistency diagnostics on PRISM and PluriHarms show that removing inconsistent annotators flips majority harm classifications for 18.6% of prompts.
-
Responsible Evaluation of AI for Mental Health
Proposes an interdisciplinary framework and taxonomy for responsible evaluation of AI mental health tools based on analysis of 135 publications identifying gaps in metrics, expert involvement, safety, and equity.
-
Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer
An LLM annotation pipeline for Schwartz human values in Russian social media, calibrated against expert labels and transferred to a smaller encoder, changes measured value prevalence but not necessarily overall trend direction.
-
Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agents
Industry markets AI agents for orchestration, creation, and insight, but a usability study with 31 participants reveals users face challenges from capability misalignment and lack of meta-cognition in tools like Operator and Manus.
-
AI as a Tool for Simulation-Based Experiments in Literary Studies
Proposes AI-driven simulations for literary-historical experiments and reports preliminary text-generation results claiming the first limited in-distribution outputs matching human novels.
-
Making AI Evaluation Deployment Relevant Through Context Specification
Context specification is a process that turns diffuse stakeholder perspectives into explicit definitions of properties, behaviors, and outcomes to guide context-aware AI evaluations.