Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

PULSE, a prediction-powered evaluation framework, combines sparse human ratings with model-filled pseudo-labels to measure how agent design changes affect user satisfaction, cutting confidence-interval widths by ~40% and revealing that the

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 10:23 UTC pith:MU5C35BC

load-bearing objection Genuinely useful deployment and benchmark-vs-human comparison, but the PPI-based CI reduction and one significance claim rely on training f on the same labeled data used in the rectifier—no cross-fitting—so the statistical headline is not reliable as written. the 3 major comments →

arxiv 2510.09801 v3 pith:MU5C35BC submitted 2025-10-10 cs.AI

How can we assess human-agent interactions? Case studies in software agent design

Valerie Chen , Rohit Malhotra , Xingyao Wang , Juan Michelini , Xuhui Zhou , Aditya Bharat Soni , Hoang H. Tran , Calvin Smith
show 2 more authors
Ameet Talwalkar Graham Neubig
This is my paper
classification cs.AI
keywords PULSEhuman-agent interactionprediction-powered inferenceLLM coding agentsA/B testinguser satisfactionbenchmark evaluationsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to make human assessment of AI agents rigorous and affordable. It argues that benchmarks only measure full automation, missing the collaborative reality of human-agent work, so it proposes PULSE, a three-step framework: collect sparse voluntary ratings, train a model to predict satisfaction from interaction features, then use prediction-powered inference to merge human labels with model-generated pseudo-labels. Deployed on an open-source coding agent across 36k sessions and 15k users, PULSE yields effect-size estimates with confidence intervals roughly 40% narrower than a standard A/B test, allowing conclusions from far fewer human ratings. The deployment's empirical finding is that the choice of LLM backbone changes user satisfaction by 6–8%, while planning or memory changes shift it by less than 3% or not at all. The paper also documents that benchmark rankings can contradict human preference, arguing for human-in-the-loop evaluation.

Core claim

The central claim is that the effect of an agent design change on user satisfaction can be estimated with valid, narrower confidence intervals by augmenting a small set of human ratings with predictions from a trained model on a 20x larger unlabeled pool. Formally, for each condition the estimator combines the labeled sample mean with a tuned PPI correction term, and under standard PPI regularity conditions it is asymptotically normal; the paper reports an average 39.5% reduction in CI width across four comparisons, turning inconclusive naive differences into significant ones. Empirically, deploying this at scale shows the LLM backbone is the dominant lever: claude-4-sonnet is preferred over

What carries the argument

The load-bearing object is PULSE's extension of prediction-powered inference (PPI), a statistical estimator that corrects a model's predictions on unlabeled data using the labeled data. For each condition c, the augmented mean is the sample average of human ratings plus λ times the difference between the model's mean prediction on unlabeled sessions and its mean prediction on labeled sessions, with λ chosen by a plug-in covariance-variance ratio. Under PPI's regularity conditions this estimator is asymptotically normal and, when the predictor correlates with true ratings, has smaller variance than the naive empirical mean. The predictor itself is a random forest trained on 1,747 labeled traj

Load-bearing premise

The whole statistical construction assumes that users who give ratings are no different, in how satisfied they are, from users who stay silent, and that the model used to fill in missing ratings can be treated as fixed when calculating the uncertainty.

What would settle it

Run the same four agent comparisons with fully labeled data — e.g., by giving a random subset of non-raters an incentive to rate, or by exhaustively labeling a held-out sample of sessions — and compare the fully-labeled effect sizes and confidence intervals to PULSE's augmented estimates. If the augmented point estimates fall outside the fully-labeled CIs more than 5% of the time, or if the feature distributions of volunteer raters and non-raters differ significantly on variables like session length or user message count, the exchangeability or fixed-f assumption is invalid. A second check: re

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Agent teams can run reliable A/B tests on design changes with only a small fraction of the human ratings needed today, because unlabeled usage logs already in hand can be leveraged.
  • Comparisons that look inconclusive under a naive bootstrap (e.g., claude-3.7 vs claude-4; plan vs no-plan) become statistically decisive under the augmented estimator.
  • The empirical hierarchy — backbone >> scaffolding — suggests prioritizing investment in stronger base models over planning or memory tweaks when the goal is user satisfaction.
  • Benchmark leaderboard moves should not be taken at face value for human-facing agents; task-specific human pilots are needed, especially for tasks like fixing CI where benchmarks and human ratings diverge.
  • High-importance features (user sentiment, git push) offer cheap, continuously available proxies for satisfaction that can guide online monitoring.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The PPI-style estimator's benefit depends on the predictor's correlation with true ratings; in settings where interaction dynamics change (e.g., a new agent that alters user message style), the pretrained f would need periodic recalibration or the CI reduction would shrink.
  • The same machinery could be applied to other human-AI metrics such as binary helpfulness, task completion time, or trust, as long as a predictor can be trained on a labeled subset.
  • Because ratings are voluntary, the exchangeability assumption is the main threat; a simple placebo test would check whether the feature distribution of rated sessions matches unrated sessions within each condition.
  • A direct extension would use the feature importances to build an explicit early-warning system: drops in git push rate or user message rate could flag dissatisfaction before explicit ratings arrive.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PULSE, a framework for human-centric evaluation of LLM-powered agents. PULSE collects sparse user ratings, trains an ML model to predict satisfaction from agent trajectories, and combines human labels with model predictions via prediction-powered inference (PPI) to estimate effect sizes of agent design changes. The framework is deployed on a web platform built around the OpenHands software agent, with data from over 36k sessions and 15k users. Three case studies compare LLM backbones (claude-3.7 vs claude-4; claude-4 vs gpt-5), a planning mechanism, and a memory management parameter. The paper claims that PULSE reduces 95% CI widths by ~40% relative to a standard A/B test, that LLM backbone is the dominant driver of user satisfaction, and that benchmark performance often disagrees with human satisfaction. The stated contribution is a validated methodology plus practical insights for agent design.

Significance. If the statistical claims hold, the paper makes a valuable contribution by showing a concrete way to run efficient human-in-the-loop evaluations of deployed agents. The scale of the deployment—15k users and 36k sessions—and the open-sourced platform are notable strengths. The framework's combination of feature-based satisfaction modeling with PPI is a sensible use of existing statistical machinery, and the empirical finding that benchmark gains do not always translate to user satisfaction is an important cautionary result for the community. The paper is also transparent about its limitations (single agent, single metric, privacy constraints on releasing trajectories). However, the validity of the headline CI-reduction and significance claims depends on statistical assumptions that are currently unverified, so the significance of the contribution, while potentially high, is not yet established.

major comments (3)
  1. [§2.2–2.3, Table 3] The PPI estimator in Section 2.3 uses a satisfaction model f trained on the full labeled dataset (N=1747, §2.1) and then applies the same f to the labeled sessions in the rectifier term (1/n_c)Σ f(X_i). No cross-fitting or sample splitting is described. The cited PPI variance formula assumes f is fixed (independent of the estimation sample); when f is fit on the same data, in-sample residuals are overfitted, which biases the rectifier and can understate variance. The practical consequence is visible in Table 3: the Claude-3.7 vs Claude-4 effect moves from a non-significant naive CI [-3.95%,11.78%] to a significant augmented CI [1.99%,8.06%]. Please provide cross-fitted estimates or explicitly justify why the current procedure preserves validity; otherwise the reported significance and the average 39.5% CI reduction are not supported.
  2. [Figure 3, §3.3] The reported CI reduction is computed unfairly. Section 3.3 states 'For fair comparison of the effect of PPI, we subsample to the same number of data points in human-only (i.e., 150 per condition).' A standard A/B test would use all available human labels, not a subsample of 150. Since CI width scales roughly as 1/√n, artificially shrinking the naive sample inflates PPI's apparent benefit. Please report naive CIs on the full labeled data (or on the same unlabeled-augmented sample) so the 39.5% reduction claim is a like-for-like comparison.
  3. [§2.1] The validity of PULSE rests on treating the ~5% of sessions with voluntary user ratings as a random sample of the full population of sessions. If rating propensity is correlated with satisfaction (e.g., dissatisfied users are more likely to rate), both the naive and augmented estimators target a selected population, not the full deployment population. The paper provides no missingness analysis, comparison of observed features between labeled and unlabeled sessions, or weighting scheme. This is a potential source of bias that a cross-fitted f cannot fix.
minor comments (5)
  1. [§2.3] The notation in the PPI formulas is inconsistent: 'bµc(bλc)' and 'bλc' are used with mismatched subscripts, and the convergence statement has a typo ('trajectores').
  2. [§3.3/Table 3] Please clarify whether the reported naive CIs use the subsample of 150 per condition or all available labels. The text in §3.3 suggests subsampling, but Table 3 does not state this; this ambiguity directly affects interpretation of the 39.5% reduction claim.
  3. [§5, Table 2] The text reports a correlation of ρ = -0.11 for the claude-4 vs gpt-5 comparison, while Table 2 reports ρ = -0.18. Please reconcile.
  4. [Throughout] The model naming is inconsistent: the paper alternates between 'claude-4-sonnet' and 'claude-sonnet-4'. Use one consistent name.
  5. [Figure 1] The caption contains a duplicated word: 'enables' appears twice in the first sentence.

Circularity Check

1 steps flagged

PULSE's CI reduction is partly an in-sample artifact: f is trained on the same labeled trajectories used in the PPI rectifier and variance formula, with no cross-fitting, so the reported 39.5% CI shrinkage and newly significant Claude-3.7 vs Claude-4 result are partly driven by the model's fit to its own training labels.

specific steps
  1. other [Section 2.2–2.3 and Section 3.3; Table 3]
    "Given a dataset of labeled trajectories D = {(Xi, ¯Yi)}, we train an ML model f to infill user satisfaction ¯Yi, when it is ∅... bµc(bλc) = 1/nc Σ_i Yi + bλc(1/Nc Σ_j f( ˜Xj) − 1/nc Σ_i f(Xi)) ... we use the same feedback model, rather than training a specific feedback model per case study ... using the best performing f we trained (i.e., random forest model)."

    The estimator applies f to the same labeled trajectories used to train it: the rectifier term (1/n_c)Σ f(X_i) and the variance plug-in both use in-sample fitted values, since §2.2 describes training f on the full labeled set and §3.3 confirms the same model is used for all case studies; no cross-fitting or data splitting is described. Under the PPI regularity conditions the paper invokes, f must be independent of the labeled estimation sample (or be cross-fitted). Otherwise the residuals Y_i − f(X_i) are overly small, so σ̂² and the CI widths are understated. The advertised 39.5% CI reduction, and the Claude-3.7 vs Claude-4 interval moving from [−3.95%, 11.78%] to [1.99%, 8.06%] in Table 3, are therefore partly an artifact of fitting f to the very labels it is used to correct rather than a

full rationale

Aside from the single in-sample-fitting issue, the paper's derivation is mostly self-contained: PULSE's three-step pipeline is a direct application of prediction-powered inference, the effect-size estimator is explicitly defined as a difference of PPI means, and the comparisons to benchmarks use independent external benchmark data. The human labels anchor the estimate and the predictive model has nonzero error (Table 1), so the point estimates are not literally equal to the model's fitted values. The circularity is partial and concentrated in the variance/CI machinery: because f is trained on the same labeled data that enters the rectifier and variance formula, the credibility of the reported CI reduction and of the newly significant result for Claude-3.7 vs Claude-4 depends on a regularity condition (f independent of the estimation sample) that the paper does not enforce. This is a correctness/validity concern that borders on fitted-input-called-prediction, but it is not a full self-definitional collapse; hence the moderate score.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The core statistical machinery is PPI, whose regularity conditions are assumed (§2.3). The main additional load-bearing choices are the missingness assumption and the data-dependence of f, neither of which is verified.

free parameters (2)
  • PPI tuning parameter λ per condition = estimated per condition (ratio of covariance to variance)
    The 40% CI reduction claim depends on this plug-in estimate; it is fitted to the data rather than fixed a priori. Invoked in §2.3.
  • max step threshold (memory case study) = 80 vs 120
    The operational decision of when to condense context was selected in simulation (§3.2, Appendix A.3) to balance cost and performance, not from the human-satisfaction data itself; it is a hand-chosen design parameter defining the treatment contrast.
axioms (4)
  • domain assumption Labeled and unlabeled sessions are exchangeable: missing ratings are ignorable.
    PPI uses unlabeled sessions to reduce variance; if users who rate differ systematically in satisfaction from those who don't (~5% respond), the augmented estimator is biased. Stated nowhere in §2.
  • domain assumption The feedback model f can be treated as fixed when computing PPI variance.
    §2.3 applies the PPI variance formula directly, but f was trained on the same labeled trajectories used in the rectifier; no cross-fitting or correction for estimation error is described.
  • domain assumption Conversations are independent despite repeated users.
    A/B randomization is at conversation level and same users can contribute multiple trajectories (§3.1); the analysis uses per-trajectory estimators without user-level clustering.
  • domain assumption LLM-judge feature extraction is accurate.
    Features like 'misunderstood intention' are produced by gpt-5-mini (Appendix A.2); errors in these features propagate into f and the feature-difference explanations.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of How can we assess human-agent interactions? Case studies in software agent design." pith.science (2026). https://pith.science/paper/MU5C35BC

@misc{pith2026251009801,
  author       = {Pith},
  title        = {Pith review of: How can we assess human-agent interactions? Case studies in software agent design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MU5C35BC}},
  note         = {Machine review of arXiv:2510.09801}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of real-world use cases. In this paper, we make two major steps towards the rigorous assessment of human-agent interactions. First, we propose PULSE, a framework for more efficient human-centric evaluation of agent designs, which comprises collecting user feedback, training an ML model to predict user satisfaction, and computing results by combining human satisfaction ratings with model-generated pseudo-labels. Second, we deploy PULSE in software engineering -- one of the highest-impact, real-world domains for LLM agents -- via a large-scale web platform built around the open-source agent OpenHands. Across 15k users, we evaluate how three agent design decisions impact developer satisfaction rates. We also show how PULSE can lead to more robust conclusions about agent design, reducing confidence intervals by 40\% compared to a standard A/B test. Finally, we find substantial discrepancies between in-the-wild results with benchmark performance (e.g., the anti-correlation between claude-sonnet-4 and gpt-5), underscoring the limitations of benchmark-driven evaluation. Our framework PULSE provides guidance for future evaluations, and our findings identify opportunities for better software agent designs.

Figures

Figures reproduced from arXiv: 2510.09801 by Aditya Bharat Soni, Ameet Talwalkar, Calvin Smith, Graham Neubig, Hoang H. Tran, Juan Michelini, Rohit Malhotra, Valerie Chen, Xingyao Wang, Xuhui Zhou.

Figure 1
Figure 1. Figure 1: PULSE enables efficient human-centric evaluation of agent designs enables insights into how user experience varies with agent design and comparison to benchmark performance. We instantiate PULSE in a software engineering agent use case and conduct a set of case studies to identify novel insights into designing useful, collaborative agents. for large swaths of human work [36], in reality, current agentic sy… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of statistics of in-the-wild deployment of our human-agent evaluation framework. Our evaluations spanned over 15k users who were working on a diverse set of problems in terms of programming and natural languages, task category, and interaction style with the agent (as seen through user message count). given nc labeled and Nc unlabeled trajectories: µbc(λbc) = 1 nc X i Yi | {z } sample mean of labe… view at source ↗
Figure 3
Figure 3. Figure 3: (bottom) shows how features described in Section 2.2 across conditions change. We 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the feedback setup: (a) the 5-star feedback interface, and (b) the [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Average API cost per-step with varying max step values [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

    cs.SE 2026-08 conditional novelty 7.0

    Injected task-conflicting user edits lower coding agents' average resolve rate by 7.7 points on SWE-bench Verified, and agents often retain or mishandle the conflicting code.

  2. Align AI to Dynamic Human-AI Workflows

    cs.AI 2026-07 conditional novelty 5.0

    Alignment should be redefined as optimizing joint human-AI interaction trajectories in which preferences co-evolve, rather than emulating static preferences over isolated outputs.

  3. Characterizing initial human-AI proof formalization workflows

    cs.AI 2026-06 unverdicted novelty 5.0

    A controlled user study and qualitative survey find that AI assistance raises formalization accuracy for math proofs, with users flexibly combining multiple tools while retaining oversight.

Reference graph

Works this paper leans on

71 extracted references · 24 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Prediction-powered inference.Science, 382(6671):669–674, 2023

    Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I Jordan, and Tijana Zrnic. Prediction-powered inference.Science, 382(6671):669–674, 2023

    2023

  2. [2]

    Ppi++: Efficient prediction- powered inference.arXiv preprint arXiv:2311.01453, 2023

    Anastasios N Angelopoulos, John C Duchi, and Tijana Zrnic. Ppi++: Efficient prediction- powered inference.arXiv preprint arXiv:2311.01453, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Anthropic economic index: Ai’s impact on software development, 2025

    Anthropic. Anthropic economic index: Ai’s impact on software development, 2025

    2025

  4. [4]

    Claude code overview, 2025

    Anthropic. Claude code overview, 2025

    2025

  5. [5]

    Manage costs effectively, 2025

    Anthropic. Manage costs effectively, 2025

    2025

  6. [6]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. 2024

    2024

  7. [7]

    Fast planning through planning graph analysis.Artificial Intelligence, 90(1-2):281–300, 1997

    Avrim Blum and Merrick Furst. Fast planning through planning graph analysis.Artificial Intelligence, 90(1-2):281–300, 1997

    1997

  8. [8]

    Long code arena: a set of benchmarks for long-context code models.arXiv preprint arXiv:2406.11612, 2024

    Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. Long code arena: a set of benchmarks for long-context code models.arXiv preprint arXiv:2406.11612, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657, 2025

    Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail?arXiv preprint arXiv:2503.13657, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Code with me or for me? how increasing ai automation transforms developer workflows.arXiv preprint arXiv:2507.08149, 2025

    Valerie Chen, Ameet Talwalkar, Robert Brennan, and Graham Neubig. Code with me or for me? how increasing ai automation transforms developer workflows.arXiv preprint arXiv:2507.08149, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080, 2024

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Introducing devin, the first ai software engineer, 2024

    Cognition. Introducing devin, the first ai software engineer, 2024. 10

    2024

  13. [13]

    Efficient randomized experiments using foundation models.arXiv preprint arXiv:2502.04262, 2025

    Piersilvio De Bartolomeis, Javier Abad, Guanbo Wang, Konstantin Donhauser, Raymond M Duch, Fanny Yang, and Issa J Dahabreh. Efficient randomized experiments using foundation models.arXiv preprint arXiv:2502.04262, 2025

    arXiv 2025

  14. [14]

    Prediction-powered generalization of causal inferences.arXiv preprint arXiv:2406.02873, 2024

    Ilker Demirel, Ahmed Alaa, Anthony Philippakis, and David Sontag. Prediction-powered generalization of causal inferences.arXiv preprint arXiv:2406.02873, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    An interactive agent foundation model.arXiv preprint arXiv:2402.05929, 2024

    Zane Durante, Bidipta Sarkar, Ran Gong, Rohan Taori, Yusuke Noda, Paul Tang, Ehsan Adeli, Shrinidhi Kowshika Lakshmikanth, Kevin Schulman, Arnold Milstein, et al. An interactive agent foundation model.arXiv preprint arXiv:2402.05929, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Customer feedback and data collection techniques in software r&d: a literature review

    Aleksander Fabijan, Helena Holmstr¨ om Olsson, and Jan Bosch. Customer feedback and data collection techniques in software r&d: a literature review. InInternational Conference of Software Business, pages 139–153. Springer, 2015

    2015

  17. [17]

    Fikes and Nils J

    Richard E. Fikes and Nils J. Nilsson. STRIPS: A new approach to the application of theorem proving to problem solving. InProceedings of the 2nd International Joint Conference on Artificial Intelligence (IJCAI), pages 608–620, 1971

    1971

  18. [18]

    Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al. Magentic-one: A generalist multi-agent system for solving complex tasks.arXiv preprint arXiv:2411.04468, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system

    Zhiyuan Hu, Yue Feng, Anh Tuan Luu, Bryan Hooi, and Aldo Lipani. Unlocking the potential of user feedback: Leveraging large language model as user simulators to enhance dialogue system. InProceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 3953–3957, 2023

    2023

  20. [20]

    Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models.arXiv preprint arXiv:2310.05736, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2023

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2023

    2023

  23. [23]

    Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- r1: Training llms to reason and leverage search engines with reinforcement learning.ArXiv, abs/2503.09516, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Kautz and Bart Selman

    Henry A. Kautz and Bart Selman. Planning as satisfiability. InEuropean Conference on Artificial Intelligence (ECAI), pages 359–363, 1992

    1992

  25. [25]

    Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714, 2023

    Pith/arXiv arXiv 2023

  26. [26]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. 11

    2024

  27. [27]

    Turn-level user satisfaction estimation in E-commerce customer service

    Runze Liang, Ryuichi Takanobu, Feng-Lin Li, Ji Zhang, Haiqing Chen, and Minlie Huang. Turn-level user satisfaction estimation in E-commerce customer service. In Shervin Malmasi, Surya Kallumadi, Nicola Ueffing, Oleg Rokhlenko, Eugene Agichtein, and Ido Guy, editors, Proceedings of the 4th Workshop on e-Commerce and NLP, pages 26–32, Online, August

  28. [28]

    Interpretable user satisfaction estimation for conversational systems with large language models.arXiv preprint arXiv:2403.12388, 2024

    Ying-Chun Lin, Jennifer Neville, Jack W Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, et al. Interpretable user satisfaction estimation for conversational systems with large language models.arXiv preprint arXiv:2403.12388, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Learning attitudes and attributes from multi-aspect reviews

    Julian McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi-aspect reviews. In2012 IEEE 12th International Conference on Data Mining, pages 1020–1025. IEEE, 2012

    2012

  30. [30]

    Gaia: a benchmark for general ai assistants

    Gr´ egoire Mialon, Cl´ ementine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  31. [31]

    Swt-bench: Testing and validating real-world bug-fixes with code agents.Advances in Neural Information Processing Systems, 37:81857–81887, 2024

    Niels M¨ undler, Mark M¨ uller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real-world bug-fixes with code agents.Advances in Neural Information Processing Systems, 37:81857–81887, 2024

    2024

  32. [32]

    When benchmarks talk: Re-evaluating code llms with interactive feedback.arXiv preprint arXiv:2502.18413, 2025

    Jane Pan, Ryan Shar, Jacob Pfau, Ameet Talwalkar, He He, and Valerie Chen. When benchmarks talk: Re-evaluating code llms with interactive feedback.arXiv preprint arXiv:2502.18413, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Prediction-powered inference for clinical trials.medRxiv, 2025

    Pierre-Emmanuel Poulet, Maylis Tran, Sophie Tezenas du Montcel, Bruno Dubois, Stanley Durrleman, and Bruno Jedynak. Prediction-powered inference for clinical trials.medRxiv, 2025

    2025

  34. [34]

    Seizing the agentic ai advantage, 2025

    AI by McKinsey QuantumBlack and McKinsey Technology. Seizing the agentic ai advantage, 2025

    2025

  35. [35]

    Boosting randomized control trials with language models as synthetic units

    Shiv Shankar and Madalina Fiterau. Boosting randomized control trials with language models as synthetic units. 2024

    2024

  36. [36]

    ’i do have a fair amount of concern.’ the ceo of$61 billion anthropic says ai will take over a crucial part of software engineers’ jobs within a year, 2025

    Sherin Shibu. ’i do have a fair amount of concern.’ the ceo of$61 billion anthropic says ai will take over a crucial part of software engineers’ jobs within a year, 2025

    2025

  37. [37]

    Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366, 2023

    Noah Shinn, Francesco Cassano, Divya Gopinath, Anna Rumshisky, and Petar Velick- ovic. Reflexion: Language agents with verbal reinforcement learning.arXiv preprint arXiv:2303.11366, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    John Wiley & Sons, 2015

    Dan Siroker and Pete Koomen.A/B testing: The most powerful way to turn clicks into customers. John Wiley & Sons, 2015

    2015

  39. [39]

    Openhands context condensensation for more efficient ai agents.All Hands AI Blog, April 2025

    Calvin Smith. Openhands context condensensation for more efficient ai agents.All Hands AI Blog, April 2025

    2025

  40. [40]

    Coding agents with multimodal browsing are generalist problem solvers.arXiv preprint arXiv:2506.03011, 2025

    Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. Coding agents with multimodal browsing are generalist problem solvers.arXiv preprint arXiv:2506.03011, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Autopdl: Automatic prompt optimization for llm agents.arXiv preprint arXiv:2504.04365, 2025

    Claudio Spiess, Mandana Vaziri, Louis Mandel, and Martin Hirzel. Autopdl: Automatic prompt optimization for llm agents.arXiv preprint arXiv:2504.04365, 2025. 12

    arXiv 2025

  42. [42]

    Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L

    T. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architec- tures for language agents.ArXiv, abs/2309.02427, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Simulating user satisfaction for the evaluation of task-oriented dialogue systems

    Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2499–2506, New York, NY, USA,

  44. [44]

    Interactive agents to overcome ambiguity in software engineering.arXiv preprint arXiv:2502.13069, 2025

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neu- big. Interactive agents to overcome ambiguity in software engineering.arXiv preprint arXiv:2502.13069, 2025

    arXiv 2025

  45. [45]

    Association for Computing Machinery

  46. [46]

    Llm-powered autonomous agents.lilianweng.github.io, Jun 2023

    Lilian Weng. Llm-powered autonomous agents.lilianweng.github.io, Jun 2023

    2023

  47. [47]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  48. [48]

    Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859, 2024

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains?arXiv preprint arXiv:2410.03859, 2024

    Pith/arXiv arXiv 2024

  49. [49]

    Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprint arXiv:2412.14161, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Masrouter: Learning to route llms for multi-agent systems.arXiv preprint arXiv:2502.11133, 2025

    Yanwei Yue, Guibin Zhang, Boyang Liu, Guancheng Wan, Kun Wang, Dawei Cheng, and Yiyan Qi. Masrouter: Learning to route llms for multi-agent systems.arXiv preprint arXiv:2502.11133, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    React: Syner- gizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Dian Yang, Nan Cui, Karthik Narasimhan, and Yuefeng Cao. React: Syner- gizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    Commit0: Library generation from scratch.arXiv preprint arXiv:2412.01769, 2024

    Wenting Zhao, Nan Jiang, Celine Lee, Justin T Chiu, Claire Cardie, Matthias Gall´ e, and Alexander M Rush. Commit0: Library generation from scratch.arXiv preprint arXiv:2412.01769, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 13 A Additional Framework Details A.1 Data collection details To make feedback minimally intrusive, we prompt user...

    Pith/arXiv arXiv 2023

  55. [55]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. InThirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  56. [57]

    User sentiment: positive, negative, neutral

  57. [58]

    Number of user messages

  58. [59]

    Task Category: we identified 8 common tasks that developers use software agents for

  59. [60]

    Misunderstood Intention: Agent misunderstood the user’s goal or intent

  60. [61]

    Did not follow instruction: Agent ignored or failed to comply with explicit

  61. [62]

    Insufficient Analysis: Didn’t explore existing materials sufficiently (prior code/docs/exam- ples) before acting

  62. [63]

    Insufficient testing: Skipped reasonable verification/tests for non-trivial or risky changes (note: trivial edits may be acceptable)

  63. [64]

    Insufficient Debugging: Did not investigate or reduce failing behavior when needed to make progress

  64. [65]

    Incomplete Implementation: Delivered unfinished or non-functioning work

  65. [66]

    14 11.git commit 12.git push 13.git pull 14.git reset 15.git rebase We use LLM as a judge to detect features 1,3,4,5,6,7,8,9,10

    Scope Creep: Implemented unrequested features without approval. 14 11.git commit 12.git push 13.git pull 14.git reset 15.git rebase We use LLM as a judge to detect features 1,3,4,5,6,7,8,9,10. We analyze the event stream to detect features 2,11,12,13,14,15. Below we include the prompt used withgpt-5-mini. Prompt for labeling features Analyze the fo ll ow ...

  66. [67]

    One sentence d e s c r i b i n g what the user is trying to a c c o m p l i s h

  67. [68]

    Classify the overall se nt im en t of the user ’s messages into one of these c a t e g o r i e s : [ Positive , Negative , Neutral ] and explain why

  68. [69]

    Classify the type of task into exactly one of these c a t e g o r i e s ( choose only one that best fits ) : [ Fix Bugs , I mp le me nt Features , Create Programs from Scratch , Fix Failing C o n t i n u o u s Integration , Fix Merge Conflicts , Write Documentation , Perform Deployments , Perform Data Analysis ]

  69. [70]

    Classify the d e v e l o p m e n t cluster into up to two of these c a t e g o r i e s ( choose only the ones that are the best fits ) : - Write code from scratch - Fix python issues - Fix u n d e r s p e c i f i e d issues - Fix Java issues - Testing code - Web browsing and research - A d m i n i s t r a t i v e tasks - Fix c o n t i n u o u s i n t e g ...

  70. [71]

    t a s k _ d e s c r i p t i o n

    Provide 1 -2 brief example messages from the c o n v e r s a t i o n that support your se nt im en t c l a s s i f i c a t i o n ( truncate if too long ) Format your response as JSON with these fields : {{ " t a s k _ d e s c r i p t i o n ": " one sentence " , " se nt i me nt ": {{ " c l a s s i f i c a t i o n ": " Positive / Negative / Neutral " , " e ...

  71. [2021]

    Association for Computational Linguistics

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.