Pith. sign in

REVIEW 4 major objections 3 minor 25 references

Task alignment—finding out what an underspecified user actually wants before acting—is a measurable ability current LLMs largely lack, with intended-task recovery of 22–32% versus 48% for humans.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Under ambiguous user requests, current LLMs recover the intended task only 22–32% of the time, well below human accuracy of 48%, and post-training only partially closes the gap.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection The framework is a real contribution; the 22–32% headline numbers are simulator-dependent and should be read with that caveat. the 4 major comments →

arxiv 2607.16412 v1 pith:KDZC7ZC4 submitted 2026-07-17 cs.AI

Interactive Task Alignment as a POMDP

Andy Dai , Zexue He , Zhenyu Zhang , Alex Pentland , Jiaxin Pei This is my paper
classification cs.AI
keywords task alignmentunderspecified user intentPOMDPinteractive evaluationuser simulationintent evolutionLLM agentsuncertainty resolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the real-world bottleneck for language-model assistants is not executing a known task but figuring out which task an underspecified user actually wants. It formalizes that skill as task alignment: the assistant must infer a latent task specification from partial, evolving hints in dialogue and decide when to stop asking and act. Converting three existing benchmarks (shopping, coding, professional work) into interactive, underspecified episodes, the authors find that current LLMs recover the intended task only 22–32% of the time, versus 48% for humans. The paper further shows that task alignment is a separable, domain-general ability—performance does not track task capability—and that the human advantage comes from asking better questions, not from better inference. Post-training with SFT and RL substantially improves alignment, but models still lag behind humans.

Core claim

The paper claims that task alignment is a measurable, separable ability distinct from task capability: across shopping, coding, and professional-work settings, 14 state-of-the-art LLMs recover the user's intended specification only 22–32% of the time when the user starts from abstract intent, even though the same models largely succeed once the task is fully specified. Humans in the same shopping setting reach 48.0%. The paper further claims that the human advantage is communication, not inference: when LLMs are given the human assistant's transcripts, median task recovery rises to 49.5%, close to the human level. Finally, post-training Qwen3.5-9B with SFT and RL roughly doubles CommitReward

What carries the argument

The central object is a POMDP with a latent task specification x* and a stateful user-intent process. At each episode the ground-truth x* is hidden; a user simulator with privileged access to x* starts from an abstract, moderate, or concrete intent seed and updates its intent via an "intent evolver" E that moves toward x* only as fast as the conversation justifies. The assistant observes only the dialogue history and must choose (a) a free-form response, (b) a stop/commit signal that ends the episode, and (c) a ranking ρt over a fixed set of 15 candidate specifications (containing x*) produced in a separate thread. This ranking is the measurement instrument: it exposes the assistant's curren

Load-bearing premise

The LLM-based user simulator, with its intent evolver that gradually reveals a privileged ground-truth task, faithfully models how real users expose and change their goals; the headline numbers are measured against this simulator, and the paper's own ablation shows the evolver design materially changes results.

What would settle it

Run identical task-alignment episodes with real human users who hold genuinely abstract, novel goals (not recalled purchases) as the user role, and measure the assistant's CommitAcc@1 against the simulator-based figure. If human-user recovery is substantially above 22–32%, the simulator overstates the failure; if rates are near or below, the numbers hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Existing capability benchmarks likely overstate real-world utility: a model that can execute a task once specified may still fail as an agent because it cannot identify the user's goal.
  • Task alignment can be measured and trained separately from task capability; it appears domain-general, so improvements in one domain may transfer to others.
  • Post-training on alignment (SFT/RL) reduces confidently-wrong behavior and increases honest uncertainty, providing a concrete training signal (CommitReward) for more reliable agents.
  • Human-level interactive alignment (~48%) sets a target that current models remain far from, suggesting interaction skill—not reasoning—is the current bottleneck.
  • The framework applies to any benchmark with specifiable tasks, so existing evaluation suites can be converted into interactive alignment tests without new task design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since the intent evolver ablation materially changes measured performance on every assistant, real-user intent evolution is likely even more variable; the 22–32% figures should be read as tied to this simulator design rather than as canonical numbers.
  • The confidence/entropy metrics are available only for a subset of models, so the four-quadrant taxonomy may be partly an artifact of which model families emit logprobs; a standardized calibration protocol across all models would clarify this.
  • The human-transcript replay result suggests an asymmetric opportunity: improving the assistant's question-asking policy may yield larger gains than improving the underlying inference model, and human assistant transcripts could serve as training data.
  • Measuring alignment against a closed 15-candidate set of plausible tasks may underestimate real-world difficulty, where the space of possible tasks is open-ended; if so, the reported gap is conservative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a POMDP-based framework for evaluating task alignment: given a fully specified benchmark task x*, it constructs an underspecified interaction with an LLM user simulator whose latent intent evolves from abstract to concrete, and asks an assistant to identify x* among 15 candidates before committing. The framework is instantiated on Shopping-MMLU, GDPVal, and Terminal-Bench with 14 LLMs. The paper reports abstract-tier CommitAcc@1 between 22% and 32% for models, 48% for humans on a small Shopping subset; argues that task alignment is separable from task capability and domain-general; shows that SFT/RL on Qwen3.5-9B improves alignment; and performs a post hoc human validation of the simulator.

Significance. If the results hold, the paper makes a useful contribution: it operationalizes a capability that most benchmarks ignore, separates task discovery from execution, and offers metrics that distinguish guessing from genuine uncertainty resolution. Strengths include the explicit POMDP formulation, the empty-conversation probe for candidate-set leakage, the inference-replay ablation isolating communication from inference, and the authors' candor in the Limitations section. The qualitative finding that LLMs under-resolve ambiguous requests is credible and supported by several ablations. However, the headline numeric claims rely on an abstract-tier user simulator that has no direct human validation, and the human comparisons are statistically underpowered; the exact magnitudes should be treated as provisional until the robustness analyses are added.

major comments (4)
  1. [§3.2, Table 2; Limitations; Appendix A.3, Table 7] The headline '22–32% recovery' is a property of the authors' LLM user simulator, not a model-independent measurement. The Limitations section states that the absolute numbers are 'tied to a particular simulator design rather than a canonical user model,' and the evolver ablation shows that removing the intent evolver raises CommitReward by roughly 0.19–0.28 on the concrete tier (e.g., GPT-5.4: 0.98→1.22). Since the human validation (Appendix A.2) covers only the concrete tier and the authors state that abstract intent cannot be captured in humans, the abstract-tier simulation—the source of the headline range—has no direct external check. Please provide a robustness analysis over plausible evolver/user-policy variants, or re-center the claims on relative/qualitative comparisons that are stable across simulator choices.
  2. [§3.5, Tables 4–5] The claim that 'humans outperform all evaluated models' is not backed by a statistical test. The human mean is 48.0±7.1% and the best model (GLM-5.1) is 36.9±6.0%; the difference is roughly 1.2 standard errors of the difference and is not significant at conventional levels. Moreover, the comparison is based on only 5 products; per-item human scores range from 18.2% to 66.7%, and several models achieve 100% on the 'dog treats' item, so the aggregate is highly sensitive to item selection. Report a paired or mixed-effects test across items/participants, or soften the claim to a descriptive finding.
  3. [Appendix A.2, Table 6] The claim that human and simulated users are 'comparable and within 1 SEM' holds only after pooling. For gpt-5.4, human CommitReward is 1.15±0.04 versus simulated 0.98±0.05, a difference of about 2.6 pooled SEM; CommitAcc@1 is 50.0 versus 36.5. The pooled comparison hides a substantial per-assistant discrepancy. Report per-assistant tests and justify whether the aggregate 'within 1 SEM' conclusion is appropriate, or qualify the simulator-validation claim accordingly.
  4. [§3.4, Figure 5] The capability-orthogonal component of CommitRank variance is reported as 39% with bootstrap p=0.055 on GDPVal, which is not significant at the 0.05 level. While the Shopping control and residual Kendall's W provide supporting evidence for separability, the text's phrasing '39% of variance on both tasks' overstates the GDPVal result. Report the confidence interval and interpret the GDPVal point estimate with the appropriate caveat.
minor comments (3)
  1. [§2.2, Eq. for CommitRank] The 'normalized rank' metric is inverted: a higher value means a better (lower) rank position. Consider calling it 'normalized inverse rank' or explicitly noting the inversion to avoid confusion.
  2. [Appendix A.3, spec-token coverage] The definition of 'content tokens' should be pinned down (e.g., stopword removal, tokenization scheme). As written, the metric depends on an unspecified preprocessing choice.
  3. [Throughout] Minor typos and phrasing issues, e.g., 'Figure 3 showcases a subset models' behavior' should read 'a subset of models' behavior'; several references to 'Figure 3' appear before the figure is introduced. A careful proofread is recommended.

Circularity Check

0 steps flagged

No significant circularity: the headline recovery rates are simulator-dependent measurements, not results forced by definition; the main caveat is external validity, not circularity.

full rationale

The paper's derivation chain is an evaluation pipeline, not a derivation that re-imports its conclusion. Task alignment is operationalized as a POMDP with latent x* and an LLM user simulator; CommitAcc@1/CommitRank/CommitReward are defined as functions of the model's ranking of x* in candidate set C. The 22–32% figure is an empirical measurement of 14 models under that simulator, not an equation-level identity: nothing in the metric definitions forces a particular recovery rate, and no parameter is fitted to the reported values and then renamed a prediction. The 'separable axis' claim is supported by a variance decomposition (capability explains 43%/50%, orthogonal component 39%) and the 'domain-general' claim by Kendall's W=0.62 residual W=0.58; these are empirical statistics. The human study is an independent external check at the concrete tier, and the inference-replay ablation is a controlled comparison. The paper itself flags the one genuine weakness: Limitations (2) states 'it is not possible to capture abstract intent in humans' and 'our absolute numbers are tied to a particular simulator design rather than a canonical user model,' and the intent-evolver ablation (Table 7) shows the simulator design materially changes CommitReward. That makes the abstract-tier headline vulnerable to simulator choice, but it is a construct-validity/robustness problem, not circularity: the headline does not reduce to the framework's inputs by construction, and there is no load-bearing self-citation chain. Post-training results are standard held-out evaluations (train on concrete tier, evaluate on abstract tier) and are not self-referential predictions. Accordingly no circular step is exhibited.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central measurement depends on the authors' simulator, candidate-set construction, metric coefficients, and tier definitions. These are all explicitly described, which is good, but the headline numbers inherit the assumptions. There are no newly postulated physical or conceptual entities; the 'latent task' and 'intent state' are standard latent-variable constructs.

free parameters (5)
  • CommitReward coefficients α, λ = α=1.5, λ=0.02 (evaluation); α=1.0, λ=0 (RL training)
    Hand-chosen trade-off weights between terminal rank and conversation length; all reward-based headline numbers depend on them.
  • Candidate set size |C|=15, N=14 distractors = 15 / 14
    Design choice that sets the chance baseline at 6.7%; all reported accuracies are relative to this constructed set.
  • Interaction cap of 15 turns = 15 user + 15 assistant turns
    Caps each episode; affects commit timing and reward; several Qwen models hit this cap.
  • Intent tier definitions (abstract/moderate/concrete) = word/character constraints in prompts
    Hand-authored tiers determine difficulty; the abstract tier drives the 22–32% headline, and abstract intent cannot be human-validated.
  • Low/high entropy median split = median across all rollouts
    Four-quadrant uncertainty taxonomy depends on this dataset-dependent median split.
axioms (6)
  • domain assumption LLM user simulator with intent evolver faithfully emulates how real users gradually specify their goals
    Central to all measurements; validated post hoc only at the concrete tier, and the paper acknowledges the simulator cannot capture the full distribution of real user behavior (Limitations; Appendix A.3).
  • domain assumption Letter-shuffle logprob softmax provides a valid estimate of assistant belief over candidates
    Appendix C defines belief from next-token letter logprobs; only 5 of 14 evaluated models emit usable confidences, and the calibration step adds further assumptions.
  • domain assumption Candidate sets do not leak the ground-truth task
    Validated with an empty-conversation probe (Table 8), but the probe uses the same ranker and the acceptance criteria are author-chosen.
  • domain assumption Reused capability baselines from the Artificial Analysis leaderboard are comparable across models
    GDPVal and Terminal-Bench capability scores are taken from an external leaderboard (Section 3.1), and the variance decomposition relies on them as a covariate.
  • domain assumption Human users specifying a past purchase approximates the concrete intent tier
    The paper states that humans in the validation study specify tasks they have already completed, which maps conceptually to concrete intent; abstract/moderate tiers were not human-validated.
  • standard math Standard POMDP formalism
    Section 2.1 uses textbook POMDP definitions for state, action, transition, and observation.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Interactive Task Alignment as a POMDP." pith.science (2026). https://pith.science/paper/KDZC7ZC4

@misc{pith2026260716412,
  author       = {Pith},
  title        = {Pith review of: Interactive Task Alignment as a POMDP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDZC7ZC4}},
  note         = {Machine review of arXiv:2607.16412}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Current benchmarks for language models primarily evaluate execution on fully specified tasks. However, real user tasks are often ambiguous. Users arrive with incomplete, exploratory, or even inconsistent goals, requiring the assistant to first determine the intended task before carrying it out. We study this problem as task alignment: the ability to align with a user on their intended task. We introduce a general framework for converting specified tasks into underspecified interactions, formalized as a POMDP in which the model must infer a latent task from partial and evolving user intent. We validate our user simulator post hoc with a human user study. Across shopping, coding, and professional work settings, we find that while models often perform well once the task is specified, models still struggle with task alignment: current models act prematurely, interact ineffectively, and fail to resolve ambiguous requests. Models on average recover the user's intended task only 22-32% of the time under ambiguity. In a human study in the same setting, humans reach 48%, outperforming all evaluated models. We show that post-training with supervised fine-tuning and reinforcement learning improves task alignment, but models still lag behind humans in resolving uncertainty through interaction. Together, our results suggest that current models still lack key interaction abilities required for reliable agency.

Figures

Figures reproduced from arXiv: 2607.16412 by Alex Pentland, Andy Dai, Jiaxin Pei, Zexue He, Zhenyu Zhang.

Figure 1
Figure 1. Figure 1: Task alignment un￾der uncertainty. Users provide observable instructions, while their real needs remain latent. We study the task alignment stage: identifying the intended task through interaction before executing the task. Although task execution may involve additional latent environment states, these downstream execu￾tion dynamics are out of scope. To this end, we propose a domain-general framework for t… view at source ↗
Figure 2
Figure 2. Figure 2: Workflow of our POMDP-based task-alignment framework, illustrated with an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-turn trajectories of belief entropy H˜ t (left) and normalized rank rˆt (right) at the abstract tier, one row per task. Score-vector subset (GPT-5.4 family, Qwen3.5-9B, Qwen3.5-397B); trained Qwen3.5-9B SFT / RL added on Shopping. Trajectory lines terminate at mean ⌈T ∗ ⌉, and already-terminated rollouts contribute their terminal value thereafter for icon placement. GPT￾5.4 family trajectories end earl… view at source ↗
Figure 4
Figure 4. Figure 4: Normalized entropy H˜ at commit vs. normalized rank at commit, per model (mean ± SEM), across all three tasks. Qwen3.5 extracts more information from conversation than GPT-5.4 Qwen3.5 achieves substantial entropy reductions on every task (TotalIG up to 0.29 on Shopping and 0.21 on Terminal-Bench), driven by more interaction turns (T ∗ ∈ [6, 8]), while the GPT-5.4 family hovers around TotalIG = 0 on Shoppin… view at source ↗
Figure 5
Figure 5. Figure 5: Variance decomposition of across-model alignment (CommitRank) per task. We now analyze the relationship between task capability and alignment. Models that perform tasks well may also trivially identify them well, in which case our evaluation setting would add little beyond existing capability evaluations. We test this by exploiting the capability profiles of our three settings (Section 3.1). We use Shop￾pi… view at source ↗
Figure 6
Figure 6. Figure 6: Human evaluation: (a) humans outperform all models at task recovery; (b) the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Human vs. simulated user, pooled across the three assistants ( [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Per-turn spec-token coverage = |Ut ∩ S|/|S|, where Ut is the set of unique content tokens in the cumulative user text up to turn t and S is the set of unique content tokens in the ground-truth specification. Averaged across the 14 models at the abstract tier; shadings are ±SEM across models. B Task setting creation Task setting creation turns each fully specified benchmark task x ⋆ into a size-15 candidate… view at source ↗
Figure 9
Figure 9. Figure 9: Zero-shot and full-specification performance. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-model behavioral statistics for all 14 evaluated models. [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Behavioral decomposition along (1) the model’s initial guess [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Commit turn T ⋆ vs. normalized rank at commit, per model (abstract tier). Left: Shopping. Middle: GDPVal. Right: Terminal-Bench. Full 14-model per-turn rank trajectories. Figures 13–15 extend the main-body per-turn trajectory ( [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Shopping: per-turn trajectories for all 14 models, split into 2 groups of 7. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: GDPVal: per-turn trajectories for all 14 models, split into 2 groups of 7. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Terminal-Bench: per-turn trajectories for all 14 models, split into 2 groups of 7. [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Cumulative information gain TotalIG = H˜ 0 − H˜ T⋆ per (model, task), averaged across the three intent tiers. Bars are mean ± pooled SEM across tiers. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Qwen3.5-9B training curves, stacked top-to-bottom. [PITH_FULL_IMAGE:figures/full_fig_p028_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Qwen3.5-9B per-turn normalized entropy H˜ t on Shopping abstract for base, SFT, and RL. F Prompts Task-specific templates (ranker, user simulator, candidate ranking, intent evolver, candidate generation) are shown here in their Shopping form. GDPVal and Terminal-Bench variants have the same structure and only substitute domain nouns (“product” → “professional task specification” / “terminal-task instructi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 15 linked inside Pith

  1. [1]

    Star-gate: Teaching language models to ask clarifying questions

    Chinmaya Andukuri, Jan-Philipp Fr \"a nken, Tobias Gerstenberg, and Noah D Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Learning to clarify: Multi-turn conversations with action-based contrastive self-training

    Maximillian Chen, Ruoxi Sun, Tomas Pfister, and Sercan Arik. Learning to clarify: Multi-turn conversations with action-based contrastive self-training. In International Conference on Learning Representations, volume 2025, pp.\ 32244--32279, 2025

    2025

  3. [3]

    Clark and Susan E

    Herbert H. Clark and Susan E. Brennan. Grounding in communication, 1991. URL https://psycnet.apa.org/record/1991-98452-006

    1991

  4. [4]

    Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Z. Li. Bayesian preference elicitation with language models, 2024. URL https://arxiv.org/abs/2403.05534

    Pith/arXiv arXiv 2024

  5. [5]

    Teaching language models to gather information proactively

    Tenghao Huang, Sihao Chen, Muhao Chen, Jonathan May, Longqi Yang, Mengting Wan, and Pei Zhou. Teaching language models to gather information proactively. Findings of the Association for Computational Linguistics: EMNLP 2025, pp.\ 15588--15599, 2025

    2025

  6. [6]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  7. [7]

    Shopping mmlu: A massive multi-task online shopping benchmark for large language models, 2024

    Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, Qiang Yang, Meng Jiang, and Bing Yin. Shopping mmlu: A massive multi-task online shopping benchmark for large language models...

    Pith/arXiv arXiv 2024

  8. [8]

    Discoverllm: From executing intents to discovering them, 2026

    Tae Soo Kim, Yoonjoo Lee, Jaesang Yu, John Joon Young Chung, and Juho Kim. Discoverllm: From executing intents to discovering them, 2026. URL https://arxiv.org/abs/2602.03429

    Pith/arXiv arXiv 2026

  9. [9]

    Li, Been Kim, and Zi Wang

    Belinda Z. Li, Been Kim, and Zi Wang. Questbench: Can llms ask the right question to acquire information in reasoning tasks?, 2025 a . URL https://arxiv.org/abs/2503.22674

    arXiv 2025

  10. [10]

    Prefpalette: Personalized preference modeling with latent attributes

    Shuyue Stella Li, Melanie Sclar, Hunter Lang, Ansong Ni, Jacqueline He, Puxin Xu, Andrew Cohen, Chan Young Park, Yulia Tsvetkov, and Asli Celikyilmaz. Prefpalette: Personalized preference modeling with latent attributes. arXiv preprint arXiv:2507.13541, 2025 b

    Pith/arXiv arXiv 2025

  11. [11]

    Clarifymt-bench: Benchmarking and improving multi-turn clarification for conversational large language models, 2025

    Sichun Luo, Yi Huang, Mukai Li, Shichang Meng, Fengyuan Liu, Zefa Hu, Junlan Feng, and Qi Liu. Clarifymt-bench: Benchmarking and improving multi-turn clarification for conversational large language models, 2025. URL https://arxiv.org/abs/2512.21120

    arXiv 2025

  12. [12]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, An...

    Pith/arXiv arXiv 2026

  13. [13]

    Asking clarifying questions for preference elicitation with large language models, 2025

    Ali Montazeralghaem, Guy Tennenholtz, Craig Boutilier, and Ofer Meshi. Asking clarifying questions for preference elicitation with large language models, 2025. URL https://arxiv.org/abs/2510.12015

    arXiv 2025

  14. [14]

    Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Tworek. Gdpval: Evaluating ai model performance on real-worl...

    Pith/arXiv arXiv 2025

  15. [15]

    Tell me more! towards implicit user intention understanding of language model driven agents

    Cheng Qian, Bingxiang He, Zhong Zhuang, Jia Deng, Yujia Qin, Xin Cong, Zhong Zhang, Jie Zhou, Yankai Lin, Zhiyuan Liu, et al. Tell me more! towards implicit user intention understanding of language model driven agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1088--1113, 2024

    2024

  16. [16]

    Ambig-swe: Interactive agents to overcome underspecificity in software engineering, 2026

    Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, and Graham Neubig. Ambig-swe: Interactive agents to overcome underspecificity in software engineering, 2026. URL https://arxiv.org/abs/2502.13069

    arXiv 2026

  17. [17]

    Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning

    Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025 a

    Pith/arXiv arXiv 2025

  18. [18]

    Position: Humans are missing from ai coding agent research, 2025 b

    Zora Zhiruo Wang, John Yang, Kilian Lieret, Alexa Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik Narasimhan, Ludwig Schmidt, et al. Position: Humans are missing from ai coding agent research, 2025 b

    2025

  19. [19]

    Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL https://arxiv.org/abs/2504.12516

    Pith/arXiv arXiv 2025

  20. [20]

    Collabllm: From passive responders to active collaborators, 2025

    Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators, 2025. URL https://arxiv.org/abs/2502.00640

    Pith/arXiv arXiv 2025

  21. [21]

    Aligning llms with individual preferences via interaction, 2024

    Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. Aligning llms with individual preferences via interaction, 2024. URL https://arxiv.org/abs/2410.03642

    Pith/arXiv arXiv 2024

  22. [22]

    Webshop: Towards scalable real-world web interaction with grounded language agents, 2023

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. URL https://arxiv.org/abs/2207.01206

    Pith/arXiv arXiv 2023

  23. [23]

    Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi. Modeling future conversation turns to teach llms to ask clarifying questions, 2025. URL https://arxiv.org/abs/2410.13788

    Pith/arXiv arXiv 2025

  24. [24]

    Clarify when necessary: Resolving ambiguity through interaction with lms

    Michael JQ Zhang and Eunsol Choi. Clarify when necessary: Resolving ambiguity through interaction with lms. In Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 5526--5543, 2025

    2025

  25. [25]

    Do llms recognize your preferences? evaluating personalized preference following in llms, 2025

    Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms, 2025. URL https://arxiv.org/abs/2502.09597

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.