Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

Bo Wang; Leilei Sun; Liangzhe Han; Ruixing Zhang; Tongyu Zhu; Yang Zhang; Yunqi Liu

REVIEW 3 major objections 3 minor 18 references

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read User simulators can be steered by a six-way intent directive, and the authors' model reaches 86.6% turn-level intent accuracy on real human–assistant conversations while preserving response fidelity.

desk verdict Per-turn intent control is a real contribution, but the flashiest within-context number is self-measured by the same verifier that trained the model—size uncertain, effect real. read the letter →

arxiv 2608.09420 v1 pith:77WZAHJA submitted 2026-08-10 cs.CL

Bo Wang , Ruixing Zhang , Yunqi Liu , Yang Zhang , Liangzhe Han , Tongyu Zhu , Leilei Sun This is my paper · ORCID

classification cs.CL

keywords usersimulationinteractionintentintent-conditionedgenerationintent-calibratedpolicyoptimizationcontrollabledialogueevaluationLMSYS-Chat-1MgrouprelativeLLMsimulators

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the missing control variable in user simulation is the per-turn interaction intent: what the next user turn is supposed to do in the conversation, as opposed to how it is phrased. Because the same dialogue context supports many valid continuations, a simulator that only imitates response style can produce a fluent turn that advances the dialogue in the wrong direction, for example accepting an answer that should be repaired. To fix this, the authors introduce UserIDA, which conditions generation on an explicit six-way intent directive, trains the simulator with supervised fine-tuning, and then applies intent-calibrated policy optimization that ensures intent-violating candidates never outrank compliant ones. On the LMSYS-USP benchmark the resulting simulator reaches 86.6% turn-level intent accuracy, outperforming the strongest dedicated baseline by 24.3 percentage points while preserving or improving semantic and stylistic fidelity. If the claim holds, interaction intent is a complementary, controllable dimension of user simulation rather than something that can be left implicit in response imitation.

What carries the argument

The load-bearing object is the intent directive, a six-way categorical interface that separates what the next user turn should do from how it is expressed. It is implemented as an additional control input in the prompt, and it is verified by a frozen retrospective classifier that assigns one of the six canonical labels to any generated or observed user turn. The other central mechanism is the intent-calibrated relative reward: for each group of $K$ sampled candidates the method computes a group-specific shift $\Delta_t = \max(0, \max_{i:b_i=0} q_i - \min_{j:b_j=1} q_j + m)$, subtracts it from every violating candidate's quality score, and proves that the resulting ordering survives group-level standardization. A fixed intent bonus would fail because quality-score spread can overwhelm any constant; calibration instead computes the minimum shift needed to separate compliant from violating sets while retaining quality discrimination among compliant candidates.

What would settle it

Take a balanced sample of dialogue states and have independent human annotators label the generated user turns for the six intents, focusing on the confusable pairs Initiate/Amend, Amend/Repair, and Repair/SetRegister. If human-judged intent accuracy falls substantially below the verifier-based 86.6% figure for the same generated turns, the central claim of reliable per-turn intent control collapses; if an independent audit reproduces the small human-versus-verifier gap reported in the paper's own Appendix C.5, the claim stands.

Watch

Extended reading notes

Core claim

The central claim is that next-user behavior should be modeled as a conditional realization of a chosen local interaction intent, not as a marginal imitation of observed user turns. UserIDA instantiates this by factorizing the user policy as $\pi(u_t \mid c_t, p) = \sum_{z \in \mathcal{Z}} P(z \mid c_t, p) \, \pi(u_t \mid c_t, p, z)$, exposing the categorical intent $z$ as an explicit per-turn directive. The six-way taxonomy is deliberately surface-underspecified: it describes what the turn does to the dialogue state (opens a thread, amends an open task, supplies material, repairs a mismatch, sets a register, or grounds, accepts, and closes), without constraining wording. Training proceeds in two stages: intent-conditioned supervised fine-tuning teaches the model to generate distinct valid continuations from the same context under different directives, and intent-calibrated policy optimization fixes the quality–intent mismatch by shifting all violating candidates' rewards below all compliant candidates within each sampled group. The paper's evidence is that this produces 86.6% turn-level intent accuracy, 58% all-turn trajectory success (up from 13% for the strongest baseline), and 91.7% within-context four-of-six directive success, with quality scores at least as high as before.

Load-bearing premise

The results stand on a frozen classifier's intent labels being trustworthy: the same model labels the training data, the reinforcement-learning rewards, and the evaluation metric, so any systematic confusion between nearby intents (for example Amend versus Initiate) could inflate the reported accuracy beyond what human judges would confirm, and the human audit in Appendix C.5 mitigates but does not eliminate this risk.

Editorial extensions

If this is right

Scripted directive sequences become a practical way to generate reproducible counterfactual dialogues, since the simulator can be told to repair, amend, or accept at each step instead of leaving the next action to chance.
Trainers of interactive assistants can now stress-test error recovery explicitly: forcing a simulated user to issue Repair turns reveals whether an assistant detects and corrects its own mismatches, which imitation-based simulators often mask.
Because quality-only reinforcement learning in the ablation reduces intent accuracy, future simulator training objectives should treat intent adherence as a separate reward axis rather than expecting it to emerge from response fidelity.
The per-intent results show that the interface helps minority intents as well as dominant ones; macro-F1 should be reported alongside accuracy for any work building on this interface.
In a diagnostic closed-loop pilot, intent-controlled user turns also improved goal success and constraint satisfaction, suggesting the gains survive when simulated turns are actually fed back into an assistant conversation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The six-way intent interface could double as a compact action space for training assistant policies: an assistant could first choose which local transition to perform and then select words, mirroring the factorization this paper applies to the user.
The taxonomy was built on open-domain English chat data; a transfer experiment to task-oriented or customer-support dialogues would test whether the same six labels stay operational when Supply and Repair dominate.
The calibration guarantee is an ordering guarantee within each sampled group, not a guarantee about absolute update size; future variants could add per-intent margins or absolute floors to control how strongly violating candidates are pushed down.
Swapping the frozen verifier for an independent classifier and rerunning only the turn-level test would quantify how much of the reported 86.6% is verifier-specific rather than human-visible intent control.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: UserIDA gives user simulation something it didn't have—an explicit per-turn intent knob you can set at generation time—and the paper shows, with mostly honest evidence, that you can turn that knob reliably. The catch: the headline within-context number is measured with the same verifier that trained the model, so its exact size is questionable even though the effect is real.

What's new: prior LLM simulators condition on profiles and global goals but leave the local action ambiguous. UserIDA defines a six-way intent interface (Initiate, Amend, Supply, Repair, SetRegister, GroundAccept), trains with directive-conditioned SFT, then applies a clever calibration: instead of a fixed intent bonus, it computes the minimum group-specific reward shift that puts every violating candidate below every compliant one. Appendix B's ordering guarantee is simple and correct. The quality-only GRPO ablation cleanly demonstrates the motivating problem—response quality can improve while intent adherence drops—and the full method recovers both. They also run a human audit on generated turns (C.5) that confirms the system ordering UserIDA > UserIDA w/o RL > USP on the natural distribution, and they report verifier-human agreement (87.88% acc, kappa 0.82) with honest boundary confusion stats. Code is public.

Where it's soft: the same frozen Qwen3.5-9B verifier labels the training data, supplies the RL reward, and scores all automatic metrics. The human audit does not cover the within-context intervention protocol in Table 4, the one producing the flashiest number (91.7% 4-of-6). In that protocol the states are pre-screened for six-way feasibility and every directive is requested once, so surface cues like imperative frames for SetRegister or long payloads for Supply could make the verifier easier to satisfy than a human judge. The boundary confusions in Table 10 (Initiate-Amend, etc.) show the verifier isn't perfect. So the exact size of the within-context success is uncertain. That's a fixable issue: collect human labels on that exact intervention set and report CIs for the main tables. It doesn't sink the paper—the qualitative case in Figure 5 is convincing, the turn-level human audit is reassuring—but it means the strongest claim is not yet fully independent of the measuring instrument.

Bottom line: per-turn intent control is a genuinely useful contribution to user simulation and synthetic assistant evaluation. It deserves a serious referee and could be a solid paper after adding the missing human labels. I'd read it carefully and would cite it once the intervention numbers are externally verified.

Referee Report

3 major / 3 minor

Summary. The paper proposes UserIDA, a user-simulation framework that separates the per-turn interaction intent of a simulated user from its surface realization. It defines a six-way canonical intent interface (Initiate, Amend, Supply, Repair, SetRegister, GroundAccept), trains a directive-conditioned generator with supervised fine-tuning, and applies an intent-calibrated group-relative policy optimization whose reward explicitly ranks intent-compliant candidates above intent-violating ones. Experiments on LMSYS-USP report large gains in turn-level intent accuracy, multi-turn trajectory success, and within-context directive interventions, alongside human audits on generated turns and a small recursive closed-loop pilot.

Significance. If the results hold, the paper makes a useful and timely contribution: it identifies a missing control variable in next-user simulation, provides a concrete six-way interface that is surface-underspecified, and demonstrates that explicit intent calibration improves directive adherence without degrading semantic or stylistic fidelity. The Appendix B ordering proof is a clean formal property of the calibration, the ablations isolate the effect of intent calibration, and the human audit in Appendix C.5 is a genuine external check on the turn-level ordering. The paper also releases code and includes a recursive closed-loop pilot, which strengthens reproducibility. The main caveat is that the headline automatic metrics, especially the Table 4 within-context intervention numbers, are produced by the same frozen verifier used during annotation and reward computation; the human audit does not cover that protocol. This is fixable but must be addressed before the central claim can be fully accepted.

major comments (3)

[§4.2, Eq. (4); Appendix C.5; Appendix E.3; Table 4] The same frozen verifier g_ret supplies corpus labels (Appendix C), the RL compliance signal (Eq. 4), and the intent-accuracy labels in Tables 2–4, including the 91.7% 4-of-6 Success in Table 4. Appendix B's ordering guarantee is correctness relative to g_ret's labels, not to human intent judgments, and Table 10 shows substantial boundary errors (e.g., 10.32% Amend→Initiate). The human audit in Appendix C.5 covers generated turns under the natural test distribution and does not evaluate the within-context intervention protocol of Table 4, where each state is pre-selected for six-way feasibility and surface cues may make the verifier easier to satisfy. Please add a human evaluation on the Table 4 intervention set (or an independently trained verifier) and report the resulting 4-of-6 Success; without it, the headline number may measure verifier consistency rather than human-judged intent control.
[Tables 2–4] Main tables report point estimates without confidence intervals or significance tests; the only interval is the paired-bootstrap improvement for Table 4 reported in the text. Because the margins are large this may be a formality, but for claims that UserIDA outperforms baselines by 24.3 points on a verifier-derived metric, report bootstrap confidence intervals (or per-seed variance) for the three main tables so readers can assess the stability of the stated ordering.
[§4.2, Eqs. (4)–(5)] The paper states that all-violating groups provide no relative intent preference and that the all-violating-group rate is recorded 'as a diagnostic', but this rate is never reported. If a substantial fraction of groups, particularly for rare intents such as Supply (1.9% of training turns), contains no compliant candidate, the calibration may leave rare-intent performance unchanged; please report the all-violating-group rate overall and per intent, and discuss its effect on the RL result.

minor comments (3)

[Figure 3] The per-intent figure uses abbreviated labels 'ACCEPT' and 'REGISTER' while the taxonomy and tables use 'GroundAccept' and 'SetRegister'; please unify the labels to avoid confusion.
[Abstract and Table 4] The abstract's '22.9% for the strongest external baseline' is somewhat ambiguous because Table 4 reports different external baselines; please name the exact system (LLaMA-3-8B Base with Directive) in the abstract or table caption.
[Appendix E.3] The construction of the within-context intervention set says it is 'balanced across six domains, four dialogue-depth buckets, and two levels of profile salience' but does not report how many states survive the affordance screen; please report the number of retained states and the screen's inter-annotator agreement.

Circularity Check

1 steps flagged · score 5.0 of 10

Headline intent-accuracy numbers are scored by the same frozen verifier that labeled training data and supplied RL rewards; the C.5 human audit only partially breaks the loop.

fitted input called prediction [Experiments, 'Intent Directive Annotation'; Eqs. (4)-(5); Appendix E.1 Eq. (13)]
"We use one frozen Qwen3.5-9B retrospective verifier g_ret (Qwen Team 2026) for corpus annotation, rollout compliance, and automatic intent evaluation. ... For each candidate, a frozen canonical verifier predicts the realized interaction intent: ẑ_i = g_ret(c_t, p, y_i). We define exact directive compliance as b_i = 1[ẑ_i = z_t]."

The same g_ret produces the SFT training labels, defines the RL compliance bit b_i in Eq. (4), and computes every automatic Intent Accuracy and Macro-F1 in Tables 2-4 via Eq. (13). UserIDA is rewarded for increasing 1[g_ret(...) = z_t], and the headline 'intent accuracy' is then the frequency of exactly that event. Thus the 86.6% turn-level and 91.7% within-context numbers are agreement-with-the-training-labeler scores rather than direct human-judged intent realization. The C.5 human audit of generated turns is a genuine external anchor for the turn-level ordering, but it does not evaluate the within-context intervention protocol of Table 4, whose 4-of-6 Success is scored entirely by g_ret.

full rationale

The core derivation is otherwise self-contained: Intent-SFT (Eq. 2) is a standard conditional language-model loss, and the RL update (Eqs. 4-7) is a transparent group-relative policy objective with an explicit ordering guarantee in Appendix B. No load-bearing self-citation, uniqueness theorem, or imported ansatz forces the design; the six-intent taxonomy is operationally defined in Appendix A and checked against expert labels. The only substantive circularity is evaluative: one frozen verifier simultaneously labels the corpus, provides the RL compliance signal, and calculates the main automatic metrics, so the headline numbers in Tables 2-4 partially measure consistency with the model's own training signal. The paper's C.5 human audit on generated turns mitigates this for the turn-level claim, preserving the UserIDA > UserIDA w/o RL > USP ordering and giving UserIDA 90.83% balanced human intent accuracy. However, that audit samples the natural test distribution and does not cover the within-context intervention protocol behind the headline 91.7% 4-of-6 Success. A score of 5 reflects this partial self-measurement while acknowledging the human audit and external baselines give the central claim independent content.

Assumptions & free parameters 2 free parameters · 6 assumptions · 1 invented entities

The central method rests on the six-way intent abstraction being a valid and complete partition of local user actions, on the frozen verifier being a trustworthy labeler, and on LMSYS-USP being representative human-LLM dialogue. The calibration guarantee itself only requires standard group-relative RL math. No claim-relevant constants are fitted to the test data; the listed parameters are training choices without sensitivity analysis.

free parameters (2)

Composite reward weights alpha_sem, alpha_sty, alpha_ai = 0.50, 0.02, 0.05
Coefficients in Eq. 3 for the quality score q_i; chosen by hand with no reported sensitivity sweep, so the quality-intent balance in the final model depends on them.
Calibration margin m = 0.10
Margin in Eq. 4 that controls separation between compliant and violating candidates; the ordering guarantee holds for any positive margin but the magnitude affects policy gradient strength.

assumptions (6)

domain assumption One-to-many premise: the same profile and dialogue context support multiple plausible next-user intents.
This is the paper's motivating premise in the Introduction and Problem Formulation; if false, a single deterministic label would suffice and per-turn intent control would be unnecessary.
domain assumption The six-way canonical taxonomy is an operationally valid partition of local interaction intents in open-domain dialogue.
Tables 1 and 6 define labels with priority rules; Ambiguous handles leftovers. Human annotation audit supports labeling but does not prove completeness across all dialogue states.
domain assumption The frozen Qwen3.5-9B verifier g_ret provides labels accurate enough for annotation, reward, and evaluation.
Used for corpus construction (Appendix C), Eq. 4 compliance, and Tables 2 through 4. Human audit reports 87.88 percent accuracy and kappa 0.8223 on a 1315-turn sample, indicating strong but imperfect reliability.
standard math GRPO group standardization and clipped surrogate updates are valid RL optimizers.
Background from Shao et al. 2024 and Schulman et al. 2017; the ordering preservation in Appendix B.2 uses the monotonicity of affine standardization.
domain assumption Reference-free LLM judges and similarity encoders measure interaction quality and user authenticity.
Rubric prompts in Appendix F and cross-judge ranking consistency in E.5 support relative ordering, but the absolute quality scores are model-based judgments rather than direct measurements.
domain assumption LMSYS-Chat-1M with USP-style implicit profiles is an adequate distribution of human-LLM interactions.
Data source and preprocessing follow USP (Wang et al. 2025); no validation against other dialogue corpora is reported.

invented entities (1)

Per-turn interaction intent directive z_t (six-way canonical interface) independent evidence
purpose: Explicit control variable for the local interaction transition realized by the next user turn, separating action selection from surface realization.
Human expert labels (Appendix C.4) and the human audit on generated turns (Table 11) provide external judgment that requested intents are realized; if the taxonomy is real, raters can check it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation." pith.science (2026). https://pith.science/paper/77WZAHJA

@misc{pith2026260809420,
  author       = {Pith},
  title        = {Pith review of: Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/77WZAHJA}},
  note         = {Machine review of arXiv:2608.09420}
}

read the original abstract

User simulators are widely used as scalable environments for training and evaluating interactive assistants. Generating the next user turn is inherently one-to-many: the same profile and dialogue context may support multiple plausible continuations with different local interaction intents. A fluent response may therefore advance the dialogue through an inappropriate intent, such as acceptance rather than repair. Our key insight is that controllable user simulation should separate which local interaction intent the next user turn should realize from how that intent is expressed in language. We introduce UserIDA (User Intent-Directive Alignment), which exposes interaction intent as an explicit per-turn directive. UserIDA defines a six-way intent interface, learns directive-conditioned generation through supervised fine-tuning, and uses intent-calibrated policy optimization during group-based reinforcement learning. The reward preserves composite response quality while ensuring that intent-violating candidates rank below compliant alternatives in mixed groups. On LMSYS-USP, UserIDA achieves 86.6\% intent accuracy, outperforming the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, it realizes at least four of the six target intents in 91.7\% of evaluated dialogue states, compared with 22.9\% for the strongest external baseline. These results establish per-turn intent control as a complementary dimension to response fidelity in user simulation.

Figures

Figures reproduced from arXiv: 2608.09420 by the authors.

**Figure 2.** Overview of our proposed User Intent-Directive Alignment (UserIDA) framework. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Per-intent turn-level results. full UserIDA objective. Quality-only GRPO uses the same composite quality score qi and the same group-based policy objective as UserIDA, but removes intent calibration by setting ri = qi for every candidate. This comparison isolates Intent-Calibrated Relative Reward from the semantic, stylistic, and AI-likeness quality components. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Effect of supervised directive conditioning and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Within-context intervention case. Boldface marks [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Directive-specific context dependence [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Cross-judge groupwise evaluation. a 0–5 scale. Let ds and fs indicate goal drift and controller failure, respectively. For session s, we define GoalSuccesss = 1[gs ≥ 4 ∧ ds = 0 ∧ fs = 0] , (32) ConstraintSats = 1[is ≥ 4 ∧ ds = 0] , (33) SessionScores = 20 3 (rs + is + …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 15 canonical work pages

[501]

52Consider profile consistency, stable identity and speaking style, natural 53variation, and absence of assistant-role behavior or mechanical profile 54copying

role_authenticity 51Evaluate whether the same plausible user is maintained across checkpoints. 52Consider profile consistency, stable identity and speaking style, natural 53variation, and absence of assistant-role behavior or mechanical profile 54copying. Do not require explicit profile mentions when irrelevant. 55

work page
[562]

Remember that 60checkpoints use independent gold prefixes

interaction_performance 57Evaluate correct interpretation of each previous assistant turn, local 58directive realization and transitions, state tracking, efficient responses, 59appropriate supply, and avoiding premature acceptance. Remember that 60checkpoints use independent gold prefixes. 61

work page
[581]

directive_realization 59Does the candidate realize the requested target interaction directive, 60including its boundary relative to the other directives? 61

work page
[622]

dialogue_state_coherence 63Does the candidate correctly respond to the immediately previous assistant 64turn, preserve the active task state and prior constraints, and avoid 65contradiction, unsupported assumptions, or abrupt topic drift? 66

work page
[623]

scores": { 92

goal_progress 63Evaluate preservation and advancement of goals and constraints, supplying 64missing information, repairing mistakes, meaningful amendments, unresolved 65concerns, and avoiding unjustified abandonment or premature closure. 66 67================================================== 68HARD RULES 69================================================...

work page
[673]

information_appropriateness 68Does the candidate provide the information needed for this particular turn, 69with appropriate specificity and without material omissions, unsupported 70details, irrelevant elaboration, or unnecessary repetition? 71

work page
[724]

profile_consistency 73Is the candidate consistent with explicit profile facts, preferences, 74behavioral tendencies, and speaking style? 75Do not penalize a candidate merely because it does not explicitly mention 76profile details when they are irrelevant. 77

work page
[785]

83Length alone must never increase the score

user_authenticity_and_economy 79Does the candidate sound like a natural human user’s next message? 80Penalize assistant-style service language, role reversal, synthetic templates, 81control-label leakage, excessive formality, unnecessary structure, and 82verbosity that is inappropriate for the current turn. 83Length alone must never increase the score. 84...

work page

Show all 18 references

[1121]

Classify the candidate’s actually observed user action as predicted_directive
[1132]

Compare predicted_directive with the TARGET_DIRECTIVE label
[1143]

This cap is mandatory even when the candidate is fluent, 116coherent, or performs its different action well

If they differ, set wrong_user_action=true and set directive_realization to 1151.5 or lower. This cap is mandatory even when the candidate is fluent, 116coherent, or performs its different action well
[1174]

119Otherwise use at least 0.5 and explain the defect

Use 0.0 only when the candidate provides no usable evidence for that 118dimension (for example empty, unintelligible, or clearly wrong-role text). 119Otherwise use at least 0.5 and explain the defect
[1205]

121If there is any minor weakness, use 4.5 or lower

Use 5.0 only for a rare, fully convincing result with no visible defect. 121If there is any minor weakness, use 4.5 or lower
[1226]

predicted_directive

Verify that every score, flag, predicted_directive, and evidence sentence is 123mutually consistent before returning JSON. 124 125================================================== 126OUTPUT 127================================================== 128 129Return one valid JSON obj...
[2013]

Can you help me plan a warm vacation?

POMDP-Based Statistical Spoken Dialog Systems: A Review.Proceedings of the IEEE, 101(5): 1160–1179. Zhang, F.; Li, S.; Zhang, C.; Ma, Z.; Xu, J.; Gao, J.; Hao, J.; He,R.;Xu,J.;andLiu,H.2026. UserLM-R1:ModelingHu- manReasoninginUserLanguageModelswithMulti-Reward Reinforcement L...

2026
[2024]

Wang, K.; Li, X.; Yang, S.; Zhou, L.; Jiang, F.; and Li, H

DeepSeekMath: Pushing the Limits of Mathemati- cal Reasoning in Open Language Models.arXiv preprint arXiv:2402.03300. Wang, K.; Li, X.; Yang, S.; Zhou, L.; Jiang, F.; and Li, H

arXiv
[2025]

InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 21082–21107

KnowYouFirstandBeYouBetter:ModelingHuman- Like User Simulators via Implicit Profiles. InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 21082–21107. Association for Computational Linguistics. Wang, Y. S.; Zhang...

2026 arXiv
[2026]

Young, S.; Gasic, M.; Thomson, B.; and Williams, J

HumanLM: Simulating Users with State Alignment BeatsResponseImitation.arXivpreprintarXiv:2603.03303. Young, S.; Gasic, M.; Thomson, B.; and Williams, J. D

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [501]

52Consider profile consistency, stable identity and speaking style, natural 53variation, and absence of assistant-role behavior or mechanical profile 54copying

role_authenticity 51Evaluate whether the same plausible user is maintained across checkpoints. 52Consider profile consistency, stable identity and speaking style, natural 53variation, and absence of assistant-role behavior or mechanical profile 54copying. Do not require explicit profile mentions when irrelevant. 55

work page

[2] [562]

Remember that 60checkpoints use independent gold prefixes

interaction_performance 57Evaluate correct interpretation of each previous assistant turn, local 58directive realization and transitions, state tracking, efficient responses, 59appropriate supply, and avoiding premature acceptance. Remember that 60checkpoints use independent gold prefixes. 61

work page

[3] [581]

directive_realization 59Does the candidate realize the requested target interaction directive, 60including its boundary relative to the other directives? 61

work page

[4] [622]

dialogue_state_coherence 63Does the candidate correctly respond to the immediately previous assistant 64turn, preserve the active task state and prior constraints, and avoid 65contradiction, unsupported assumptions, or abrupt topic drift? 66

work page

[5] [623]

scores": { 92

goal_progress 63Evaluate preservation and advancement of goals and constraints, supplying 64missing information, repairing mistakes, meaningful amendments, unresolved 65concerns, and avoiding unjustified abandonment or premature closure. 66 67================================================== 68HARD RULES 69================================================...

work page

[6] [673]

information_appropriateness 68Does the candidate provide the information needed for this particular turn, 69with appropriate specificity and without material omissions, unsupported 70details, irrelevant elaboration, or unnecessary repetition? 71

work page

[7] [724]

profile_consistency 73Is the candidate consistent with explicit profile facts, preferences, 74behavioral tendencies, and speaking style? 75Do not penalize a candidate merely because it does not explicitly mention 76profile details when they are irrelevant. 77

work page

[8] [785]

83Length alone must never increase the score

user_authenticity_and_economy 79Does the candidate sound like a natural human user’s next message? 80Penalize assistant-style service language, role reversal, synthetic templates, 81control-label leakage, excessive formality, unnecessary structure, and 82verbosity that is inappropriate for the current turn. 83Length alone must never increase the score. 84...

work page