{"id":"8b0840bf-ab12-4ced-b57c-f98104224c7a","arxiv_id":"2608.09420","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"UserIDA separates per-turn interaction intent from surface wording in LLM user simulators, reaching 86.6 percent intent accuracy and 91.7 percent four-of-six within-context control.","lead":"This paper presents UserIDA, a user simulator that follows an explicit per-turn instruction about which conversational move to make, such as repairing a misunderstanding or accepting an answer, before writing the actual user words. It reports large gains in intent accuracy over previous user simulators on LMSYS-USP and shows that controlling intent is distinct from producing fluent text.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closed-loop verifier threatens the within-context headline: Table 4's 91.7% 4-of-6 success is scored by the same frozen verifier used for training labels and RL rewards, and the C.5 human audit does not cover that protocol.","rationale":"I read the paper in good faith and credit its genuine strengths: the intent-calibrated reward ordering guarantee is proved correctly in Appendix B; the human audits in Appendix C.5 and Table 11 partially break the closed evaluation loop; the cross-judge evaluation in E.5 shows consistent system ordering; and the context-dependence probe (Figure 6) provides independent evidence that the six intents are not surface-only categories. The reader's weakest assumption is verifier reliability, and I agree that is the central risk. However, I want to sharpen the concern to the precise place where the human audit does not reach. The turn-level headline (86.6%) is mitigated by the C.5 human labels on generated turns, which show the same ordering and even slightly higher human-judged accuracy for UserIDA on the natural distribution. The within-context intervention (Table 4), which supports the strongest claim of compositional controllability (91.7% 4-of-6 Success), has no human validation at all. Because that protocol is artificial and surface cues are more predictive of the verifier's labels, the verifier could be inflating UserIDA's apparent controllability there. This is not an accusation of gaming; it is a concrete, testable gap. If the proposed human audit on the intervention set confirms the verifier-based numbers, the central claim is robust and the paper is a solid contribution. If not, the headline should be revised. Since the reader already recommended CONDITIONAL and my read does not change that verdict, I set verdict_should_be to UNCHANGED and note partial agreement because my concern is a narrower, more specific version of the reader's broader verifier-reliability concern.","tokens_in":24565,"tokens_out":7950,"duration_ms":72595,"concrete_test":"Run the Appendix C.5 expert labeling protocol on the Table 4 within-context intervention outputs: for all 48 contexts x 6 directives, have three trained experts independently label the UserIDA and USP generated turns (576 turns per system, or a pre-registered subsample) with the canonical taxonomy, then compute human-judged 4-of-6 Success, per-directive accuracy, and Cohen's kappa against g_ret. Compare with the verifier-based values of 91.67% and 2.08%. The concern is settled if human 4-of-6 Success for UserIDA remains above roughly 85% with non-overlapping intervals versus USP; if it drops below 70%, Table 4's headline numbers should be revised and the controllability claim weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The same frozen verifier g_ret supplies corpus labels (Appendix C), the RL compliance signal (Eq. 4), and the intent-accuracy labels for all main tables, including Table 4's 4-of-6 Success. Appendix B's ordering guarantee is correctness relative to g_ret's labels, not to human intent judgments; Table 10 shows systematic boundary errors such as Initiate->Amend (10.32%) and Amend->Initiate (7.81%). The Appendix C.5 human audit on generated turns is genuinely reassuring for the turn-level protocol: human balanced accuracy for UserIDA is 90.83% vs 84.17% for UserIDA w/o RL, matching the automatic ordering. But that audit samples the natural test distribution; it does not evaluate the within-context intervention protocol of Table 4, where every state is pre-selected for six-way feasibility and each directive is requested once. In this artificial setting, surface cues are more predictive: imperative frames for SetRegister, long payloads for Supply, short acknowledgments for GroundAccept. Consequently, g_ret could be substantially easier to satisfy than a human judge. The headline claim's most striking number, 91.7% 4-of-6 Success (vs 2.08% for USP), therefore rests on unvalidated automatic labels. If human labels on the intervention set reveal lower 4-of-6 success, the central claim that UserIDA reliably realizes per-turn intent control would be overstated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UserIDA, a user-simulation framework that separates the per-turn interaction intent of a simulated user from its surface realization. It defines a six-way canonical intent interface (Initiate, Amend, Supply, Repair, SetRegister, GroundAccept), trains a directive-conditioned generator with supervised fine-tuning, and applies an intent-calibrated group-relative policy optimization whose reward explicitly ranks intent-compliant candidates above intent-violating ones. Experiments on LMSYS-USP report large gains in turn-level intent accuracy, multi-turn trajectory success, and within-context directive interventions, alongside human audits on generated turns and a small recursive closed-loop pilot.","tokens_in":24861,"tokens_out":4449,"duration_ms":41528,"significance":"If the results hold, the paper makes a useful and timely contribution: it identifies a missing control variable in next-user simulation, provides a concrete six-way interface that is surface-underspecified, and demonstrates that explicit intent calibration improves directive adherence without degrading semantic or stylistic fidelity. The Appendix B ordering proof is a clean formal property of the calibration, the ablations isolate the effect of intent calibration, and the human audit in Appendix C.5 is a genuine external check on the turn-level ordering. The paper also releases code and includes a recursive closed-loop pilot, which strengthens reproducibility. The main caveat is that the headline automatic metrics, especially the Table 4 within-context intervention numbers, are produced by the same frozen verifier used during annotation and reward computation; the human audit does not cover that protocol. This is fixable but must be addressed before the central claim can be fully accepted.","major_comments":[{"comment":"The same frozen verifier g_ret supplies corpus labels (Appendix C), the RL compliance signal (Eq. 4), and the intent-accuracy labels in Tables 2–4, including the 91.7% 4-of-6 Success in Table 4. Appendix B's ordering guarantee is correctness relative to g_ret's labels, not to human intent judgments, and Table 10 shows substantial boundary errors (e.g., 10.32% Amend→Initiate). The human audit in Appendix C.5 covers generated turns under the natural test distribution and does not evaluate the within-context intervention protocol of Table 4, where each state is pre-selected for six-way feasibility and surface cues may make the verifier easier to satisfy. Please add a human evaluation on the Table 4 intervention set (or an independently trained verifier) and report the resulting 4-of-6 Success; without it, the headline number may measure verifier consistency rather than human-judged intent control.","section":"§4.2, Eq. (4); Appendix C.5; Appendix E.3; Table 4"},{"comment":"Main tables report point estimates without confidence intervals or significance tests; the only interval is the paired-bootstrap improvement for Table 4 reported in the text. Because the margins are large this may be a formality, but for claims that UserIDA outperforms baselines by 24.3 points on a verifier-derived metric, report bootstrap confidence intervals (or per-seed variance) for the three main tables so readers can assess the stability of the stated ordering.","section":"Tables 2–4"},{"comment":"The paper states that all-violating groups provide no relative intent preference and that the all-violating-group rate is recorded 'as a diagnostic', but this rate is never reported. If a substantial fraction of groups, particularly for rare intents such as Supply (1.9% of training turns), contains no compliant candidate, the calibration may leave rare-intent performance unchanged; please report the all-violating-group rate overall and per intent, and discuss its effect on the RL result.","section":"§4.2, Eqs. (4)–(5)"}],"minor_comments":[{"comment":"The per-intent figure uses abbreviated labels 'ACCEPT' and 'REGISTER' while the taxonomy and tables use 'GroundAccept' and 'SetRegister'; please unify the labels to avoid confusion.","section":"Figure 3"},{"comment":"The abstract's '22.9% for the strongest external baseline' is somewhat ambiguous because Table 4 reports different external baselines; please name the exact system (LLaMA-3-8B Base with Directive) in the abstract or table caption.","section":"Abstract and Table 4"},{"comment":"The construction of the within-context intervention set says it is 'balanced across six domains, four dialogue-depth buckets, and two levels of profile salience' but does not report how many states survive the affordance screen; please report the number of retained states and the screen's inter-annotator agreement.","section":"Appendix E.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is well motivated and the training recipe is interesting. My main concern is that the most striking numbers, especially Table 4, are self-measured by the verifier used in training. I would want to see a human evaluation on the intervention protocol or an equivalent external validation before recommending acceptance. The requested additions are within the scope of a revision and do not require changing the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: UserIDA gives user simulation something it didn't have—an explicit per-turn intent knob you can set at generation time—and the paper shows, with mostly honest evidence, that you can turn that knob reliably. The catch: the headline within-context number is measured with the same verifier that trained the model, so its exact size is questionable even though the effect is real.\n\nWhat's new: prior LLM simulators condition on profiles and global goals but leave the local action ambiguous. UserIDA defines a six-way intent interface (Initiate, Amend, Supply, Repair, SetRegister, GroundAccept), trains with directive-conditioned SFT, then applies a clever calibration: instead of a fixed intent bonus, it computes the minimum group-specific reward shift that puts every violating candidate below every compliant one. Appendix B's ordering guarantee is simple and correct. The quality-only GRPO ablation cleanly demonstrates the motivating problem—response quality can improve while intent adherence drops—and the full method recovers both. They also run a human audit on generated turns (C.5) that confirms the system ordering UserIDA > UserIDA w/o RL > USP on the natural distribution, and they report verifier-human agreement (87.88% acc, kappa 0.82) with honest boundary confusion stats. Code is public.\n\nWhere it's soft: the same frozen Qwen3.5-9B verifier labels the training data, supplies the RL reward, and scores all automatic metrics. The human audit does not cover the within-context intervention protocol in Table 4, the one producing the flashiest number (91.7% 4-of-6). In that protocol the states are pre-screened for six-way feasibility and every directive is requested once, so surface cues like imperative frames for SetRegister or long payloads for Supply could make the verifier easier to satisfy than a human judge. The boundary confusions in Table 10 (Initiate-Amend, etc.) show the verifier isn't perfect. So the exact size of the within-context success is uncertain. That's a fixable issue: collect human labels on that exact intervention set and report CIs for the main tables. It doesn't sink the paper—the qualitative case in Figure 5 is convincing, the turn-level human audit is reassuring—but it means the strongest claim is not yet fully independent of the measuring instrument.\n\nBottom line: per-turn intent control is a genuinely useful contribution to user simulation and synthetic assistant evaluation. It deserves a serious referee and could be a solid paper after adding the missing human labels. I'd read it carefully and would cite it once the intervention numbers are externally verified.","headline":"Per-turn intent control is a real contribution, but the flashiest within-context number is self-measured by the same verifier that trained the model—size uncertain, effect real.","tokens_in":25409,"tokens_out":3162,"would_cite":true,"duration_ms":28646,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"User simulators can be steered by a six-way intent directive, and the authors' model reaches 86.6% turn-level intent accuracy on real human–assistant conversations while preserving response fidelity.","keywords":["user simulation","interaction intent","intent-conditioned generation","intent-calibrated policy optimization","controllable dialogue evaluation","LMSYS-Chat-1M","group relative policy optimization","LLM user simulators"],"falsifier":"Take a balanced sample of dialogue states and have independent human annotators label the generated user turns for the six intents, focusing on the confusable pairs Initiate/Amend, Amend/Repair, and Repair/SetRegister. If human-judged intent accuracy falls substantially below the verifier-based 86.6% figure for the same generated turns, the central claim of reliable per-turn intent control collapses; if an independent audit reproduces the small human-versus-verifier gap reported in the paper's own Appendix C.5, the claim stands.","tokens_in":24351,"feed_emoji":"💬","tokens_out":8057,"duration_ms":67156,"temperature":0.7,"pith_summary":"This paper argues that the missing control variable in user simulation is the per-turn interaction intent: what the next user turn is supposed to do in the conversation, as opposed to how it is phrased. Because the same dialogue context supports many valid continuations, a simulator that only imitates response style can produce a fluent turn that advances the dialogue in the wrong direction, for example accepting an answer that should be repaired. To fix this, the authors introduce UserIDA, which conditions generation on an explicit six-way intent directive, trains the simulator with supervised fine-tuning, and then applies intent-calibrated policy optimization that ensures intent-violating candidates never outrank compliant ones. On the LMSYS-USP benchmark the resulting simulator reaches 86.6% turn-level intent accuracy, outperforming the strongest dedicated baseline by 24.3 percentage points while preserving or improving semantic and stylistic fidelity. If the claim holds, interaction intent is a complementary, controllable dimension of user simulation rather than something that can be left implicit in response imitation.","feed_headline":"Simulated users now follow intent commands 86.6% of the time","feed_subtitle":"A new simulator separates what a user turn does from how it is worded, so fake users can be steered through scripted dialogues.","key_machinery":"The load-bearing object is the intent directive, a six-way categorical interface that separates what the next user turn should do from how it is expressed. It is implemented as an additional control input in the prompt, and it is verified by a frozen retrospective classifier that assigns one of the six canonical labels to any generated or observed user turn. The other central mechanism is the intent-calibrated relative reward: for each group of $K$ sampled candidates the method computes a group-specific shift $\\Delta_t = \\max(0, \\max_{i:b_i=0} q_i - \\min_{j:b_j=1} q_j + m)$, subtracts it from every violating candidate's quality score, and proves that the resulting ordering survives group-level standardization. A fixed intent bonus would fail because quality-score spread can overwhelm any constant; calibration instead computes the minimum shift needed to separate compliant from violating sets while retaining quality discrimination among compliant candidates.","core_discovery":"The central claim is that next-user behavior should be modeled as a conditional realization of a chosen local interaction intent, not as a marginal imitation of observed user turns. UserIDA instantiates this by factorizing the user policy as $\\pi(u_t \\mid c_t, p) = \\sum_{z \\in \\mathcal{Z}} P(z \\mid c_t, p) \\, \\pi(u_t \\mid c_t, p, z)$, exposing the categorical intent $z$ as an explicit per-turn directive. The six-way taxonomy is deliberately surface-underspecified: it describes what the turn does to the dialogue state (opens a thread, amends an open task, supplies material, repairs a mismatch, sets a register, or grounds, accepts, and closes), without constraining wording. Training proceeds in two stages: intent-conditioned supervised fine-tuning teaches the model to generate distinct valid continuations from the same context under different directives, and intent-calibrated policy optimization fixes the quality–intent mismatch by shifting all violating candidates' rewards below all compliant candidates within each sampled group. The paper's evidence is that this produces 86.6% turn-level intent accuracy, 58% all-turn trajectory success (up from 13% for the strongest baseline), and 91.7% within-context four-of-six directive success, with quality scores at least as high as before.","pith_inferences":["The six-way intent interface could double as a compact action space for training assistant policies: an assistant could first choose which local transition to perform and then select words, mirroring the factorization this paper applies to the user.","The taxonomy was built on open-domain English chat data; a transfer experiment to task-oriented or customer-support dialogues would test whether the same six labels stay operational when Supply and Repair dominate.","The calibration guarantee is an ordering guarantee within each sampled group, not a guarantee about absolute update size; future variants could add per-intent margins or absolute floors to control how strongly violating candidates are pushed down.","Swapping the frozen verifier for an independent classifier and rerunning only the turn-level test would quantify how much of the reported 86.6% is verifier-specific rather than human-visible intent control."],"forward_implications":["Scripted directive sequences become a practical way to generate reproducible counterfactual dialogues, since the simulator can be told to repair, amend, or accept at each step instead of leaving the next action to chance.","Trainers of interactive assistants can now stress-test error recovery explicitly: forcing a simulated user to issue Repair turns reveals whether an assistant detects and corrects its own mismatches, which imitation-based simulators often mask.","Because quality-only reinforcement learning in the ablation reduces intent accuracy, future simulator training objectives should treat intent adherence as a separate reward axis rather than expecting it to emerge from response fidelity.","The per-intent results show that the interface helps minority intents as well as dominant ones; macro-F1 should be reported alongside accuracy for any work building on this interface.","In a diagnostic closed-loop pilot, intent-controlled user turns also improved goal success and constraint satisfaction, suggesting the gains survive when simulated turns are actually fed back into an assistant conversation."],"supporting_citations":[{"why":"Supplies the LMSYS-Chat-1M corpus from which every training, validation, and test turn in this paper is expanded.","marker":"Zheng et al. 2024"},{"why":"Defines the USP simulator and the LMSYS-USP preprocessing and implicit user profiles; it is the strongest dedicated baseline that UserIDA must beat.","marker":"Wang et al. 2025"},{"why":"Provides the UserLM dedicated-simulator baseline and the protocol for expanding conversations into next-user-turn instances.","marker":"Naous et al. 2026"},{"why":"Introduces the group-relative policy optimization objective into which UserIDA injects its calibrated rewards.","marker":"Shao et al. 2024"},{"why":"Supplies the frozen Qwen3.5-9B verifier that labels the corpus, checks reinforcement-learning compliance, and scores intent accuracy in the main results.","marker":"Qwen Team 2026"},{"why":"Supplies the mixed-reward calibration idea that informs the compliant-over-violating ordering constraint.","marker":"Liao et al. 2025"},{"why":"Defines SimCSE, the semantic similarity measure used inside the composite quality reward and evaluation.","marker":"Gao, Yao, and Chen 2021"}],"fun_headline_variants":["User simulators: intent control beats response imitation","Six-way intent steering lifts user sim accuracy to 86.6%","Separate intent from wording, then direct fake users","UserIDA: per-turn intent control for 86.6% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand on a frozen classifier's intent labels being trustworthy: the same model labels the training data, the reinforcement-learning rewards, and the evaluation metric, so any systematic confusion between nearby intents (for example Amend versus Initiate) could inflate the reported accuracy beyond what human judges would confirm, and the human audit in Appendix C.5 mitigates but does not eliminate this risk.","fun_headline_variants_meta":{"raw":{"variants":["User simulators: intent control beats response imitation","Six-way intent steering lifts user sim accuracy to 86.6%","Separate intent from wording, then direct fake users","UserIDA: per-turn intent control for 86.6% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1497,"prompt_tokens":1075,"completion_tokens":422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":691,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":691,"tokens_out":422,"duration_ms":4433,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:24:09.767591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a balanced sample of dialogue states and have independent human annotators label the generated user turns for the six intents, focusing on the confusable pairs Initiate/Amend, Amend/Repair, and Repair/SetRegister. If human-judged intent accuracy falls substantially below the verifier-based 86.6% figure for the same generated turns, the central claim of reliable per-turn intent control collapses; if an independent audit reproduces the small human-versus-verifier gap reported in the paper's own Appendix C.5, the claim stands.","supporting_citations":[],"review_version":2}