Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Reverse Preference Optimization for Complex Instruction Following

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By reversing the constraints a response fails, RPO builds preference pairs whose chosen response is perfect and strictly dominates the rejected one, yielding multi-constraint instruction-following gains over DPO and, at 70B scale, results…

desk verdict RPO is a genuinely clever data-construction trick with real gains, but the paper oversells the noise-free guarantee that its own audit undercuts. read the letter →

arxiv 2505.22172 v1 pith:2VX4IERK submitted 2025-05-28 cs.CL

Xiang Huang , Ting-En Lin , Feiteng Fang , Yuchuan Wu , Hangyu Li , Yuzhong Qu , Fei Huang , Yongbin Li This is my paper · ORCID
classification cs.CL
keywords instructionfollowingpreferenceoptimizationDPOconstraintreversalmulti-turndialoguesystempromptsmulti-constraintalignmentsyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that noisy preference pairs are the main obstacle to aligning language models with complex, multi-constraint instructions, and that this noise can be removed without expensive sampling of perfect responses. Its method, Reverse Preference Optimization (RPO), takes any response, finds the constraints it violates, and rewrites those constraints into their opposites, so the same response becomes a flawless answer to the new instruction. Such reversed pairs have a chosen response that is perfect and strictly dominates the rejected one, which the paper says eliminates misleading optimization signals. On SysBench and Multi-IF, RPO beats the DPO baseline by about 4.6 and 2.5 average points on Llama-3.1 8B, and a 70B RPO model surpasses GPT-4o. A reader should care because the method makes preference-data collection cheaper and more reliable for multi-turn instruction following.

What carries the argument

The central object is the reversed instruction $x_{S_i}$, built by negating the set of constraints that a response $y_i$ fails. The key identity is that if $S_{y_i} \neq S_{y_j}$, then for $x_{S_i}$, $y_i$ satisfies every constraint and strictly dominates $y_j$. The training loss is the DPO objective with an adaptive margin $\gamma_g$ that depends on the number $g$ of constraint-level differences between the two responses, sharpening the preference signal. Reversal is performed by an LLM prompt that asks for the exact logical complement, covering the entire opposite domain rather than a partial negation. The machinery also includes a fine-grained LLM evaluator that labels each constraint True or False and a role-driven self-play data generator that produces the SysBank corpus of 30K system prompts.

What would settle it

Take the constraint 'at least three keywords must appear' and its LLM-reversed 'at most three keywords must appear'; a response with exactly three keywords satisfies both, disproving perfect complementarity for that case. A systematic version is to sample reversed constraint pairs from the paper's training data and check whether any response can satisfy both the original and the reversed constraint; if the overlap rate is substantially above zero, RPO's central 'chosen is perfect and rejected is strictly worse' property fails on those pairs.

Watch

Extended reading notes

Core claim

The central claim is that when two responses differ on which constraints they satisfy, the original instruction makes at least one of them imperfect, so any preference pair built from it carries noise: the chosen response may violate constraints that the rejected response satisfies, teaching the model to ignore those constraints. RPO reverses every constraint the would-be chosen response fails, constructing a new instruction for which that response is perfect and, by construction, no worse than the other response on any constraint. The paper asserts that this eliminates noise and enlarges the chosen-rejected gap, making the DPO-style gradient direction clearer and more robust to evaluation errors. Experiments on two multi-turn benchmarks support the claim, including scaling to 70B parameters, where the RPO model surpasses GPT-4o on most metrics.

Load-bearing premise

The load-bearing premise is that reversing a constraint produces its exact logical complement: every response that violated the original must satisfy the reversed one, and the reversed one must cover the entire opposite domain with no overlap. The paper's own manual audit finds this holds perfectly for only 85.2% of reversals, so roughly one in ten preference pairs does not have a perfect, strictly dominating chosen response.

Editorial extensions

If this is right

  • Preference pairs can be built from any two responses that differ on at least one constraint, so RPO removes the need to sample a perfect response; the paper reports that reversal yields perfect pairs 100% of the time versus 59% for direct sampling.
  • Because the chosen response is perfect and dominates the rejected one, the training signal no longer teaches the model to ignore constraints that the chosen response happens to violate.
  • Larger chosen-rejected gaps improve performance: models trained on preference pairs with a gap of at least three constraints outperform those trained on pairs with a gap of one on both SysBench and Multi-IF.
  • RPO scales across model sizes (7B to 72B) and model families (Llama-3.1 and Qwen-2.5), with the 70B RPO model surpassing GPT-4o on most SysBench and Multi-IF metrics.
  • General capabilities on AlignBench, GSM8K, and HumanEval do not degrade under RPO training, and at 70B scale RPO can even improve some of these benchmarks relative to the base Instruct model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the 8.4% bad reversals and 2.4% non-reversible constraints found in its own audit directly weaken the claimed noise-free property; adding an automatic overlap check, which rejects reversed constraints that are also satisfied by the original response, is a testable extension that should further close that gap.
  • The failure cases in Appendix D.3, such as reversing 'at least three keywords' into 'at most three keywords', suggest that numeric and threshold constraints need rule-based complement handling before LLM reversal, since a response with exactly three keywords satisfies both the original and the reversed constraint.
  • Because RPO makes any sampled response perfect by construction, it could combine naturally with online DPO or iterative self-refinement loops where a weak model produces poor responses; this may reduce the need for strong teacher models in preference-data collection.
  • The SysBank corpus of 30K system prompts drawn from real GPTs, independent of the RPO training objective, is a reusable resource for future work on system-prompt following and multi-turn agent alignment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Reverse Preference Optimization (RPO), a modification of DPO for multi-constraint instruction following. For each response that fails some constraints, RPO rewrites the instruction by reversing the failed constraints so that the response becomes a "perfect" chosen response, while another response that differs in at least one constraint adherence serves as the rejected response. The objective in Eq. (4) adds an adaptive margin proportional to the number of constraint differences. Training data are generated through a role-driven self-play pipeline that yields a 30K system-prompt corpus (SysBank). On SysBench and Multi-IF, RPO improves over DPO across Llama-3.1 and Qwen-2.5 at the 8B/7B and 70B/72B scales, and the 70B RPO model surpasses GPT-4o on several metrics. The paper also reports sample-efficiency, data-volume, and general-ability analyses.

Significance. The central idea—turning a failing response into a perfect one by reversing the constraints it violates—is elegant and practically attractive, since it avoids expensive sampling of perfect responses. If validated, RPO would be a simple, scalable drop-in modification of DPO for multi-constraint alignment. The paper is commendably transparent: it includes an honest Limitations section, a detailed error analysis in Appendix D.3, sample-efficiency comparisons, and general-ability checks. The SysBank resource and the self-play construction recipe are potentially useful to the community. However, the paper's headline theoretical claim is not fully supported by its own audit, and the empirical evidence lacks uncertainty quantification; these gaps need to be addressed.

major comments (4)
  1. [§4.3, Eq. (4), Appendix D.3, Limitations] The "noise-free" property is load-bearing. Section 4.3 states that after reversal the chosen response "dominates the rejected ones and performs no worse than the rejected response in any aspect, thus eliminating noise," and Eq. (4) is derived under this assumption. Appendix D.3 (Table 7) reports that only 85.2% of reversals are perfect, with 4.0% slight-error, 8.4% bad, and 2.4% non-reversible. The bad case in Table 10 ("Replies must end with an exclamation mark" reversed to "Replies must end with a period") is a direct counterexample: a response ending in a question mark violates both the original and the reversed constraint, so the "chosen" response is not perfect and does not dominate. The Limitations section concedes that bad cases "lead to error propagation." Since roughly one in ten training pairs does not satisfy the premise of Eq. (4), the paper needs either to filter such reversals and re-run the experiments, or to quantify how often bad reversals occur in the actual training set and show that the results are insensitive to them. Without this, the empirical gains cannot be attributed to the noise-free mechanism rather than to the enlarged margin or data augmentation.
  2. [§6.5, Table 1] The main results are all single runs without seeds, variance, or significance testing. The headline improvements over DPO are 2.5 points on Multi-IF average and 2.5-6.6 points on SysBench components for Llama-3.1-8B; for Multi-IF Step 1 the Llama-8B RPO improvement over DPO is about 2.9 points, which is small relative to run-to-run variability in fine-tuning. The paper should report means and standard deviations over at least three seeds, or otherwise justify that the differences are outside noise. This is needed to support the wording "significant improvements" in the abstract and Section 6.5.
  3. [§6.1, §6.6, Table 6] The margin hyperparameter γ is tuned directly on the evaluation benchmarks. Table 6 sweeps γ on SysBench and Multi-IF, and the main results in Table 1 use γ=0.05, the value selected on those same benchmarks. Since the test sets are used for model selection, the reported improvements may be optimistic. The paper should either fix γ before seeing the benchmarks, tune on a held-out validation subset, or report the full sensitivity and show that the conclusions are robust across the γ range.
  4. [§6.6.3, Table 4] Table 4 reports that the Reverse method achieves a 1.00 Perfect rate, but this is true by construction only if every reversal is a perfect logical complement. Appendix D.3 shows this is not the case, so the 1.00 figure overstates the sample-efficiency advantage. The Perfect column should be recomputed using the reversal-quality audit, or the table should be restricted to the 85.2% perfect reversals.
minor comments (6)
  1. [§6.5] The sentence "Across the three metrics of SysBench, RPO outperforms the DPO baseline by 2.9, 2.6, and 2.1 points, respectively" appears in the paragraph about Multi-IF and is inconsistent with the preceding sentence; clarify whether these numbers refer to Multi-IF steps or to SysBench.
  2. [§6.6.2] The text says "As shown in Table 1, across different training data volumes" but the relevant results are in Table 3; the cross-reference should be corrected.
  3. [Eq. (4)] There is a missing space in "γ, βare hyper-parameters"; also, g should be defined more precisely before first use, although the surrounding paragraph does explain it.
  4. [§5.1] Typo: "enrichh" should be "enrich".
  5. [Appendix D.3] The audit is based on 500 constraints; report the number of annotators and inter-annotator agreement, and clarify whether the 85.2% figure was computed at the constraint level or at the preference-pair level.
  6. [Figure 2] The figure caption contains raw JSON-like text that is not readable; consider moving this to an appendix or formatting it as a proper example.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: RPO's noise-free property is a definitional feature of its data construction, not a prediction, and the reported gains come from external benchmarks.

full rationale

The paper's central claim is not a derived prediction reducible to its inputs. Section 4.1 defines xSi as the instruction obtained by reversing the constraints that yi fails, and then states 'For xSi, yi is a perfect response and is strictly dominates yj.' That statement is true by construction: a response that failed an original constraint satisfies the reversed constraint, and where the rejected response originally satisfied that constraint, it now fails the complement. This is the method's definition rather than a hidden equivalence, so it is not circularity in the sense of predicting an output from its own fitted input. RPO is evaluated on external benchmarks (SysBench and Multi-IF, plus GSM8K/HumanEval/AlignBench), and the training corpus SysBank is not used as the evaluation metric. The only benchmark-dependent design choice is the hyperparameter gamma, whose sweep is reported in Table 6 on the same two benchmarks; while this means the reported gains are not a fully out-of-sample forecast of the method, tuning a margin is not a derivation that reduces the central result to its inputs. The paper's own audit (Table 7, Appendix D.3) finds only 85.2% perfect reversals, 4.0% slight-error, 8.4% bad, and 2.4% non-reversible, and the Limitations section concedes 'bad cases, leading to error propagation.' That contradicts the complete noise-free guarantee, but it is a correctness/robustness limitation, not a circularity. The self-citations (IOPO and the DPO survey) appear only as related-work contrasts and are not load-bearing. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on DPO math, the quality of LLM-generated reversals, the quality of LLM-as-judge constraint checking, and the assumption that training on reversed constraints transfers to original prompts. The second and third are empirical assumptions that are only partially validated by the paper's own error analysis.

free parameters (3)
  • gamma (margin scale) = 0.05
    Scales the adaptive margin gamma*g in Eq. (4). Chosen from the sweep in Table 6, which reports performance directly on SysBench and Multi-IF.
  • beta (DPO temperature) = 0.1
    Inverse temperature in the DPO-style loss; reported in Section 6.1 with no ablation.
  • LoRA rank and alpha = not reported
    All preference training runs use LoRA and DeepSpeed (Section 6.1), but the rank and alpha are not given, so the effective parameter budget is unspecified.
assumptions (4)
  • standard math Bradley-Terry model and DPO reparameterization hold for the modified instruction xSi
    Eq. (3) and Eq. (4) rely on the standard BT/DPO identity; no new math is introduced.
  • ad hoc to paper Reversing a failed constraint yields a semantically exact logical complement
    Section 4.3 and Appendix E. The paper's own D.3 audit shows this holds perfectly in only 85.2 percent of sampled constraints, with 8.4 percent bad and 2.4 percent non-reversible.
  • domain assumption LLM-based fine-grained evaluation (GPT-4o-mini) correctly determines adherence for each constraint
    Appendix I; training pairs depend on this judge, and errors propagate into the preference data.
  • domain assumption Training on reversed-instruction pairs transfers to original instructions at inference
    Models are trained on xSi but evaluated on original x; the paper does not isolate or measure this transfer effect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reverse Preference Optimization for Complex Instruction Following." pith.science (2026). https://pith.science/paper/2VX4IERK

@misc{pith2026250522172,
  author       = {Pith},
  title        = {Pith review of: Reverse Preference Optimization for Complex Instruction Following},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2VX4IERK}},
  note         = {Machine review of arXiv:2505.22172}
}
read the original abstract

Instruction following (IF) is a critical capability for large language models (LLMs). However, handling complex instructions with multiple constraints remains challenging. Previous methods typically select preference pairs based on the number of constraints they satisfy, introducing noise where chosen examples may fail to follow some constraints and rejected examples may excel in certain respects over the chosen ones. To address the challenge of aligning with multiple preferences, we propose a simple yet effective method called Reverse Preference Optimization (RPO). It mitigates noise in preference pairs by dynamically reversing the constraints within the instruction to ensure the chosen response is perfect, alleviating the burden of extensive sampling and filtering to collect perfect responses. Besides, reversal also enlarges the gap between chosen and rejected responses, thereby clarifying the optimization direction and making it more robust to noise. We evaluate RPO on two multi-turn IF benchmarks, Sysbench and Multi-IF, demonstrating average improvements over the DPO baseline of 4.6 and 2.5 points (on Llama-3.1 8B), respectively. Moreover, RPO scales effectively across model sizes (8B to 70B parameters), with the 70B RPO model surpassing GPT-4o.

Figures

Figures reproduced from arXiv: 2505.22172 by the authors.

Figure 1
Figure 1. Noise in a multi-preference scenario. Any [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of RPO and data construction. We first model various users and systems and then engage them [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Variation in reward differences between cho [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 36 canonical work pages

  1. [1]

    at least three profession-related keywords must appear

    For constraints involving numbers, for example, if the original constraint is "at least three profession-related keywords must appear", the opposite should be "at most three times" rather than "only once" or "at most twice", as these don’t cover all situations opposite to the original constraint

    work page

  2. [2]

    forbidden to mention Van Gogh

    Your reversed constraints should be explicit constraints. For example, if the original constraint is "forbidden to mention Van Gogh", while its opposite is "can mention Van Gogh", this isn’t explicit enough. I can’t measure whether a response satisfies this reversed constraint because it’s ambiguous - it can either be satisfied or not. An explicit reverse...

    work page

  3. [3]

    Preprint, arXiv:2305.18290

    Direct preference optimization: Your lan- guage model is secretly a reward model. Preprint, arXiv:2305.18290. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao- hua Dong, Ran Lin, and Ruohui Huang. 2024. Conifer: Improving complex constrained instruction- following ability of large language models. arxiv preprint arXiv:2404.02823. Zhengwei Tao, Ting-En L...

    arXiv 2024

  4. [4]

    character-driven

    Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (V olume 3: System Demonstra- tions), Bangkok, Thailand. Association for Computa- tional Linguistics. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zh...

    arXiv 2023

  5. [5]

    Logic (in some cases, please combine with actual scenarios, use cautiously): - For example, for problems involving calculations, the answer must be increased by one, etc. -

    work page

  6. [6]

    I don’t know,

    Behavior/Actions: - ‘Refusal‘: For example, when the user mentions certain topics such as politics, topics unrelated to skills, robot privacy, offensive questions, etc., refuse to answer or provide suggestions, or say "I don’t know," or steer the conversation back, etc. - ‘Invitation‘: For example, when the user shows interest in certain things/mentions c...

    work page

  7. [7]

    Hello," a catchphrase, a specific line of poetry, self-introduction. - ‘Ending content‘ (used in a minority of cases, less than 15%): For example, ending with

    Content: - ‘What content should be included in responses‘: For example, keywords, emojis, punctuation, numbers, content from input, quotes from poems, idioms, jokes, examples, etc. - When certain content is mentioned, what must or must not be mentioned. - ‘Number of times specific content should be included‘: For example, at least three times. - ‘Content ...

    work page

  8. [8]

    Format: - ‘Overall format‘: For example, returning in JSON format, markdown format, expressing in paragraphs, returning according to a given template. - ‘Partial format‘: For example, if multiple steps are involved, they should be answered point by point; titles should be highlighted using ##; enclosed in angle brackets <>; certain parts should be bolded ...

    work page

Show all 40 references
  1. [9]

    (The constraints added here must not conflict with any possible character setting in the given character reference information, so please add such constraints cautiously)

    Style: - ‘Tone‘: For example, responses must be professional/informal/angry/sarcastic/impatient/vintage style/cute/rugged/gentlemanly... (The constraints added here must not conflict with any possible character setting in the given character reference information, so please ad...

  2. [10]

    we," or

    Role (used in very few cases, less than 5%, please combine with actual scenarios, use cautiously): - ‘Identity‘: For example, each response must first declare who they are, etc. - ‘Self-reference‘: For example, must refer to oneself as "we," or "this system," etc. - ‘Narration...

  3. [13]

    The name of the character

  4. [14]

    Generate a vivid and three-dimensional description of the character based on the given reference information, with no less than 100 characters

    Character description and basic information. Generate a vivid and three-dimensional description of the character based on the given reference information, with no less than 100 characters

  5. [15]

    Skill points, used to provide the user with references on which issues can be discussed with this chatbot

  6. [16]

    Generate 6 to 12 non-repetitive, non-conflicting, and distinctive constraints as needed

    Constraints related to the character’s description and skills, serving as preferences for the chatbot’s responses to user questions. Generate 6 to 12 non-repetitive, non-conflicting, and distinctive constraints as needed. Constraints refer to the preferences a chatbot must mee...

  7. [17]

    Generate more constraints that can be objectively verified, even constraints that can be directly checked using Python code

    The content of the constraints and their trigger conditions must be clear and objectively verifiable, not vague or ambiguous. Generate more constraints that can be objectively verified, even constraints that can be directly checked using Python code. Avoid generating constrain...

  8. [18]

    The provided system is just a reference and not complete

    Constraints need not be limited to the given constraint system. The provided system is just a reference and not complete. Use your imagination fully and brainstorm. Encourage thinking outside the given ontology’s preference limitations and generate constraints with high creati...

  9. [19]

    It is not necessary to generate constraints one by one according to the ontology

    Constraints should fit the usage scenario of this chatbot. It is not necessary to generate constraints one by one according to the ontology. Generate constraints suitable for these scenarios based on the specific situation of the character. 4. There must be absolutely no confl...

  10. [20]

    Constraints do not necessarily need to be simple atomic constraints and can be complex composite constraints

  11. [21]

    if the user is silent for more than 30 seconds,

    The constraints you generate will be used only for immediate, single-round, pure text dialogue scenarios. Absolutely avoid generating triggers like "if the user is silent for more than 30 seconds," which are unrelated to pure text dialogue; absolutely avoid generating constrai...

    2024

  12. [22]

    Whenever the user mentions ’boss’ or ’supervisor’, the response must use ’wise decision-maker’ at least once to maintain a positive and respectful tone

  13. [23]

    In any response that mentions a solution, it must begin with ’First, stay calm; second, ...’ to emphasize the importance of composure. 3. Negative words (such as: bad, stupid, wrong, etc.) are prohibited in replies. Use more tactful expressions (such as: not quite appropriate,...

  14. [24]

    For each question raised by the user, include a specific case or story in the response for added persuasive effect

    Every reply must end with an encouraging statement, such as ’You are fully capable of handling this!’ or ’Keep it up, you’re the best!’ 5. For each question raised by the user, include a specific case or story in the response for added persuasive effect

  15. [25]

    When the user expresses obvious frustration or stress, first offer emotional support, such as ’I understand how you’re feeling right now, it’s really tough,’ and then provide specific advice

  16. [26]

    If the user asks about handling relationships with colleagues, the response must mention the importance of building team spirit at least once

  17. [27]

    Avoid mentioning any specific company names directly or indirectly in responses to prevent potential legal risks

  18. [28]

    When users mention sustainable development, you must mention at least one relevant success case

    When the user asks questions unrelated to work, skillfully steer the conversation back to work-related topics, such as ’That’s indeed an interesting question, but back to work, how can we better protect ourselves from being affected?’ Table 19: Example of original GPTs from GP...

  19. [29]

    An AI Agent’s profile information (including the AI Agent’s basic information and constraints they must satisfy in responses)

  20. [30]

    Please design the user’s query (query) for the next round of dialogue based on the provided information

    The completed conversation history between the AI Agent and user. Please design the user’s query (query) for the next round of dialogue based on the provided information. The user queries must meet the following requirements (these requirements are very important and must be f...

  21. [31]

    Use more colloquial expressions, fewer overly formal expressions

    Natural and concise expression: The expression must not be stiff and rigid like robot speech. Use more colloquial expressions, fewer overly formal expressions. Can (but not must) have appropriate omissions. Expression should be concise (no more than 20 words), colloquial, and diverse

  22. [32]

    Hey", "I’m thinking

    Diverse querying methods: Forbidden to use querying methods similar to queries in the previous history. Forbidden to constantly start with: "Hey", "I’m thinking", "Hey", "AI Agent name", "Do you know", "If", "What if", "Recently", "I heard", and other common expressions. Forbi...

  23. [33]

    You can choose any tone to generate queries, including but not limited to: characteristic, etc., or combinations of these tones (but tones cannot conflict with each other)

    Diverse tone and attitude: Please use your imagination to create queries with as diverse tones as possible. You can choose any tone to generate queries, including but not limited to: characteristic, etc., or combinations of these tones (but tones cannot conflict with each other)

  24. [34]

    Consistent persona: Expression should be consistent with the user modeling in the user profile I provided, conforming to the user’s gender, age, background, occupation, tone, personality traits, communication style, and other settings in the user profile

  25. [35]

    You cannot directly copy their name exactly to address them, as that’s too mechanical and unlike normal human expression

    Do not use AI Agent’s full name: If the AI Agent’s name is too formal or too long, such as: Rail Fence Cipher Master, Mathematical Superhero (Second Generation). You cannot directly copy their name exactly to address them, as that’s too mechanical and unlike normal human expre...

  26. [36]

    ### In terms of query content: 1

    Do not use emojis. ### In terms of query content: 1. Diverse query content: Please fully utilize your imagination and creativity. You can ask very nonsensical queries, and have very diverse, imaginative, unexpected queries and ideas. 2. Chat topics: User queries don’t necessar...

  27. [37]

    Generated queries should be challenging enough, focusing on generating queries that will trigger the constraints this AI Agent needs to satisfy or queries that have continuity related to previous conversation history

  28. [38]

    Since you just mentioned xx is interesting

    Continuity doesn’t mean rigidly following up on elements in the AI Agent’s response, but reasonably continuing the dialogue following previous history. Don’t use rigid ways like "Since you just mentioned xx is interesting" to establish connections with historical dialogue

  29. [39]

    Generated queries can relate to content from any previous round of dialogue, not necessarily just the last query

  30. [40]

    When describing AI application scenarios, at least one metaphor must be used,

    Continuity is not mandatory and must be based on reasonable, natural premises. If it’s difficult to produce naturally continuous queries, you can generate queries without strong connections to previous dialogue history. Your output should only be the user query itself, without...

  31. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou

    arXiv

  32. [2024]

    arXiv preprint arXiv:2406.13542

    Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with...

    2022 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.