Pith. sign in

REVIEW 4 major objections 5 minor 39 references

ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ORPP confines prompt search to role-playing prompts, iteratively refines them on a small sample under a reward model, then transfers the best ones to new questions via few-shot learning, reporting accuracy gains over general prompt…

desk verdict Novel combination and broad experiments, but an unvalidated reward model and a wrong case study undermine the central claim. read the letter →

arxiv 2506.02480 v1 pith:WM4YBK7G submitted 2025-06-03 cs.CL

Yifan Duan , Yihong Tang , Kehai Chen , Liqiang Nie , Min Zhang This is my paper
classification cs.CL
keywords role-playingpromptspromptoptimizationfew-shotlearningrewardmodellargelanguagemodelsreasoningbenchmarksplug-and-playprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ORPP is a prompt optimization framework that deliberately searches only over role-playing prompts, rather than the full space of possible prompt text. The authors argue this narrowed search makes optimization cheaper and more tractable, because a role prompt can be evaluated quickly and improved iteratively on a handful of samples. They show that the best role prompts found for a few training questions can be transferred to new questions via few-shot prompting, and that the resulting system prompt improves accuracy on most of the six reasoning benchmarks they test. They also report that ORPP can be combined with existing prompting methods such as chain-of-thought, rephrase-and-respond, and step-back prompting, improving those methods' accuracy in most but not all cases.

What carries the argument

The load-bearing mechanism is the two-stage ORPP pipeline. In the first stage, for a random subset of about ten questions, the model generates several candidate role-playing prompts, answers the question under each prompt, and a reward model (ArmoRM-Llama3-8B-v0.1) scores the answers. The optimizer model then receives the best-scoring and worst-scoring prompts and answers from all rounds so far, produces a textual analysis (a 'text gradient'), and generates a new set of improved role prompts; after ten rounds the highest-scoring prompt per question is kept. In the second stage, the top few question-and-optimal-prompt pairs are used as few-shot examples to generate a role-playing prompt for each new test question, and that prompt becomes the system prompt for the final answer.

What would settle it

Run ORPP's optimization on 100 held-out questions per benchmark while also recording the ground-truth correctness of each candidate answer; if the reward model's scores and correctness labels disagree significantly (e.g., rank correlation below 0.3), the prompts chosen as 'best' are not actually optimizing the measured accuracy, contradicting the central claim.

Watch

Extended reading notes

Core claim

The central claim is that constraining prompt optimization to role-playing prompts activates latent capabilities of large language models more reliably than general prompt optimization. The paper's evidence: on GPQA, AGIEval-Math, MATH, MMLU-Pro, MMLU-CF, and MedQA, ORPP with Qwen2.5-14B and Qwen2.5-32B matches or beats OPRO and SPO in most settings, and when its role-playing system prompt is added on top of CoT, Rephrase, or Step-back, the combined method usually beats the base method alone. The paper also claims that role-playing prompts optimized with a smaller model (Qwen2.5-14B) improve a larger model (Qwen2.5-32B) on most tasks, indicating transferability.

Load-bearing premise

The reward model's score is assumed to be a reliable proxy for true accuracy on all six benchmarks; if it ranks a wrong answer above a correct one, the optimization will select prompts that do not maximize benchmark performance.

Editorial extensions

If this is right

  • Prompt optimization becomes cheaper: only ten training questions and ten refinement rounds are needed to produce role prompts that generalize to full test sets.
  • ORPP acts as a plug-and-play system prompt: stacking it with CoT, Rephrase, or Step-back improves those methods' accuracy on most of the six benchmarks.
  • Role prompts are transferable across model sizes: prompts optimized on Qwen2.5-14B improve Qwen2.5-32B on most tasks, suggesting small models can be used to tune prompts for larger ones.
  • The optimal number of few-shot examples is task-specific, so ORPP requires per-task tuning of that hyperparameter for best results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if role-playing prompts work mainly by making the model adopt a consistent expert reasoning style, the same few-shot transfer mechanism could be tested on open-ended generation tasks (creative writing, dialogue) where role adoption may have larger effects than on multiple-choice reasoning.
  • Beyond the paper: the reward model's alignment is the main risk; validating ArmoRM's scores against ground-truth labels on each benchmark before running the optimization would make the method's gains more reliable and could turn the reported 'most cases' into a stable ordering.
  • Beyond the paper: because ORPP optimizes only the system prompt, it should combine with any user-level prompt strategy; a task-aware selection rule for which base method to pair with ORPP could remove the negative cases reported for some datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ORPP, a two-stage framework for automatically generating role-playing prompts for LLMs. In Stage 1, a small subset of training questions is used to iteratively optimize role-playing prompts, with the prompt selection driven by the reward model ArmoRM-Llama3-8B-v0.1 (Eq. 1, Algorithm 1). In Stage 2, the top-scoring question-prompt pairs are used as few-shot examples to generate role-playing prompts for test questions (Algorithm 2), and the final answer is produced using the generated prompt as the system prompt. The method is evaluated on six benchmarks (GPQA, MMLU-CF, MMLU-Pro, MATH, AGIEval-Math, MedQA) with Qwen2.5-14B and Qwen2.5-32B, comparing against CoT, Rephrase, Step-back, OPRO, and SPO. The paper reports that ORPP generally outperforms these methods, can be combined with them as a plug-in, and that prompts optimized on the 14B model transfer to the 32B model.

Significance. If the reported results are reliable, the paper makes a practically useful contribution: restricting prompt optimization to role-playing scenarios and transferring optimization experience through few-shot examples is an inexpensive and flexible recipe that could lower the cost of prompt optimization while improving accuracy. The transferability experiment (smaller model optimizing prompts for a larger model) is a genuinely interesting finding, and the breadth of benchmarks is a strength. The paper also has honest limitation statements acknowledging negative plugin effects. However, the central mechanism is not yet established: the optimization objective is an unvalidated reward model score, and the headline comparisons are single-run numbers without variance estimates. These are fixable with additional experiments rather than being fundamental flaws, but they currently leave the main claims undersupported.

major comments (4)
  1. [§3.2, §4.1, Eq. (1), Algorithm 1, Table 6] The entire optimization loop selects prompts by maximizing S(M(Q|R)), where S is implemented as ArmoRM-Llama3-8B-v0.1, but the paper never validates that this reward score tracks actual benchmark accuracy. No correlation between reward scores and exact-match correctness is reported for any of the six benchmarks, no alternate reward model is tested, and no ablation replaces S with accuracy feedback on the small optimization subset. Table 6 illustrates the risk: the ORPP-selected response is marked as a success, yet the displayed chemical reasoning misrepresents trans-cinnamaldehyde as CH3-CH=CH-CHO (dropping the phenyl group) and reaches the answer through an incorrect structure. This is exactly the reward-hacking pattern that would break the claimed mechanism. A precondition for the central claim is evidence that the reward model's ranking agrees with accuracy on the evaluation benchmarks, or an ablation showing that optimization with accuracy as S gives similar results.
  2. [Table 2, §4.2] All results are based on a single random seed (42) with no error bars, confidence intervals, or significance tests. Several of the claimed advantages are small: for Qwen2.5-14B, ORPP scores 45.45 on GPQA versus 43.94 for CoT and OPRO, and 71.12 on MMLU-CF versus 70.02 for the base model. Without repeated runs or bootstrap estimates, the claim that ORPP 'not only matches but in most cases surpasses' competing methods cannot be distinguished from sampling noise. At minimum, the authors should report variance over at least three seeds or provide bootstrap confidence intervals for the main table.
  3. [Abstract, §4.2, Table 3, Limitations] The plug-and-play claim is stated too strongly relative to the data. Table 3 contains several substantial negative effects when ORPP is combined with other methods: on GPQA, CoT+ drops 4.04 points, Step-back+ drops 3.03, and OPRO+ drops 3.54; similar negative deltas appear for AGIEval-Math and MMLU-CF. The Limitations section acknowledges this, but the abstract still says ORPP can be integrated with various methods and 'further enhance their effectiveness.' The paper should either soften the abstract/contribution language to match the mixed results, or provide an analysis of when the plugin helps and when it hurts. Without such a criterion, 'in most cases' is not a quantitative claim and the strong compatibility conclusion in §5 is not supported.
  4. [§4.3, Figure 4] The transferability claim is based on a single transfer direction (14B-generated prompts applied to 32B) and reports a performance drop on GPQA. The text mentions this drop but does not analyze why GPQA behaves differently, even though the paper's own case study for GPQA (Table 6) shows that the optimized prompt can produce fluent but chemically incorrect reasoning. If the reward model systematically prefers confident, verbose but incorrect answers on GPQA-like questions, the transfer drop is expected. The paper should at least discuss this connection, or report whether the same drop occurs when the 32B model is given 14B-optimized prompts on a second difficult reasoning benchmark.
minor comments (5)
  1. [§4.2, Plugin Integration Effects] The text refers to 'Table X' instead of citing Table 3; the placeholder should be replaced with the actual table number.
  2. [§3.3] 'inputed' should be 'input' in the sentence describing the final answer generation.
  3. [§5] 'We proposes' should be 'We propose'.
  4. [§4.2] The text writes 'OPRR' when referring to ORPP ('using our method (OPRR) alone'); the acronym should be consistently ORPP throughout.
  5. [§4.4, Appendix C] Section 4.4 says case studies appear in Appendix C, but the appendix mainly describes an AGIEval-Math example while Table 6 is a GPQA example; the relationship between the two should be clarified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ORPP's optimization objective (reward-model score) is distinct from the reported evaluation metric (benchmark accuracy), and no step reduces to its inputs by construction.

full rationale

The paper's derivation chain is not circular. The optimization objective in Eq. (1) is R* = argmax_R S(M(Q|R)), where S is implemented as the external reward model ArmoRM-Llama3-8B-v0.1 (Section 4.1). Algorithm 1 selects Rbest(q) by that reward score, and Algorithm 2 selects few-shot exemplars by reward-score improvement. The reported results in Tables 2 and 3 are benchmark accuracies, not reward-model scores, so the 'prediction' is not identical to the optimization target by construction. No parameter is fitted to test-set labels, and the few-shot transfer stage is evaluated against held-out accuracy rather than against the reward model used for selection. The only self-citation found is Tang et al. (2024), cited in Related Work as background evidence that LLMs have potential in role-playing; it is not load-bearing for ORPP's central claim. The paper's own Limitations section concedes that role-playing prompts sometimes hurt performance, which further confirms that the claimed gains are empirical outcomes rather than definitional consequences. The unvalidated alignment between the reward model and accuracy is a legitimate correctness risk, as the skeptic notes, but it is not circularity under the stated criteria.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The method rests on three domain assumptions: the reward model score is a valid proxy for accuracy, the restricted role-playing prompt space is sufficiently expressive, and few-shot transfer generalizes from optimized examples. There are seven hand-chosen experimental parameters, most with no sensitivity analysis. No invented entities are introduced.

free parameters (7)
  • Number of optimized training samples = 10
    Randomly sampled from each training set with seed 42; the paper does not test sensitivity to this count, and it directly determines the few-shot example pool.
  • Optimization rounds N = 10
    Chosen by hand; no ablation varying N is reported.
  • Candidate prompts per round k = 3
    Chosen by hand; no ablation varying k.
  • Few-shot examples m = 3
    Chosen by hand; Figure 3 shows optimal m varies by task, so m=3 is a task-independent choice that may not be optimal.
  • Prompt generation temperature = 0.2
    Used only when generating role-playing prompts to add diversity; fixed without sensitivity analysis.
  • Reward model = ArmoRM-Llama3-8B-v0.1
    Chosen as the scoring function; no comparison to other reward models or validation of its alignment with accuracy.
  • Random seed = 42
    Used for sampling training subset and initial instances; single seed, so sampling variance is not estimated.
assumptions (3)
  • domain assumption The reward model score S(M(q|R)) is a valid proxy for answer quality and task accuracy.
    The optimization objective in Eq. (1) maximizes S, but all reported claims are about accuracy; if the reward model is misaligned, "best" prompts may not improve accuracy.
  • domain assumption The restricted space R of role-playing prompts contains prompts that are at least as effective as unrestricted prompts for the tested tasks.
    Section 3.1 defines the optimization over R only; if the optimal prompt is not role-playing, ORPP cannot find it.
  • domain assumption Few-shot in-context learning can transfer optimization patterns from m question-prompt pairs to unseen questions.
    Stage 2 (Algorithm 2) assumes the model generalizes from few-shot examples; the paper notes the optimal m is task-dependent, so this transfer is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities." pith.science (2026). https://pith.science/paper/WM4YBK7G

@misc{pith2026250602480,
  author       = {Pith},
  title        = {Pith review of: ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WM4YBK7G}},
  note         = {Machine review of arXiv:2506.02480}
}
read the original abstract

High-quality prompts are crucial for eliciting outstanding performance from large language models (LLMs) on complex tasks. Existing research has explored model-driven strategies for prompt optimization. However, these methods often suffer from high computational overhead or require strong optimization capabilities from the model itself, which limits their broad applicability.To address these challenges, we propose ORPP (Optimized Role-Playing Prompt),a framework that enhances model performance by optimizing and generating role-playing prompts. The core idea of ORPP is to confine the prompt search space to role-playing scenarios, thereby fully activating the model's intrinsic capabilities through carefully crafted, high-quality role-playing prompts. Specifically, ORPP first performs iterative optimization on a small subset of training samples to generate high-quality role-playing prompts. Then, leveraging the model's few-shot learning capability, it transfers the optimization experience to efficiently generate suitable prompts for the remaining samples.Our experimental results show that ORPP not only matches but in most cases surpasses existing mainstream prompt optimization methods in terms of performance. Notably, ORPP demonstrates superior "plug-and-play" capability. In most cases, it can be integrated with various other prompt methods and further enhance their effectiveness.

Figures

Figures reproduced from arXiv: 2506.02480 by the authors.

Figure 1
Figure 1. ORPP: A two-stage framework for generating [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of ORPP: A two-stage role-playing prompt generation and optimization framework that [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The impact of the number of High-Quality Question & Role-Playing Prompt Pairs in the few-shot [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: A comparison of the accuracy on different [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 15 canonical work pages

  1. [1]

    Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024 a . From persona to personalization: A survey on role-playing language agents. ArXiv, abs/2404.18231

    arXiv 2024

  2. [2]

    Jing Chen, Xinyu Zhu, Cheng Yang, Chufan Shi, Yadong Xi, Yuxiang Zhang, Junjie Wang, Jiashu Pu, Rongsheng Zhang, Yujiu Yang, and Tian Feng. 2024 b . Hollmwood: Unleashing the creativity of large language models in screenwriting via role playing. In Conference on Empirical Methods in Natural Language Processing

    work page 2024

  3. [3]

    Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T

    Yun-Shiuan Chuang, Zach Studdiford, Krirk Nirunwiroj, Agam Goyal, Vincent V. Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy T. Rogers. 2024. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. ArXiv, abs/2406.17232

    arXiv 2024

  4. [4]

    Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2023. Rephrase and respond: Let large language models ask better questions for themselves. ArXiv, abs/2311.04205

    arXiv 2023

  5. [5]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. ArXiv, abs/2309.16797

    arXiv 2023

  6. [6]

    Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, Xiaoqian Jiao, Chun Yong Chong, Shan Gao, and Michael R. Lyu. 2025. The prompt alchemist: Automated llm-tailored prompt optimization for test case generation. ArXiv, abs/2501.01329

    arXiv 2025

  7. [7]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  8. [8]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421

    2021

Show all 39 references
  1. [9]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024 a . https://doi.org/10.18653/v1/2024.naacl-long.228 Better zero-shot reasoning with role-play prompting . In Proceedings of the 2024 Conference of the North America...

    2024 doi

  2. [10]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024 b . https://arxiv.org/abs/2308.07702 Better zero-shot reasoning with role-play prompting . Preprint, arXiv:2308.07702

    2024 arXiv

  3. [11]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Jiaming Zhou, and Haoqin Sun. 2024 c . https://doi.org/10.48550/arXiv.2407.08995 Self-prompt tuning: Enable autonomous role-playing in llms . CoRR, abs/2407.08995

    2024 doi arXiv

  4. [12]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  5. [13]

    Xinyuan Li and Yunshi Lan. 2025. Large language models are good annotators for type-aware data augmentation in grammatical error correction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 199--213, Abu Dhabi, UAE. Association for Comput...

    2025

  6. [14]

    Keming Lu, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024 a . Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment. ArXiv, abs/2401.12474

    2024 arXiv

  7. [15]

    Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung yi Lee, and Shao-Hua Sun. 2024 b . Llm discussion: Enhancing the creativity of large language models via discussion framework and role-play. ArXiv, abs/2405.06373

    2024 arXiv

  8. [16]

    Guillermo Marco, Julio Gonzalo, Ram'on del Castillo, and Mar'ia Teresa Mateo Girona. 2024. Pron vs prompt: Can large language models already challenge a world-class fiction author at creative text writing? In Conference on Empirical Methods in Natural Language Processing

    2024

  9. [17]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. In Conference on Empirical Methods in Natural Language Processing

    2023

  10. [18]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311.12022

    2023 arXiv

  11. [19]

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. https://openreview.net/forum?id=AptTXihnhH Character- LLM : A trainable agent for role-playing . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  12. [20]

    Guijin Son, Sangwon Baek, Sangdae Nam, Ilgyun Jeong, and Seungone Kim. 2024. Multi-task inference: Can large language models follow multiple instructions at once? In Annual Meeting of the Association for Computational Linguistics

    2024

  13. [21]

    Yihong Tang, Bo Wang, Dongming Zhao, Jinxiaojia Jinxiaojia, Zhangjijun Zhangjijun, Ruifang He, and Yuexian Hou. 2024. https://aclanthology.org/2024.emnlp-main.437 Morpheus: Modeling role from personalized dialogue history by exploring and utilizing latent space . In EMNLP, pag...

    2024

  14. [22]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  15. [23]

    Laurens van der Maaten and Geoffrey Hinton. 2008. http://jmlr.org/papers/v9/vandermaaten08a.html Visualizing data using t-sne . Journal of Machine Learning Research, 9(86):2579--2605

    2008

  16. [24]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 a . Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Conference on Empirical Methods in Natural Language Processing

    2024

  17. [25]

    Li Wang, Xi Chen, Xiangwen Deng, Hao Wen, Mi Hee You, Weizhi Liu, Qi Li, and Jian Li. 2024 b . Prompt engineering in consistency and reliability with the evidence-based guideline for llms. NPJ Digital Medicine, 7

    2024

  18. [26]

    Lili Wang, Ruiyuan Song, Weitong Guo, and Hongwu Yang. 2024 c . Exploring prompt pattern for generative artificial intelligence in automatic question generation. Interactive Learning Environments, 33:2559 -- 2584

    2024

  19. [27]

    Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric Xing, and Zhiting Hu. 2024 d . https://openreview.net/forum?id=22pyNMuIoa Promptagent: Strategic planning with language models enables expert-level prompt optimization . In The Twelfth ...

    2024

  20. [28]

    Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen. 2024 e . Mmlu-pro: A more robust and challenging mu...

    2024 arXiv

  21. [29]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903

    2022 arXiv

  22. [30]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    2019 arXiv

  23. [31]

    Yurong Wu, Yan Gao, Bin Benjamin Zhu, Zineng Zhou, Xiaodi Sun, Sheng Yang, Jian-Guang Lou, Zhiming Ding, and Linjun Yang. 2024. Strago: Harnessing strategic guidance for prompt optimization. In Conference on Empirical Methods in Natural Language Processing

    2024

  24. [32]

    Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. 2025. Self-supervised prompt optimization. ArXiv, abs/2502.06855

    2025 arXiv

  25. [33]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    2024

  26. [34]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic "differentiation" via text. ArXiv, abs/2406.07496

    2024 arXiv

  27. [35]

    Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, and Furu Wei. 2024. Mmlu-cf: A contamination-free multi-task language understanding benchmark. ArXiv, abs/2412.15194

    2024 arXiv

  28. [36]

    Chi, Quoc V Le, and Denny Zhou

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. https://openreview.net/forum?id=3bq3jsvcQ1 Take a step back: Evoking reasoning via abstraction in large language models . In The Twelfth International Conference on L...

    2024

  29. [37]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied Sanosi Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. In NAACL-HLT

    2023

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.