Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Prompt quality can be decomposed into four text-predictable metrics, and an evaluator trained on those metrics can guide query-by-query prompt rewrites that beat both static templates and existing query-dependent optimizers on eight benchma

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:23 UTC pith:F75FVNC3

load-bearing objection Novel closed-loop evaluator–optimizer, but the evaluation is spoiled by query-side editing and selective reporting; the central claims are unsupported as submitted. the 4 major comments →

arxiv 2511.19829 v3 pith:F75FVNC3 submitted 2025-11-25 cs.AI

Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level

Ke Chen , Yifeng Wang , Hassan Almosapeeh , Haohan Wang This is my paper
classification cs.AI
keywords prompt optimizationexecution-free evaluationquery-dependent promptingprompt quality metricsinterpretable optimizationmulti-agent systemsLLM evaluationgradient attribution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most prompt-optimization methods either tune one fixed template for a whole task or rely on executing the model repeatedly to get feedback. This paper tries to close that loop differently: it defines prompt quality as a small set of performance-relevant dimensions—confidence in the correct answer, output stability, prompt–response mutual information, and intrinsic query difficulty—and trains a lightweight evaluator that predicts those dimensions and the probability of success directly from the prompt text, without running the model. The evaluator's per-dimension attributions then tell an optimizer exactly which facet to fix for each query. The paper reports that the evaluator predicts prompt performance with 83.7% accuracy, and that the resulting query-dependent optimization beats static-template and existing query-dependent baselines across eight benchmarks and three different backbone models, including a medical domain the system never trained on. If true, this would make prompt optimization cheap enough to run inside multi-agent systems and interpretable enough to trust.

Core claim

The paper's central claim is that a good prompt is one that scores well on four complementary, performance-reflective metrics: negative log-likelihood of the correct answer (how confidently the prompt routes the model to the right output), semantic stability (how consistent the outputs are across runs), mutual information (how much the prompt shapes the output beyond the query alone), and query entropy (how ambiguous the question itself is). It then fine-tunes an 8-billion-parameter instruction-tuned language model, with lightweight adapters, to predict these four scores and a binary quality label (whether average execution accuracy would exceed 50%) from the combined text of the query and p

What carries the argument

The load-bearing object is the evaluation-instructed optimizer loop, built from three pieces: (1) a diverse training pool of 11,530 prompts assembled from five static templates, six LLM-directed prompt styles, and genetic-style recombination, so the evaluator sees prompts with varied structures and quality; (2) an execution-free evaluator that predicts the four metric scores and the binary quality label, using a bi-level training objective that lets metric regression serve classification only when useful, a gradient-informed weighting scheme that up-weights metrics with stronger influence on classification, and a metric-aware feature fusion; and (3) gradient-based attribution, where the part

Load-bearing premise

The load-bearing premise is that the evaluator learns to judge prompt quality rather than recall query difficulty—so the 83.7% accuracy assumes the train/validation split is query-disjoint—and that the reported gains come from prompt rewriting, not from the optimizer's extra freedom to edit the query itself.

What would settle it

Check the 11,530-prompt pool for query overlap between the training and validation splits; if any query appears in both, recompute evaluator accuracy on a query-disjoint split. If accuracy falls substantially (toward the ~69% of the embedding-based baseline), the central claim of execution-free evaluation is unsupported. Separately, rerun the optimization on a held-out set while forbidding any edits to the query side; if the gains over baselines disappear, the reported improvements are largely attributable to query augmentation rather than diagnostic prompt rewriting.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Prompt optimization no longer requires repeated model execution at inference time: the evaluator scores a prompt in one forward pass, then the optimized loop can run inside a multi-agent system without multiplying token costs.
  • The same evaluator, trained on prompts from a single backbone, transfers its optimization signals to other backbones, including a stronger commercially hosted model, so the approach is model-agnostic rather than tied to the training model.
  • Because the evaluator's learned weights rank query entropy highest (32.7% of the classification weight), the method implies that the largest prompt-side gains come from clarifying the question itself, not from inventing fancier reasoning templates.
  • The framework generalizes to a held-out medical dataset with consistent 5–6% gains, indicating that the four metrics capture performance-reflective dimensions that are not domain-specific.
  • Reducing a noisy set of eight candidate metrics to four performance-reflective ones improves evaluator accuracy from 79.6% to 83.7%, evidence that focusing the evaluation on a few complementary dimensions is better for learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One testable extension: train a much smaller distilled evaluator (under a billion parameters) on the same four-metric scoring task and check whether optimization gains survive; the paper uses an 8B model, so the cost-savings claim for multi-agent settings would become stronger with a lighter scorer.
  • The dominance of query entropy suggests a division of labour worth testing: for high-entropy queries, optimizing the question formulation may return more value than optimizing the reasoning template; a suite could route queries accordingly.
  • Because the optimizer is permitted to augment the query with clarifications while the baselines only rewrite prompts, a cleaner test would compare all methods under identical action spaces; if the gains persist without query-side edits, the improvement is attributable to prompt rewriting rather than query augmentation.
  • The binary quality threshold of 50% accuracy is coarse; a graded label would let the evaluator discriminate between 'just above chance' and 'clearly correct' prompts, and might change which failure attributions the optimizer trusts.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a unified evaluation-instructed prompt optimization framework. It constructs a diverse prompt pool (11,530 prompt–query pairs) via static templates, LLM-generated styles, and evolutionary recombination; selects four prompt-quality metrics (NLL, stability, MI, query entropy) using XGBoost importance; fine-tunes a LLaMA-8B LoRA evaluator that predicts these metrics and an overall success probability from text without execution; and uses gradient-based attributions from the evaluator to guide query-dependent, metric-aware prompt rewriting. The authors claim the evaluator reaches 83.7% validation accuracy and that the resulting optimization pipeline 'consistently surpasses' static-template and query-dependent baselines across eight datasets and three backbone models.

Significance. If the claims were substantiated, this would be a meaningful step toward efficient, interpretable prompt optimization: an execution-free evaluator that produces dimension-specific signals could be plugged into multi-agent systems at low cost. The paper does include a systematic metric selection, a multi-component ablation, and cross-backbone experiments, which are strengths. However, the headline claims are currently undermined by (i) an unfair comparison setup in which the proposed method may modify the test query itself, (ii) multiple cells in Table 3 where the proposed method does not outperform baselines, and (iii) an unspecified train/validation split for the evaluator. The contribution is potentially valuable, but the empirical evidence as presented does not support the core claim of consistent superiority.

major comments (4)
  1. [§3.4, Query Entropy] The optimizer is allowed to 'augment the query side with minimal clarifications and constraints' when addressing the query-entropy dimension. This changes the test input for 'Ours' only; all baselines (Self-Refine, ProRefine, APE, TextGrad) can only modify the prompt/template. Therefore, improvements on tasks like LegalBench, MedQA, GPQA, and ambiguous BBH subsets could stem from injecting task-relevant information into the query rather than from better prompt optimization. An ablation disabling query-side augmentation—or equivalently giving baselines the same ability—is necessary to make the comparison interpretable. This is load-bearing because the central claim of consistent superiority rests on Table 3.
  2. [Table 3] The abstract and conclusion state that the approach 'consistently surpasses' baselines, but Table 3 contains multiple counterexamples. On bbh web of lies (llama3.1), Ours scores 0.68, below both the LLM-only baseline (0.69) and TextGrad (0.73); on GPQA Diamond (llama3.1), Ours scores 0.27 versus ProRefine's 0.29; on bbh disambiguation qa (gpt-4o), Ours scores 0.69 versus TextGrad's 0.71. Several other rows are ties or near-ties. No error bars, confidence intervals, or repeated-seed results are reported, so 'stable' and 'consistent' are unsupported even setting aside the fairness issue. At minimum, the claims need to be moderated to match the actual results, and statistical uncertainty needs to be quantified.
  3. [§4.3, Evaluator validation split] The evaluator is trained on 80% of the 11,530-prompt pool and validated on 20%, but the paper does not state whether the split is query-disjoint. Because the pool contains multiple prompt candidates per sampled question, overlapping queries between train and validation would let the evaluator memorize query-level difficulty rather than judge prompt quality, inflating the 83.7% accuracy. The paper should clarify the split construction and, if necessary, rerun with a query-disjoint split. This is load-bearing for the evaluator's headline accuracy, which is the foundation of the optimization.
  4. [§4.4 and Table 3, MATH500] The claim of consistent wins 'across eight datasets and on three backbone models' is not reflected in the table: MATH500 results are reported only for GPT-4o. For LLaMA-3 and LLaMA-3.1, the table shows dashes for all optimization methods, meaning no optimization comparison exists on that dataset for two backbones. This overstates the empirical coverage and should be corrected by either reporting results or explicitly restricting the claim to the seven datasets where all backbones were evaluated.
minor comments (3)
  1. [§3.2] The four metrics are selected on the same 11,530-prompt pool used to train and evaluate the evaluator, with a gain threshold of 10%. The sensitivity of the optimization results to this selection is not examined; a brief robustness check (e.g., using all eight metrics or an alternative threshold) would strengthen the claim that the selected metrics are the right ones.
  2. [References] The appendix cites 'Kojima et al., 2022', 'Zhou et al., 2022a', 'Hulbert, 2023', and 'Liang et al., 2023' but these are not in the reference list. Reference [24] and [25] are the same paper (Prompt-OIRL) with different numbering; this should be de-duplicated.
  3. [Throughout] Minor typos and inconsistent terminology: 'ect.' in §3.4, 'ProteGi' vs 'ProTeGi', and inconsistent capitalization of 'LLaMA-8B' vs 'LLaMA-3-8B'. The paper would also benefit from stating the number of optimization iterations used for the main results (it says max 3) and whether the same iteration budget was used for all baselines.

Circularity Check

0 steps flagged

No significant circularity: the evaluator and optimizer are measured against independent execution accuracy; the only self-citation (stability score) is non-load-bearing, and the main caveats are validity concerns, not circular construction.

full rationale

The claimed derivation chain is not circular. Prompt-quality labels are defined by execution accuracy (>50%) in Section 3.2; the XGBoost metric-selection step is supervised feature selection on those labels, and the evaluator's 83.7% figure is the validation accuracy of a learned classifier, not a quantity returned by its own inputs. The optimization loop in Section 3.4 is explicitly untrained ('The optimization process is untrained, relying entirely on the evaluator’s predicted scores'), and the headline results in Table 3 are external execution accuracies on test examples, not evaluator predictions. The one self-citation is the stability score attributed to the authors' prior work [2], but Section 3.4 supplies the full formula S(p,q)=1-(2/(N(N-1)))Σ d_ij, so the citation is attribution rather than load-bearing. Two validity concerns should be noted but are not circularity by construction: (i) Sections 4.1 and 4.3 do not state that the evaluator's 80/20 split is query-disjoint, so if the split is sample-wise, test-query prompts may appear in evaluator training and inflate the 83.7% accuracy; (ii) Section 3.4 permits the optimizer to 'augment the query side with minimal clarifications and constraints,' changing the test input for the proposed method relative to baselines. These affect the fairness and strength of the empirical claims, but they do not make any prediction mathematically equivalent to its training input, so the circularity score remains low.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The framework introduces no new theoretical entities; the only postulated 'entity' is the trained evaluator model and the metric-specific diagnosers, which are engineering components. The free parameters and domain assumptions above are the main places where the central claim depends on choices made by the authors rather than external, falsifiable constraints.

free parameters (4)
  • Accuracy threshold for binary quality label = 0.5
    Section 3.2: a prompt is labeled 'good' if its average accuracy over 10 runs exceeds 50%, chosen for majority-vote logic. This threshold defines all training labels and the optimization trigger (y<0.5), so the entire framework is sensitive to it.
  • Metric selection gain threshold = 10%
    Section 4.2: metrics with XGBoost gain importance below 10% are discarded. This cutoff is data-driven and ad hoc, and it determines the four core metrics used as evaluator targets and optimization signals.
  • Optimization trigger threshold = 0.5
    Section 3.4: the system initiates rewriting when the evaluator predicts y<0.5. This is a hand-set threshold not varied in experiments, and it controls when optimization costs are incurred.
  • Maximum optimization iterations = 3
    Section 4.4: all methods are capped at three iterations for 'fairness,' but the choice is arbitrary and affects final accuracy; no sensitivity analysis is provided.
axioms (4)
  • domain assumption Accuracy > 50% over 10 i.i.d. executions implies majority voting recovers the correct answer
    Section 3.2 uses this to justify the binary quality label. It assumes independent sampling and that majority voting is the intended decoding strategy, which may not match how prompts are actually used.
  • domain assumption The four selected metrics computed on LLaMA-3-8B transfer as valid optimization signals to LLaMA-3.1-8B and GPT-4o
    Section 4.4 evaluates cross-model generalization but the evaluator is trained only on LLaMA-3-8B metric values; the paper assumes metric semantics are backbone-invariant, which is plausible but unverified.
  • domain assumption Gradients of the evaluator's classification loss w.r.t. predicted metrics identify the true cause of prompt failure
    Section 3.4 uses g_i = |∇_{m_i} L_cls| to attribute failure to specific metric dimensions. This assumes local gradient sensitivity corresponds to causal influence on real performance, which is not proven and could misdirect optimization.
  • domain assumption XGBoost gain-importance on the 11,530-prompt training pool generalizes to unseen tasks and query distributions
    Section 4.2 selects metrics based on their correlation with the training labels; the paper assumes these correlations persist on held-out benchmarks like MedQA and MATH500 without direct evidence.

pith-pipeline@v1.3.0-alltime-deepseek · 12504 in / 12836 out tokens · 137431 ms · 2026-08-03T20:23:32.590747+00:00 · methodology

0 comments
read the original abstract

Prompt optimization has become a central mechanism for eliciting strong performance from LLMs, and recent work has made substantial progress by proposing diverse prompt evaluation metrics and optimization strategies. Despite these advances, prompt evaluation and prompt optimization are often developed in isolation, limiting the extent to which evaluation can effectively inform prompt refinement. In this work, we study prompt optimization as a process guided by performance-relevant evaluation signals. To address the disconnect between evaluation and optimization, we propose an evaluation-instructed prompt optimization approach that explicitly connects prompt evaluation with query-dependent optimization. Our method integrates multiple complementary prompt quality metrics into a performance-reflective evaluation framework and trains an execution-free evaluator that predicts prompt quality directly from text, avoiding repeated model executions. These evaluation signals then guide prompt refinement in a targeted and interpretable manner. Empirically, the proposed evaluator achieves 83.7% accuracy in predicting prompt performance. When incorporated into the optimization process, our approach consistently outperforms existing optimization baselines across eight benchmark datasets and three different backbone LLMs. Overall, our results demonstrate that reliable and efficient evaluation signals can serve as an effective foundation for robust and interpretable prompt optimization.

Figures

Figures reproduced from arXiv: 2511.19829 by Haohan Wang, Hassan Almosapeeh, Ke Chen, Yifeng Wang.

Figure 1
Figure 1. Figure 1: Overview of our pipeline. 3.1 Training Dataset Generation To train an evaluator that accurately captures how prompts behave across different quality dimensions, we require a dataset that spans a wide spectrum of prompt quality. Rather than generating only high￾performing prompts, our goal is to construct a diverse collection of prompts with varying structures, styles, and performance levels. This diversity… view at source ↗
Figure 3
Figure 3. Figure 3: Learned metric weights of the evaluator. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Improvement of our optimized prompts over the LLM-only baseline across three backbone [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses

    cs.CL 2026-03 unverdicted novelty 7.0

    PEEM is a multi-criteria LLM-based evaluator for prompts and responses that aligns with standard accuracy while enabling zero-shot prompt optimization via feedback.

Reference graph

Works this paper leans on

30 extracted references · 15 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Efficient prompting methods for large language models: A survey.arXiv preprint arXiv:2404.01077, 2024

    Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey.arXiv preprint arXiv:2404.01077, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    K. Chen, Y. Zhou, X. Zhang, et al. Prompt stability matters: Evaluating and optimizing auto- generated prompt in general-purpose systems.arXiv preprint arXiv:2505.13546, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Jared Kaplan, Reiichiro Nakano, Ashish Power, John Schulman, Ilya Sutskever, and Wojciech Zaremba. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  4. [4]

    Introducing mapo: Momentum-aided gradient descent prompt optimization.arXiv preprint arXiv:2410.19499, 2024

    Alex Cui, Pradyumna Nandyalam, Anthony Rufail, Dipankar Das, and Andrew McCallum. Introducing mapo: Momentum-aided gradient descent prompt optimization.arXiv preprint arXiv:2410.19499, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Model performance-guided evaluation data selection for effective prompt optimization

    Ximing Dong, Shaowei Wang, Dayi Lin, and Ahmed Hassan. Model performance-guided evaluation data selection for effective prompt optimization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Association for Computational Linguistics: ACL 2025, pages 2844–2859, Vienna, Austria, July 2025. Association fo...

    2025

  6. [6]

    The llama 3 herd of models.arXiv e-prints, 2024

    Abhimanyu Dubey, Aayush Jauhri, Anurag Pandey, et al. The llama 3 herd of models.arXiv e-prints, 2024

    2024

  7. [7]

    Critic: Large language models can self-correct with tool-interactive critiquing.arXiv preprint arXiv:2305.11738, 2023

    Ziang Gou, Zifan Shao, Yidong Gong, Junjie Shen, Yichong Xu, Yingbo Miao, Yelong Shen, Denny Jiang, Jinchao Yang, Benjamin Van Durme, and Yi Wang. Critic: Large language models can self-correct with tool-interactive critiquing.arXiv preprint arXiv:2305.11738, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Ho, Christopher R´ e, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher R´ e, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. C...

    2023

  9. [9]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    2021

  10. [10]

    Qpo: Query-dependent prompt optimization via multi-loop offline reinforcement learning.arXiv preprint arXiv:2408.10504, 2024

    Yining Kong, Hao Mao, Qiyuan Zhao, Yixin Zhang, Jie Tan, Chi Li, Hang Su, and Jun Zhu. Qpo: Query-dependent prompt optimization via multi-loop offline reinforcement learning.arXiv preprint arXiv:2408.10504, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Estimating mutual information

    Alexander Kraskov, Harald St¨ ogbauer, and Peter Grassberger. Estimating mutual information. Physical Review E — Statistical, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004

    2004

  12. [12]

    Luis A. Lastras. Information theoretic lower bounds on negative log likelihood.arXiv preprint arXiv:1904.06395, 2019. 12

    Pith/arXiv arXiv 1904

  13. [13]

    Camel: Communicative agents for “mind” exploration of large language model society

    Guohao Li, Hasan Hammoud, Hani Itani, Mohamad Kaddoura, Xuanming Li, Yue Jiao, Niket Tandon, Daniel Khashabi, Renjie Shao, Bill Yuchen Huang, Ferhan Ture, and Dan Roth. Camel: Communicative agents for “mind” exploration of large language model society. InAdvances in Neural Information Processing Systems, volume 36, pages 51991–52008, 2023

    2023

  14. [14]

    A practical survey on zero-shot prompt design for in-context learning

    Yinheng Li. A practical survey on zero-shot prompt design for in-context learning. InProceedings of the 14th International Conference on Recent Advances in Natural Language Processing (RANLP 2023), pages 641–647, Varna, Bulgaria, sep 2023. INCOMA Ltd., Shoumen, Bulgaria. also available as arXiv preprint arXiv:2309.13205

    Pith/arXiv arXiv 2023

  15. [15]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Beyond prompt content: Enhancing llm performance via content-format integrated prompt opti- mization.arXiv preprint arXiv:2502.04295, 2025

    Yuxuan Liu, Jinglong Xu, Linling Zhang, Ziyun Wang, Tianyu Chen, Yuxin Zhao, and Yang Huang. Beyond prompt content: Enhancing llm performance via content-format integrated prompt opti- mization.arXiv preprint arXiv:2502.04295, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Andrew Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8086–8098. Association for Computational Linguistics, 2022

    2022

  18. [18]

    Xu, Dale Schuurmans, Denny Yang, Prithviraj Ammanabrolu, Kyle Lo, Peter Clark, Hannaneh Hajishirzi, Wen-tau Yih, and Iz Beltagy

    Aman Madaan, Niket Tandon, Prakhar Gupta, Shuyan Wang, Frank F. Xu, Dale Schuurmans, Denny Yang, Prithviraj Ammanabrolu, Kyle Lo, Peter Clark, Hannaneh Hajishirzi, Wen-tau Yih, and Iz Beltagy. Self-refine: Iterative refinement with self-feedback. InAdvances in Neural Informa- tion Processing Systems, volume 36, pages 46534–46594, 2023

    2023

  19. [19]

    Dhananjay Pandita, Thilina Chathuranga Weerasooriya, A. P. Shah, et al. Prorefine: Inference-time prompt refinement with textual feedback.arXiv preprint arXiv:2506.05305, 2025

    arXiv 2025

  20. [20]

    gradient descent

    Reid Pryzant, Dan Iter, Jing Li, Aerin Lee, Sammy Chen, Hao Zhu, Alex Bolton, Michael Zeng, and Ahmed Hassan Awadallah. Automatic prompt optimization with “gradient descent” and beam search.arXiv preprint arXiv:2305.03495, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a bench- mark. InFirst Conference on Language Modeling, 2024

    2024

  22. [22]

    Prompt optimization in large language models.Mathematics, 12(6):929, 2024

    Antonio Sabbatella, Andrea Ponti, Ilaria Giordani, Antonio Candelieri, and Francesco Archetti. Prompt optimization in large language models.Mathematics, 12(6):929, 2024

    2024

  23. [23]

    Evaluating prompt effectiveness: Key metrics and tools, November 2024

    Drishti Shah. Evaluating prompt effectiveness: Key metrics and tools, November 2024. Blog post on Portkey.ai

    2024

  24. [25]

    Query-dependent prompt evaluation and optimization with offline inverse rl.arXiv preprint arXiv:2309.06553, 2023

    Hao Sun, Alperen H¨ uy¨ uk, and Mihaela van der Schaar. Query-dependent prompt evaluation and optimization with offline inverse rl.arXiv preprint arXiv:2309.06553, 2023

    Pith/arXiv arXiv 2023

  25. [26]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Sch¨ arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Pith/arXiv arXiv 2022

  26. [27]

    Promptagent: Strategic planning with language models enables expert-level prompt optimization.arXiv preprint arXiv:2310.16427, 2023

    Xian Wang, Chenghao Li, Zihang Wang, Hongxin Zhang, Yuchen Jiang, Xiang Gao, Siva Reddy, and Junxian He. Promptagent: Strategic planning with language models enables expert-level prompt optimization.arXiv preprint arXiv:2310.16427, 2023

    Pith/arXiv arXiv 2023

  27. [28]

    Cengiz Yang, Xuezhi Wang, Yinhan Lu, Ed Chi, Denny Zhou, Jason Wei, Xinyun Chen, Yao Zhao, Zhuyun Dai, and Quoc V. Le. Large language models as optimizers. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. 13

    2024

  28. [29]

    Instance-adaptive zero-shot chain-of-thought prompting

    Xiyuan Yuan, Chenyu Shen, Shuai Yan, Shusen Dong, Zhenguo Li, Yong Chen, and Yang Liu. Instance-adaptive zero-shot chain-of-thought prompting. InAdvances in Neural Information Pro- cessing Systems, volume 37, pages 125469–125486, 2024

    2024

  29. [30]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Jacob Boen, et al. Textgrad: Automatic “differentiation” via text.arXiv preprint arXiv:2406.07496, 2024

    Pith/arXiv arXiv 2024

  30. [31]

    Let’s think step by step:

    Yujia Zhou, Andrei Ioan Muresanu, Zhexin Han, Pratyusha Sharma, and James Zou. Large language models are human-level prompt engineers. InThe Eleventh International Conference on Learning Representations (ICLR), 2023. A Appendix Static Prompt T emplates Source “Let’s think step by step:” zero-shot CoT (Kojima et al., 2022) “Let’s work this out in a step by...

    2023