Pith. sign in

REVIEW 6 major objections 6 minor 17 references

P3: Prompts Promote Prompting

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read P3 claims that jointly optimizing system and user prompts in an offline loop, then retrieving the resulting prompt complements at query time, outperforms optimization that touches only one prompt component, across general and reasoning…

desk verdict P3's joint offline prompt optimization plus ICL variant is a genuinely useful idea, but the general-QA results are partially confounded by LLM-judge scoring and the paper lacks code, data, and error bars. read the letter →

arxiv 2507.15675 v1 pith:ZTNL2PW2 submitted 2025-07-21 cs.CL

Xinyu Zhang , Yuanquan Hu , Fangchao Liu , Zhicheng Dou This is my paper · ORCID
classification cs.CL
keywords automaticpromptoptimizationsystemuserin-contextlearningLLM-as-judgequery-dependentpromptingreasoningbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

P3 (Prompts Promote Prompting) is a two-stage method for automatic prompt optimization that treats the system prompt and the user prompt as one coupled object rather than two independent knobs. In an offline stage it iteratively generates and scores candidate complementary instructions for user prompts with an LLM judge, routes low-scoring hard samples into a buffer, and uses that buffer to rewrite the system prompt in alternating rounds. The best prompt-complement pairs become a dataset that powers the online stage: either by finetuning a small model to write complements at query time, or, in P3-ICL, by retrieving similar past complements as few-shot demonstrations. The paper reports that this joint, retrieval-backed procedure outperforms existing automatic prompt optimizers on general QA benchmarks (Arena-Hard, Alpaca-Eval 2.0, and its length-controlled variant) and reasoning benchmarks (GSM8K, GPQA) across five different instruction-tuned LLMs. A sympathetic reader would take the central message to be that unilateral prompt optimization is suboptimal because the system and user components interact, and that offline search plus lazy retrieval can buy most of the gain without a dedicated online model.

What carries the argument

The load-bearing mechanism is a closed offline loop that alternates two LLM-driven optimizers. For each raw user prompt, the method samples k candidate complementary instructions, generates answers with the current system prompt, scores them with an LLM-as-judge, and iteratively refines the candidate pool via few-shot LLM-as-optimizer prompts. Complements scoring above a threshold $\epsilon$ form the training or retrieval dataset; the rest go to a hard-sample buffer that periodically drives system-prompt optimization: candidate system prompts are scored on a batch of hard samples, and the best becomes the new current system prompt for subsequent user-prompt rounds. At deployment, P3-ICL uses a lightweight embedding model to retrieve relevant prompt-complement pairs and inserts them as in-context demonstrations, so the online optimizer is a nearest-neighbor lookup instead of a 7B fine-tuned model.

What would settle it

Compare the LLM-judge scores used to select complements with human preference or ground-truth correctness on the same candidate pool; if the selected complements only lengthen, structure, or hedge responses without raising checked factual accuracy, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that concurrently optimizing system and user prompts, rather than only one component, produces consistently better LLM responses, and that the offline optimization results can be reused online through retrieval. The optimized system prompt is scenario-level and fixed at deployment, while optimized user-prompt complements are query-specific. In experiments, P3's average gain over PAS on Alpaca-Eval 2.0 across five models is about 9.6 points (57.16 vs 47.52), its GSM8K score reaches 84.8% for GPT-3.5-turbo, and P3-ICL beats P3 itself on the smaller models while cutting extra inference latency from 530 ms to 70 ms. The paper concludes that holistic optimization plus efficient online reuse yields superior automatic prompt optimization.

Load-bearing premise

The load-bearing premise is that the LLM-as-judge scores used to select every candidate are an unbiased proxy for the final benchmark metrics, and this premise is weakest where the benchmark itself is scored by an LLM judge rather than checked against ground truth.

Editorial extensions

If this is right

  • If the central claim is right, prompt-optimization systems should stop optimizing system and user prompts separately; the interaction between the two is itself a tunable axis.
  • P3's reported gains hold across five different instruction-tuned LLMs on general QA, with the largest jumps on smaller models such as GPT-3.5-turbo and Qwen2-72B, so weaker models benefit most from joint optimization.
  • On reasoning tasks, P3 lifts GPT-3.5-turbo on GSM8K from 72.9% zero-shot CoT to 84.8%, beating TextGrad, DSPy, and PAS, and improves GPT-4o's GPQA accuracy by at least 2.1 points over the same baselines.
  • P3-ICL shows that offline-synthesized prompt complements can be deployed by retrieval instead of by a fine-tuned 7B model, cutting extra latency from 530 ms to 70 ms and memory from 18,000 MiB to 5,000 MiB while staying competitive with, and sometimes ahead of, the fine-tuned variant.
  • Increasing the number of offline search iterations improves scores on Arena-Hard and Alpaca-Eval, so the method's gains scale with the optimization budget spent before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the LLM judge rewards length, structure, or hedging, part of the reported general-task gains could be stylistic rather than factual, so a human-preference or ground-truth re-scoring of the same candidate pools would separate these components.
  • Editorial inference: P3-ICL's retrieval design implies complementary instructions transfer across similar queries; a direct extension would vary the retriever, the number of demonstrations, and the similarity threshold to map how much of the gain comes from retrieval quality versus the instructions themselves.
  • Editorial inference: the hard-sample feedback loop only improves the system prompt, not the user-prompt optimizer; one could extend it to adapt the complement generator itself to hard queries, closing the loop on both sides.
  • Editorial inference: because the offline optimizer is a single model rather than the same model used in the PAS baseline, a controlled comparison holding the optimizer identical would isolate how much of the gain comes from joint optimization itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces P3, a two-stage prompt optimization framework that jointly optimizes the system prompt and user-prompt complements offline using an LLM-as-optimizer and LLM-as-judge, and then adapts online either by finetuning a small model or by retrieving complements from the offline dataset for in-context learning (P3-ICL). The evaluation compares P3 and P3-ICL against BPO, PAS, TextGrad, and DSPy on Arena-hard, Alpaca-Eval 2.0 (including the length-controlled variant), GSM8K, and GPQA across several base LLMs, reporting improvements on most settings and lower inference cost for P3-ICL. The paper also includes ablations on system-prompt optimization, diverse complement generation, and P3-ICL efficiency.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: treating the system prompt and user prompt as coupled, optimizing them jointly, and amortizing the result into a reusable complement database is a sensible design, and the P3-ICL efficiency measurements in Table 5 are concrete and valuable. The exact-match reasoning results on GSM8K and GPQA are not subject to the judge circularity that affects the general-QA benchmarks. However, the current evidence is insufficient: the general-QA gains may reflect optimization toward the LLM judge rather than toward task quality, several head-to-head claims are contradicted by the paper's own tables, and no uncertainty estimates or reproducibility artifacts are provided. The contribution is promising but the validation needs substantial work before the advertised conclusions can be accepted.

major comments (6)
  1. [Abstract / Table 1] The abstract's claim that P3 'consistently outperforms' existing prompt optimization methods is not supported by Table 1: on Arena-hard with GPT-4-1106-preview, P3 scores 77.21 while PAS scores 78.80. The paper should either qualify the claim or show that the difference is within noise; with no error bars or significance tests this cannot be established.
  2. [Section 2.2.1 / Table 1 / Appendix 7.7] The optimization signal in Section 2.2.1 is an LLM-as-judge score whose rubric (Appendix 7.7) rewards helpfulness, relevance, accuracy, depth, creativity, and level of detail, while three of the five evaluation benchmarks in Table 1 (Arena-hard and both Alpaca-Eval variants) are themselves scored by LLM judges. The optimized system prompt in Appendix 7.2.1 explicitly asks for comprehensive, step-by-step, engaging, and informative responses, a style that LLM judges may reward even after length control. Because no held-out exact-answer evaluation, human-preference check, or cross-judge transfer test is provided for general QA, the large margins in Table 1 do not yet support the conclusion that P3 improves factual response quality rather than judge-reported preference.
  3. [Tables 1-4 and 6] None of the reported comparisons include error bars, multiple runs, or significance tests. Many of the headline margins are small relative to likely variance: for example, Arena-hard GPT-4-turbo P3 78.00 vs PAS 76.90, P3-ICL 76.27 vs raw 76.60 in Table 1, and GSM8K 84.8 vs PAS 81.3 in Table 2. Without variance estimates or repeated runs, the paper's ranking claims are not statistically established.
  4. [Section 3.1.1 / Table 1] Section 3.1.1 does not state whether BPO and PAS were re-run under identical conditions (same optimizer model, same finetuning data, same evaluation protocol) or whether their numbers are copied from prior papers; the note that P3 uses GPT-4o-mini while GPT-4o was 'employed by PAS' suggests the baselines used a stronger optimizer, which would make the comparison favorable to P3. In addition, P3-ICL's retrieval details (embedding model, number of retrieved demonstrations, similarity threshold) and any overlap between the offline complement database and the evaluation prompts are not reported; if Arena-hard or Alpaca-Eval prompts occur in the offline dataset, P3-ICL could be retrieving near-exact matches.
  5. [Section 3.3.1 / Table 3] Section 3.3.1 states that P3 'consistently outperforms its variant without system prompt optimization across all benchmarks and LLMs,' but Table 3 shows the opposite on Arena-hard for GPT-4-turbo-2024-04-09 (P3 78.00 vs P3 w/o system 78.43), GPT-4-1106-preview (77.21 vs 77.87), and Qwen2-72b-Instruct (52.82 vs 53.83). This directly undermines the affinity claim and must be corrected or explained.
  6. [Availability] Neither code nor data are provided, and the appendix does not include the full set of generated complements or retrieval indices. Given that several claims rest on point estimates and on the content of the optimized prompts, releasing the optimization code, evaluation scripts, and the complement database (or at least a reproducible subset) is necessary for the empirical claims to be verifiable.
minor comments (6)
  1. [Appendix 7.2] The phrase 'we manfully include a piece of instruction' should be 'manually', and the color cue ('text in blue') is lost in black-and-white printing.
  2. [Section 1] There is a typo 'Arean-Hard', and the benchmark names are used inconsistently (Arena-hard vs Arena Hard, Alpaca-Eval 2.0 vs Alpaca-Eval (LC)).
  3. [Algorithm 1 / Appendix 7.7] The buffer names Xµ and X′µ in Algorithm 1 are inconsistent with the text's Xu and X∗u, and 'examplers' should be 'examples' (also in Appendix 7.7).
  4. [Appendix 7.1] The text refers to Figure 4 for the search-depth ablation, but no such figure appears in the manuscript, so the appendix's central claim cannot be checked.
  5. [Section 4.1] The sentence 'through revision the input prompt' is grammatically incomplete and should be rewritten.
  6. [Section 3.1.1] The descriptions of Arena-hard and Alpaca-Eval are not accurate: these are preference/comparison benchmarks scored by LLM judges, not benchmarks of 'noise resilience' or 'linguistic complexity' as written.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimization loop and the evaluation benchmarks are separable, and the LLM-as-judge overlap is a validity concern rather than a definitional reduction.

full rationale

The paper's derivation chain is not circular. P3's offline stage optimizes complementary user instructions and system prompts using LLM-as-judge scores over the PAS training dataset, while the reported evaluations are run on distinct benchmarks (Arena-hard, Alpaca-Eval, GSM8K, GPQA) against external baselines. The optimization objective is not defined in terms of the evaluation outcomes, and no parameter is fitted to the test benchmarks and then presented as a prediction. The system prompts in Appendix 7.2 are selected with the same general judge rubric used by the general QA benchmarks, so part of the gain on judge-scored benchmarks could reflect optimizing to judge preferences rather than factual quality; however, that is an external-validity and reward-hacking risk, not a circular reduction by construction. GSM8K and GPQA use exact-answer or reference-answer checks and still show gains, providing independent signal beyond the judge-based benchmarks. Citations to PAS, BPO, TextGrad, DSPy, and related work are to external prior work without author overlap with the present paper; no load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling citation appears. The claimed contribution—joint offline optimization of system and user prompts plus a retrieval-based ICL variant for online adaptation—has content beyond its inputs and is not equivalent to any fitted parameter or renamed benchmark result.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The method introduces no mathematical derivation and no new entities. Its empirical claims rest on the chosen judge, the offline dataset, and cross-model transfer assumptions. Several hyperparameters are tuned by hand, especially the per-dataset epsilon threshold.

free parameters (6)
  • epsilon (score threshold) = 6, adjusted per dataset
    Decides which optimized complements enter the training set versus the hard sample buffer; Section 7.6 says it is an empirical number adjusted based on dataset difficulty.
  • k (candidate count) = 5
    Number of candidate complements or system prompts sampled per step; affects search width in the offline loop.
  • c (new candidates per refinement round) = 5
    Number of new candidates generated by the LLM-as-optimizer during each refinement iteration.
  • C (system prompt buffer size) = 3
    Number of high-scoring new system prompts kept in the buffer during system prompt optimization.
  • T (system prompt optimization interval) = 400 for PAS and GSM8k, 80 for GPQA
    How often the system prompt is re-optimized in the offline loop; tuned per dataset.
  • search depth D = 1 in main experiments, up to 3 in ablation
    Number of refinement rounds for user prompt complements; the main result uses depth 1, which is a design choice that affects all reported gains.
assumptions (3)
  • domain assumption LLM-as-judge scores reflect true response quality
    Used to select complementary instructions and system prompts throughout the offline optimization loop, including the meta-prompt in Section 7.7.
  • domain assumption The PAS offline dataset is representative of the evaluation benchmarks
    General-task offline optimization uses the dataset provided by PAS, while evaluation is done on Arena-hard and Alpaca-Eval; Section 3.1.1 states this usage without overlap analysis.
  • domain assumption A system prompt and complement pairs optimized with one base model transfer to other LLMs
    Offline answer generation and scoring are done with a single base model (GPT-3.5-turbo-1106), but evaluation is applied to GPT-4, Qwen2, and LLaMA-3, assuming transferability across model families.

how reviews work

0 comments
Cite this review

Pith. "Pith review of P3: Prompts Promote Prompting." pith.science (2026). https://pith.science/paper/ZTNL2PW2

@misc{pith2026250715675,
  author       = {Pith},
  title        = {Pith review of: P3: Prompts Promote Prompting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZTNL2PW2}},
  note         = {Machine review of arXiv:2507.15675}
}
read the original abstract

Current large language model (LLM) applications often employ multi-component prompts, comprising both system and user prompts, to guide model behaviors. While recent advancements have demonstrated the efficacy of automatically optimizing either the system or user prompt to boost performance, such unilateral approaches often yield suboptimal outcomes due to the interdependent nature of these components. In this work, we introduce P3, a novel self-improvement framework that concurrently optimizes both system and user prompts through an iterative process. The offline optimized prompts are further leveraged to promote online prompting by performing query-dependent prompt optimization. Extensive experiments on general tasks (e.g., Arena-hard and Alpaca-eval) and reasoning tasks (e.g., GSM8K and GPQA) demonstrate that P3 achieves superior performance in the realm of automatic prompt optimization. Our results highlight the effectiveness of a holistic optimization strategy in enhancing LLM performance across diverse domains.

Figures

Figures reproduced from arXiv: 2507.15675 by the authors.

Figure 1
Figure 1. An illustration of P3’s optimization results. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of our methods. The complete process includes the offline joint optimization for user [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Plots for complements’ evaluation score by [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effect of different search depths In this ablation, we examine whether increasing the search depths will improve P3’s end-to-end per￾formance. We choose GPT-3.5-turbo-1106 as the base model, and increase the search depth (num￾ber of iterations) from 1 to 3. The evaluat…
Figure 5
Figure 5. Figure 5: Comparison between PAS prompts and P3 prompts. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [1]

    The instruction should serve as a guidance for the answer, providing feasible direction and thought process, but not revealing the answer itself

    work page

  2. [2]

    Try to keep it short and concise

    The complementary instruction should focus on general methodology. Try to keep it short and concise

    work page

  3. [3]

    Journal of Machine Learning Research, 25(70):1–53

    Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman

    work page

  4. [4]

    <score>5</score>

    When generating your response, the complementary instruction should be bracketed with <INS> and </INS>. User: <user_prompt>prompt</user_prompt> Meta Prompt for Complement Optimization Your task is to generate an instruction as a complementary text for the user prompt. ### User prompt: {prompt} Below are some previous instructions with their scores. The sc...

    work page

  5. [5]

    gradient descent

    Openai o1 system card. arXiv preprint arXiv:2412.16720. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Za- haria, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into self-improving pipelines...

    arXiv 2024

  6. [9]

    The complementary instruction should be in the same language with the user prompt

    work page

  7. [11]

    The optimized prompt should help LLMs to provide better answers for a wide range of queries

    work page

  8. [12]

    Make sure to make good use of this information

    There is always a piece of complementary instruction in user queries, which provides hints to solve the queries. Make sure to make good use of this information

    work page

Show all 17 references
  1. [13]

    When generating your response, the optimized prompt should be bracketed with <INS> and </INS>. Meta Prompt for System Prompt Optimization You are a large language model prompt engineer responsible for completing prompt optimization tasks by using historical prompt optimization...

  2. [14]

    Analyze the evolution patterns and score trends of historical prompts, and summarize the effective optimization rules

  3. [15]

    Based on the summarized optimization rules, generate a new prompt that is superior than all the previous ones

  4. [16]

    differentiation

    IEEE. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. Victor Sanh, Albert Webson, Co...

    2024 arXiv

  5. [17]

    Make sure to make good use of this information

    There is always a piece of complementary instruction in user queries, which provide hints to solve the queries. Make sure to make good use of this information

  6. [18]

    The newly generated prompt must be bracketed with <INS> and </INS> tags. 7.8 Details of Training Online Query Optimization Model As for training the online query optimization model, we finetune a Qwen2-7b-instruct model on the offline collected (user prompt, complement) pairs....

    2020

  7. [2021]

    arXiv preprint arXiv:2110.14168

    Training verifiers to solve math word prob- lems. arXiv preprint arXiv:2110.14168. Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learnin...

    2022 arXiv

  8. [2023]

    arXiv preprint arXiv:2311.04155

    Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al

    arXiv

  9. [2024]

    In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    Trace is the next autodiff: Generative opti- mization with rich feedback, execution traces, and LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.