Pith. sign in

REVIEW 4 major objections 4 minor 56 references

TAPR, a small model trained with reinforcement learning, rewrites prompts so that a frozen larger model answers them more accurately, demonstrated on question answering and arithmetic benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A small LLM trained with GRPO and LLM-judge rewards rewrites simple prompts into more effective ones, improving question-answering and arithmetic accuracy over base prompts while giving mixed, often negligible gains on summarization.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection GRPO-driven prompt rewriting is worth a look, but the paper oversells consistency and the judge setup needs work. the 4 major comments →

arxiv 2607.28657 v1 pith:7G2KCMHD submitted 2026-07-17 cs.AI

TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter

Oliver Savolainen , Emanuele Bastianelli , Hosein Azarbonyad This is my paper
classification cs.AI
keywords prompt rewritingreinforcement learningLLM-as-a-judgeGRPOprompt engineeringquestion answeringarithmetic reasoningfrozen models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a compact language model, trained only with reward signals from an LLM judge, can rewrite ordinary prompts into task-specific instructions that make a frozen, larger model perform better. The method, TAPR, uses Group Relative Policy Optimization to update the rewriter from the judge's scores of both the rewritten prompt and the downstream answer. On Natural Questions, rewritten prompts lift judged accuracy from 56.3% baseline to 59.2% with one rewriter and 62.2% with another; on GSM8K the gains are from 82.4% to 83.6% and 82.8%. The authors argue this makes automated prompt rewriting a practical way to unlock more of a frozen model's capability without manual prompt engineering. A sympathetic reader would care because success would mean non-experts can get better LLM answers by simply describing their request, and smaller models could boost larger ones at low cost.

Core claim

The central claim is that a smaller model can be trained through reinforcement learning to consistently improve the outputs of a frozen, larger model purely by editing the text of the prompt. TAPR is trained with GRPO, where each candidate rewrite is scored by an LLM judge twice: once for the quality of the prompt itself and once for the correctness of the frozen model's response, and the two scores are combined with equal weight. The authors report that this training yields stable learning and that the resulting rewriter beats both the original prompt and the untrained base model on question answering (Natural Questions and HotpotQA) and arithmetic reasoning (GSM8K), while summarization gai

What carries the argument

The load-bearing mechanism is the pairing of Group Relative Policy Optimization (GRPO) with an LLM-as-a-judge reward. GRPO updates the rewriter's policy using relative advantages across sampled rewrites for the same prompt, avoiding a separate value model; the reward combines a 0–5 prompt-quality score from one judge with a correctness score of the frozen model's answer from another judge. The prompt-quality term is what keeps training from drifting into degenerate rewrites, and the paper's ablation shows that removing it stalls learning.

Load-bearing premise

The claim rests on the judge's scores being a faithful measure of answer quality: if GPT-4o-mini's judgments are biased or can be gamed, the accuracy improvements may not reflect genuine task performance.

What would settle it

Compare TAPR-rewritten outputs on the same 1,000-sample NQ and GSM8K splits using human annotators instead of GPT-4o-mini; if the human-verified accuracy is not higher than the baseline prompt's, the central claim fails. Alternatively, flip the order of the two judged options in the comparison prompt and see whether the win-rates reverse, which would indicate the judge, not the prompt, is responsible.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, everyday users can obtain stronger LLM performance without learning prompt engineering.
  • A small, cheap rewriter model could be inserted in front of a frozen larger model, cutting cost while preserving quality.
  • LLM-as-a-judge rewards become a viable training signal for open-ended tasks where exact-match metrics are unreliable.
  • The same training recipe could be applied to other frozen models and new tasks with little modification.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the evaluation oracle (GPT-4o-mini) is also one of the training judges and exhibits a documented second-position bias, the reported gains may shrink or reverse under human evaluation; a human-annotated evaluation would settle this.
  • The 'clarity' improvements in rewritten prompts may be partly an artifact of the prompt-quality reward, which rewards adherence to prompt-engineering heuristics rather than end-task correctness; the paper's own discussion notes that reward gains do not always transfer.
  • A testable extension: apply TAPR to open-ended generation (dialogue, instruction following) where human evaluation is used as the gold standard, to see whether the judge-based gains generalize beyond benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces TAPR, a small LLM (Phi-4-mini-instruct or LLaMA-3.2-3B-Instruct) trained with GRPO to rewrite a one-line task prompt for a frozen Task LLM (LLaMA-3.1-8B-Instruct). Training rewards combine an LLM-as-a-judge score of the Task LLM's answer and an LLM-judge 'prompt quality' score. Final evaluations use GPT-4o-mini on 1,000 samples per dataset across question answering (Natural Questions, HotpotQA), summarization (CNN/Daily Mail, SciTLDR), and arithmetic reasoning (GSM8K). The paper reports gains over baseline-prompt and base-model rewriting for several settings, plus ablations with PPO, SFT, prompt-quality reward, selection mechanism, full-prompt rewriting, alternative Task LLMs, and cross-task generalization. The abstract and conclusion claim 'consistent gains,' but Section 6.1 and several tables qualify or directly contradict this claim.

Significance. The topic is timely and the paper ships code, detailed hyperparameters, multiple ablations, and an explicit discussion of LLM-judge limitations, which is commendable. The central research question is meaningful: can a small model trained with RL improve a frozen larger model by rewriting prompt text? However, the current evidence does not establish the headline claim. The main positive results on NQ and HotpotQA are measured by GPT-4o-mini, which is also one of the two reward judges used during training, and the paper documents a strong second-position bias for this judge. The only verifiable result (GSM8K last-integer accuracy) shows modest gains, while summarization results show trained LLaMA TAPR below its own base model and Phi TAPR below its base model on SciTLDR. As presented, the paper is better described as a well-documented partial/negative result than as a demonstration of consistently improved downstream performance. The authors themselves state in §6.4 that gains are not yet consistent enough to offset additional training and inference costs.

major comments (4)
  1. [§3.2, §4, §6.3] The evaluation is not independent of training. GPT-4o-mini is one of the two reward judges during GRPO training (Section 3.2) and the sole final evaluator (Section 4). The NQ and HotpotQA gains in Tables 2 and 3 may therefore reflect the rewriter learning to exploit surface features favored by this judge, rather than genuinely improving answer quality. The paper documents a strong second-position bias for GPT-4o-mini (Table 7 note, Section 6.3) and validates the judge on only 1/100 NQ cases (footnote 1), which does not rule out condition-dependent leniency toward TAPR outputs. Please provide human evaluation or an independent judge on the final outputs, or restrict the headline claim to verifiable metrics such as GSM8K last-integer accuracy.
  2. [Abstract and Conclusion vs §5–§6] The claim of 'consistent gains' is directly contradicted by the paper's own tables. In Table 5, LLaMA-3.2 TAPR scores 3.435 vs 3.794 for the untrained base model on CNN/DM; in Table 6, LLaMA TAPR scores 3.107 vs 3.749, and Phi TAPR scores 3.710 vs 3.802, on SciTLDR. Appendix Table 15 shows full-prompt TAPR below the baseline prompt for all models on NQ, and Table 17 shows multi-task TAPR below baseline on both summarization datasets. Section 6.1 itself states that gains are 'not always consistent.' Please revise the abstract and conclusion to match the evidence.
  3. [§4, Tables 2–9] No repeated runs or error bars are reported; Section 4 explicitly states that 'each training and evaluation is conducted only once.' Differences of 1–3 percentage points over 1,000 samples (e.g., Phi TAPR 59.20 vs baseline 56.30 on NQ; LLaMA TAPR 56.00 vs baseline 53.20 on HotpotQA) may be within sampling noise. Add multiple seeds, or at least bootstrap confidence intervals over the 1,000 evaluation samples, to support quantitative claims.
  4. [§3.2, §5.4, §6.1] The prompt-quality reward is defined by an LLM judge checking adherence to prompt-engineering principles, so the observed 'clearer, more instructive' rewriting (Abstract, §7) is substantially the training objective itself. Section 6.1 concedes that 'much of the reward improvement may reflect the model’s ability to satisfy prompt quality criteria rather than the end-task metrics.' This circularity should be acknowledged in the abstract and conclusion, and the qualitative claims tempered accordingly.
minor comments (4)
  1. [Footnote 1, §3.2] The 'internal test' that found only 1/100 incorrect judgments is not described in enough detail to be reproduced. Please include the evaluation protocol, sample size, model version, and how human labels were obtained.
  2. [Table 11] The PPO variant labels 'Con.', 'Mod.', 'Agg.' are defined in the caption, but the table format would be clearer if the hyperparameter values (learning rate, KL coefficient) were listed in a separate column or expanded caption.
  3. [Figures 2 and 3] The figures show raw and moving-average training rewards, but the axis labels and units are not fully specified in the captions. Please state what the y-axis represents and the window size for the moving average.
  4. [§6.2] The claim that 'training only improves performance with a specific prompt' is clear, but the example of the Phi-4-mini CNN/DM run producing a joke prompt suggests a degenerate failure mode that should be reported in the main results or a dedicated error-analysis section, not only in the discussion.

Circularity Check

2 steps flagged

Partial circularity: the NQ/HotpotQA evaluation judge was also a training reward judge, and the 'clearer prompts' finding restates the prompt-quality reward; GSM8K and other external metrics retain independent content.

specific steps
  1. fitted input called prediction [Section 3.2 (Rewards) and Section 4 (Evaluation Protocol); Tables 2–3]
    "To reduce bias and stabilize the reward signal, we average the LLM-as-a-judge scores from two different judge models (LLaMA-3.1-8B-Instruct and GPT-4o-mini). ... During evaluation, we use GPT-4o-mini as the sole judge model to score answers."

    GPT-4o-mini is one of the two reward judges used during GRPO training, and it is also the sole final evaluator. The reported NQ gains (56.30 to 59.20/62.20) and HotpotQA gains (53.20 to 56.70/56.00) are therefore measured by an oracle that the policy was directly optimized to satisfy. This is not an independent check of answer quality; it is partly a re-measurement of the training objective. The paper itself documents the same judge's strong second-position bias (Table 7 note, Section 6.3), making the overlap consequential. The reduction is partial because training averaged two judges and evaluation used held-out samples, and the GSM8K results use an external last-integer metric.

  2. self definitional [Section 3.2 (prompt quality reward) and Section 6.4 (Future Work)]
    "The prompt quality reward is a score on a scale of 0 to 5 coming from an LLM judge based on whether the rewritten prompt matches the meaning of the initial instruction and improves it according to common prompt engineering principles. ... This is partially supported by our finding that rewritten prompts tended to be clearer, more detailed, and more consistent with prompt engineering best practices than the baseline instructions."

    The qualitative claim that TAPR produces 'clearer, more detailed' prompts 'consistent with prompt engineering best practices' is the exact criterion encoded in the prompt-quality reward Q(p̃). The observed property is therefore a restatement of the training objective, not an independent finding about prompt quality. Example prompts and win-rates provide some non-circular illustration, but the general 'clarity/instructiveness' conclusion is defined by the reward itself.

full rationale

The core RL formulation is not mathematically circular: TAPR maximizes a combined reward, and the GSM8K evaluation uses a verifiable last-integer accuracy signal that is external to the judge. The paper also reports results with alternative task LLMs and win-rate comparisons, which provide independent content. However, two evaluation-design choices create partial circularity. First, GPT-4o-mini is used both as one of the two training reward judges and as the sole final evaluation judge for NQ and HotpotQA, so the headline QA improvements are partly optimized against the evaluation oracle rather than independently verified. Second, the prompt-quality reward defines 'good' prompts as those an LLM judge rates as following prompt-engineering principles, and the paper then presents the observation that TAPR prompts are 'clearer' and 'more consistent with prompt engineering principles' as a finding; this is self-referential. I found no load-bearing self-citation chain: references to PRewrite, GRPO, and related work are external baselines or methods. Because the central claim is only partially reduced to its own evaluation and reward definitions, a moderate score is appropriate rather than a full circularity finding.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 0 invented entities

The central claims rest on a large set of hand-chosen hyperparameters and on trusting LLM judges: reward weights, GRPO sampling and clipping, selection temperature, stopping rule, judge identity, and evaluation subset size. The only fully external signal is GSM8K last-integer accuracy; NQ and summarization results, plus the prompt-quality observations, all pass through the same LLM-judge paradigm used in training. No new physical or formal entities are introduced, but the 'prompt-quality reward' functions as an invented evaluation objective that is partly self-confirming.

free parameters (8)
  • Reward weights alpha, beta = 1 (equal weights)
    Combined reward R = alpha*T + beta*Q; no sensitivity analysis or tuning grid is reported, yet this balance drives training behavior.
  • GRPO learning rate = 1e-5
    Picked to converge without losing capabilities; only a single run is reported, so the GRPO stability claim depends on this choice.
  • GRPO sampling configuration = 4 completions per prompt, T=1.0, top_k=40, p=0.95
    Exploration parameters from Appendix B; no ablation isolates their effect.
  • GRPO clipping bounds and iterations = epsilon_low=0.2, epsilon_high=0.28, 2 iterations
    Stability hyperparameters chosen for the setup; no variations tested.
  • Selection mechanism configuration = 5 candidates at T=0.5 (3 for full-prompt rewriting)
    Chosen to balance diversity and cost; the paper finds selection gains are inconsistent.
  • Stopping criterion = no improvement in 25-step moving average for 100 steps
    Defines convergence; because each run is done once, the resulting checkpoint is a single draw.
  • Evaluation subset size = 1000 validation/test samples per dataset
    No bootstrap confidence intervals are reported around these point estimates.
  • Judge model choice = LLaMA-3.1-8B-Instruct + GPT-4o-mini averaged in training; GPT-4o-mini alone at evaluation
    Averaging two judges is assumed to reduce bias, but no judge-agreement analysis is provided.
axioms (6)
  • domain assumption LLM-as-a-judge scores are valid measures of downstream answer quality.
    Training and evaluation rewards rely on judge outputs; the paper documents position bias but asserts near-perfect judge accuracy on NQ based on an internal 100-sample test. Sections 3.2 and 6.3.
  • domain assumption Averaging two judge models removes enough bias for the reward signal to be useful.
    The authors average LLaMA-3.1-8B-Instruct and GPT-4o-mini scores but do not report their disagreement rate. Section 3.2.
  • ad hoc to paper The prompt-quality reward from an LLM judge reflects genuine prompt-engineering quality.
    The reward is introduced in Section 3.2 and asks a judge to score clarity and adherence to prompt-engineering principles, so 'clearer/more instructive' observations are partly defined by the same judge.
  • domain assumption GRPO optimizes the stochastic composite reward without reward hacking or degenerate outputs.
    The paper uses JSON constraints and Dr. GRPO loss to mitigate this, but PPO variants collapsed (Table 11), so algorithm stability is empirical rather than guaranteed.
  • domain assumption The chosen baseline prompts ('Answer the question', 'Summarize the text', 'SOLUTION') are fair initial prompts for evaluating rewriting gains.
    All reported improvements are relative to these particular prompts; a different baseline set could shift conclusions. Table 1.
  • domain assumption Single-run results with no seeds support the generalization claims.
    Section 4 states each training and evaluation is conducted only once; observed differences could be noise without variance estimates.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter." pith.science (2026). https://pith.science/paper/7G2KCMHD

@misc{pith2026260728657,
  author       = {Pith},
  title        = {Pith review of: TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7G2KCMHD}},
  note         = {Machine review of arXiv:2607.28657}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) often require carefully crafted prompts to unlock their full potential, which can be a barrier for non-expert users. This work addresses the challenge by introducing a Task-Aware Prompt Rewriter (TAPR), a model that reformulates user prompts into task-optimized prompts with the explicit goal of improving downstream LLM performance. We train TAPR using reinforcement learning with Group Relative Policy Optimization (GRPO), where rewards are derived from LLM-as-judge evaluations of both the reformulated prompt and the corresponding task output. Experimental results on diverse tasks, such as question answering, summarization, and arithmetic reasoning, show that our method yields consistent gains over base models in prompt rewriting ability. Fine-tuning Phi-4-mini-instruct (as the base model for TAPR) produces prompts that contain clearer and more instructive language, leading to higher accuracy on established benchmarks such as Natural Questions and GSM8K. Our code is available at: https://github.com/OliverSavolainen/task-specific-prompt-rewriter

Figures

Figures reproduced from arXiv: 2607.28657 by Emanuele Bastianelli, Hosein Azarbonyad, Oliver Savolainen.

Figure 1
Figure 1. Figure 1: Overview of the training pipeline for TAPR. Orange nodes are prompts (inputs to models), blue nodes represent models (trainable or frozen), and green nodes are outputs—either the task answer or the scalar reward. Solid arrows show the forward data flow; the green reward is fed back via GRPO (curved arrow). (prompt, response) pairs [8]. Although SFT provides strong initialization for instruction following, … view at source ↗
Figure 2
Figure 2. Figure 2: Training rewards comparison for the TAPR on the NQ dataset. The upper panel shows the raw per-step reward (blue) and its 25-step moving average or running mean (orange) with our method, and the bottom panel shows the training without the prompt quality reward [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Training rewards comparison for the TAPR on the GSM8K dataset. The upper panel shows the raw per-step reward (blue) and its 25-step moving average or running mean (orange) with our method, and the bottom panel shows the training without the prompt quality reward. 6. Discussion 6.1. Performance Discussion Our results indicate that, although the TAPR method can produce improved prompt rewrites and boost perf… view at source ↗
Figure 4
Figure 4. Figure 4: Cross-dataset generalization of TAPR. Each cell reports the accuracy when Phi-4-mini-instruct is trained on the dataset that the row is named after, and evaluated on the dataset that the column is named after. Summarization scores remain on a 1–5 scale, while QA and GSM8K are percentages. Cell shading encodes change relative to the base prompt (top row), with green indicating higher accuracy and red indica… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 1 canonical work pages

  1. [1]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in Neural Information Processing Systems 33 (2020)

    2020

  2. [2]

    J. Wei, Y. Tay, R. Bommasani,et al., Emergent abilities of large language models, arXiv preprint arXiv:2206.07682 (2022). URL: https://arxiv.org/abs/2206.07682.arXiv:2206.07682

    Pith/arXiv arXiv 2022

  3. [3]

    Srivastava, A

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. San- toro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz,et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022. URL: https: //arxiv.org/abs/2206.04615.arXiv:2206.04615

    Pith/arXiv arXiv 2022

  4. [4]

    T. Shin, Y. Razeghi, R. L. L. IV, E. Wallace, S. Singh, Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. URL: https://arxiv.org/abs/2010. 15980.arXiv:2010.15980

    Pith/arXiv arXiv 2020

  5. [5]

    W. Kong, S. A. Hombaiah, M. Zhang, Q. Mei, M. Bendersky, Prewrite: Prompt rewriting with reinforcement learning, 2024. URL: https://arxiv.org/abs/2401.08189.arXiv:2401.08189

    Pith/arXiv arXiv 2024

  6. [6]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms,

  7. [7]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, D. Guo, Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL: https://arxiv.org/abs/2402.03300.arXiv:2402.03300

    Pith/arXiv arXiv 2024

  8. [8]

    J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le, Finetuned language models are zero-shot learners, in: Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022

    2022

  9. [9]

    Christiano, J

    P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, D. Amodei, Deep reinforcement learning from human preferences, in: Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017, pp. 4299–4307

    2017

  10. [10]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, R. Lowe, Training language models to follow instructions with human feedback, in: Advances in Neural Information Processing Systems 35 (NeurI...

    2022

  11. [11]

    DeepSeek-AI, Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

  12. [12]

    B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, J. Han, Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URL: https://arxiv.org/abs/ 2503.09516.arXiv:2503.09516

    Pith/arXiv arXiv 2025

  13. [13]

    Boonstra, Prompt engineering, Google 5-Day GenAI Intensive Whitepapers-to-Text, Kaggle, 2025

    L. Boonstra, Prompt engineering, Google 5-Day GenAI Intensive Whitepapers-to-Text, Kaggle, 2025. https://www.kaggle.com/code/toddgardiner/google-5-day-genai-intensive-whitepapers-to-text/ output

    2025

  14. [14]

    Kojima, S

    T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, Y. Iwasawa, Large language models are zero-shot reasoners,

  15. [15]

    Wei, et al., Chain-of-thought prompting elicits reasoning in large language models, arXiv preprint arXiv:2201.11903 (2022)

    J. Wei, et al., Chain-of-thought prompting elicits reasoning in large language models, arXiv preprint arXiv:2201.11903 (2022)

    Pith/arXiv arXiv 2022

  16. [16]

    T. Shen, T. Lei, R. Barzilay, T. Jaakkola, Style transfer from non-parallel text by cross-alignment, in: Advances in Neural Information Processing Systems, 2017

    2017

  17. [17]

    Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, E. P. Xing, Toward controlled generation of text, in: Proceedings of the 34th International Conference on Machine Learning (ICML), 2017

    2017

  18. [18]

    D. X. Long, H. N. Ngoc, T. Sim, H. Dao, S. Joty, K. Kawaguchi, N. F. Chen, M.-Y. Kan, Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms, 2024. URL: https://arxiv.org/abs/2408.08656.arXiv:2408.08656

    Pith/arXiv arXiv 2024

  19. [19]

    Fernando, D

    C. Fernando, D. S. Banarse, H. Michalewski, S. Osindero, T. Rocktäschel, Promptbreeder: Self- referential self-improvement via prompt evolution, in: Proceedings of the 41st International Conference on Machine Learning (ICML), 2024, pp. 13481–13544

    2024

  20. [20]

    M. Deng, J. Wang, C.-P. Hsieh, Y. Wang, H. Guo, T. Shu, M. Song, E. P. Xing, Z. Hu, Rlprompt: Optimizing discrete text prompts with reinforcement learning, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022, pp. 3369–3391

    2022

  21. [21]

    Zhang, X

    T. Zhang, X. Wang, D. Zhou, D. Schuurmans, J. E. Gonzalez, Tempera: Test-time prompting via reinforcement learning, 2022. URL: https://arxiv.org/abs/2211.11890.arXiv:2211.11890

    Pith/arXiv arXiv 2022

  22. [22]

    URL: https://doi.org/10.48550/ arXiv.2305.10403, arXiv preprint arXiv:2305.10403

    Google, Palm 2 technical report, PaLM 2 Technical Report, 2023. URL: https://doi.org/10.48550/ arXiv.2305.10403, arXiv preprint arXiv:2305.10403

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.10403 2023

  23. [23]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, 2002

    2002

  24. [24]

    Lin, Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, ACL Workshop, 2004, pp

    C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, ACL Workshop, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/

    2004

  25. [25]

    Banerjee, A

    S. Banerjee, A. Lavie, Meteor: An automatic metric for mt evaluation with improved correlation with human judgments, in: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, Michigan, 2005, pp. 65–72

    2005

  26. [26]

    Zhang, V

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, 2020. URL: https://arxiv.org/abs/1904.09675.arXiv:1904.09675

    Pith/arXiv arXiv 2020

  27. [27]

    Saadany, C

    H. Saadany, C. Orăsan, Bleu, meteor, bertscore: Evaluation of metrics performance in assess- ing critical translation errors in sentiment-oriented text, in: Proceedings of the Translation and Interpreting Technology Online Conference TRITON 2021, TRITON 2021, INCOMA Ltd. Shoumen, BULGARIA, 2021, p. 48–56. URL: http://dx.doi.org/10.26615/978-954-452-071-7_...

    work page doi:10.26615/978-954-452-071-7_006 2021

  28. [28]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, et al., Judging llm-as-a-judge with mt-bench and chatbot arena, in: NeurIPS, 2023

    2023

  29. [29]

    J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Y. Wang, W. Gao, L. Ni, J. Guo, A survey on llm-as-a-judge, arXiv preprint arXiv:2411.15594 (2024)

    Pith/arXiv arXiv 2024

  30. [30]

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, S. Prakash, Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback,

  31. [31]

    Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, et al., Constitutional ai: Harmlessness from ai feedback, arXiv preprint arXiv:2212.08073 (2022)

    Pith/arXiv arXiv 2022

  32. [32]

    Kwiatkowski, J

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, S. Petrov, Natural questions: A benchmark for question answering research, Transactions of the Association for Computational Linguistics 7 (2019) 452–466....

    doi:10.1162/tacl_a_00276 2019

  33. [33]

    Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, C. D. Manning, HotpotQA: A dataset for diverse, explainable multi-hop question answering, in: E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, B...

    doi:10.18653/v1/d18-1259 2018

  34. [34]

    K. M. Hermann, T. Kočiský, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, P. Blun- som, Teaching machines to read and comprehend, 2015. URL: https://arxiv.org/abs/1506.03340. arXiv:1506.03340

    Pith/arXiv arXiv 2015

  35. [35]

    Cachola, K

    I. Cachola, K. Lo, A. Cohan, D. Weld, TLDR: Extreme summarization of scientific documents, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 4766–4777. URL: https: //aclanthology.org/2020.findings-emnlp.428/. doi:10.18653/v1/2020.findings-emnlp.428

    doi:10.18653/v1/2020.findings-emnlp.428 2020

  36. [36]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, J. Schulman, Training verifiers to solve math word problems, 2021. URL: https://arxiv.org/abs/2110.14168. doi:10.48550/arXiv.2110.14168.arXiv:2110.14168

    Pith/arXiv arXiv doi:10.48550/arxiv.2110.14168 2021

  37. [37]

    W. Li, X. Wang, W. Li, B. Jin, A survey of automatic prompt engineering: An optimization perspective, arXiv preprint arXiv:2502.11560 (2025). URL: https://arxiv.org/abs/2502.11560

    Pith/arXiv arXiv 2025

  38. [38]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, C. Finn, Direct preference optimization: Your language model is secretly a reward model, 2024. URL: https://arxiv.org/abs/2305.18290. arXiv:2305.18290

    Pith/arXiv arXiv 2024

  39. [39]

    AI, The llama 3 herd of models, 2024

    M. AI, The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783

    Pith/arXiv arXiv 2024

  40. [40]

    Microsoft Azure AI Services Blog, Introducing phi-4: Microsoft’s newest small language model specializing in complex reasoning, 2025. URL: https:// techcommunity.microsoft.com/blog/aiplatformblog/introducing-phi-4-microsoft%E2%80% 99s-newest-small-language-model-specializing-in-comple/4357090, microsoft Tech Community Blog

    arXiv 2025

  41. [41]

    URL: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, meta AI Blog

    Meta AI, Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2024. URL: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/, meta AI Blog

    2024

  42. [42]

    Team, Qwen3 technical report, 2025

    Q. Team, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388

    Pith/arXiv arXiv 2025

  43. [43]

    URL: https://huggingface.co/ datasets/data-is-better-together/10k_prompts_ranked, accessed: 2025-06-19

    Data-is-Better-Together Collective, 10k prompts ranked, 2024. URL: https://huggingface.co/ datasets/data-is-better-together/10k_prompts_ranked, accessed: 2025-06-19

    2024

  44. [44]

    Kundu, Prompt optimization dataset, 2025

    R. Kundu, Prompt optimization dataset, 2025. URL: https://huggingface.co/datasets/rishavkundu/ prompt_optimization_dataset, accessed: 2025-06-19

    2025

  45. [45]

    Accessed: 2025-06-19

    OpenAI, GPT-4o mini: advancing cost-efficient intelligence, https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/, 2024. Accessed: 2025-06-19

    2024

  46. [46]

    Hugging Face, Trl: Transformer reinforcement learning, https://huggingface.co/docs/trl/en/index,

  47. [47]

    explanation

    Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, M. Lin, Understanding r1-zero-like training: A critical perspective, 2025. URL: https://arxiv.org/abs/2503.20783.arXiv:2503.20783. Acknowledgments Thanks to the developers of ACM consolidated LaTeX styles https://github.com/borisveytsman/acmart and to the developers of Elsevier updated LATEX templa...

    Pith/arXiv arXiv 2025

  48. [51]

    Published January 28, 2025; Accessed: 2025-06-19

    2025

  49. [53]

    explanation

    Original instruction: The old instruction New Instruction: {"explanation": "Explanation of why this kind of rewriting was done", " final_rewritten_query": "The rewritten instruction."} Now write the new instruction. Respond only with valid JSON. Do not write an introduction or summary. Original instruction: "Answer the question" New Instruction: { A.2. LL...

  50. [54]

    Match at least one reference answer exactly or via an unambiguous synonym

  51. [55]

    Ignore extra, irrelevant detail; focus on whether the core answer is present

  52. [56]

    explanation

    Output 1 for correct, 0 for incorrect. Return a JSON object: { "explanation": "...short justification...", "score": 1 | 0 } Examples [QUERY] : Who wrote "Pride and Prejudice"? [RESPONSE] : Jane Austen wrote "Pride and Prejudice" in 1813. [REFERENCE] : ["Jane Austen"] { "explanation": "Mentions Jane Austen, which matches the reference exactly.", "score": 1...

  53. [2017]

    URL: https://arxiv.org/abs/1707.06347.arXiv:1707.06347

    Pith/arXiv arXiv

  54. [2023]

    URL: https://arxiv.org/abs/2205.11916.arXiv:2205.11916

    Pith/arXiv arXiv

  55. [2024]

    URL: https://arxiv.org/abs/2309.00267.arXiv:2309.00267

    Pith/arXiv arXiv

  56. [2025]

    URL: https://arxiv.org/abs/2501.12948.arXiv:2501.12948

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.