Pith. sign in

REVIEW 3 major objections 5 minor 39 references

PLHF: Prompt Optimization with Few-Shot Human Feedback

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A few human scores can train an evaluator that beats GPT-4o for prompt optimization.

desk verdict Clean two-module idea for prompt optimization without a metric, but public-dataset evidence rests on an unvalidated GPT-4o pseudo-judge; the industrial result is the only solid proof. read the letter →

arxiv 2505.07886 v1 pith:QCP2XNI7 submitted 2025-05-11 cs.CL cs.AI

Chun-Pai Yang , Kan Zheng , Shou-De Lin This is my paper · ORCID
classification cs.CLcs.AI
keywords promptoptimizationhumanfeedbackLLMevaluatorfew-shotlearningRLHFoutputqualitygradingDSPyTextGrad
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PLHF is a framework for optimizing prompts when no fixed metric defines output quality, such as joke generation, essay scoring, or dialogue satisfaction. It asks human experts to grade a small set of input–output pairs once, then prompt-optimizes a separate evaluator LLM to reproduce those judgments. That evaluator is then used as the metric for optimizing the responder's prompt. Across four datasets, including an industrial SQL question-answering product, PLHF reports the best output quality compared with using GPT-4o, exact matching, or embedding similarity as the grading function, under both DSPy and TextGrad prompt optimizers. The practical claim is that a small GPT-3.5 evaluator tuned on a few human labels can outperform a state-of-the-art general LLM judge at steering prompt optimization.

What carries the argument

The central mechanism is the evaluator module, a prompt-optimized LLM that acts as a learned grading function for the responder module. The paper builds the evaluator as an auxiliary supervised task: with a trivial metric such as Accuracy or Mean Absolute Error, an existing prompt optimizer (DSPy or TextGrad) tunes the evaluator's prompt to match a handful of human scores. The same optimizer then maximizes the evaluator's scores while searching for the responder's prompt. This makes the missing reward signal measurable in a closed loop, inspired by RLHF but with the reward model's prompt optimized on few human labels rather than its weights trained at scale.

What would settle it

Collect real human ratings on a fresh batch of outputs produced by PLHF-optimized and GPT-4o-optimized responders, then compare which set humans prefer; if humans prefer the GPT-4o-optimized outputs on any public dataset, the paper's central claim for that dataset fails. A cheaper check is to measure human–pseudo-human agreement on a held-out sample of responder outputs before trusting the reported public-dataset scores.

Watch

Extended reading notes

Core claim

The paper's central claim is that PLHF, a duo-module design, lets prompt optimization proceed without a well-defined metric by first learning the metric itself. Given a small labeled set of (input, output, score) triplets scored by human experts, PLHF optimizes the prompt of an evaluator LLM so that its scores match the human labels on a conventional loss such as Accuracy or Mean Absolute Error. Once the evaluator is prompt-optimized, PLHF uses it as the grading function to optimize the responder's prompt. The paper reports that PLHF achieves the best output quality on all four datasets, beating prompt optimization with GPT-4o as the evaluator even though PLHF's evaluator is built on GPT-3.5, and beating exact-matching and embedding-similarity grading.

Load-bearing premise

The load-bearing premise is that the pseudo-human judge — GPT-4o with a prompt optimized via DSPy — agrees with real human preferences on the newly generated outputs of the public datasets; the paper never checks this agreement, even though its own motivation is that pre-trained LLM judges diverge from humans.

Editorial extensions

If this is right

  • Prompt optimization can be run end-to-end with a single round of human scoring, removing the repeated expert queries that block iterative frameworks.
  • A comparatively weak base model with a tuned prompt can serve as a better task-specific judge than a frontier model used off-the-shelf.
  • The evaluator module turns the human preference signal into a reusable artifact, so subsequent responder updates need no further human calls.
  • Exact-matching and embedding-similarity grading are consistently worse than learned evaluators, suggesting they are poor reward proxies for open-ended generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported public-dataset gains depend on the pseudo-human judge agreeing with real human raters; replacing GPT-4o with true human labels on newly generated outputs could reorder the results.
  • PLHF could be combined with active sample selection, because the human labels are the bottleneck and choosing which input–output pairs to grade could further cut the label budget.
  • A testable extension is to take an evaluator tuned on one dataset and use it unchanged to score outputs of a related task, which the paper does not examine.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PLHF, a prompt-optimization framework with two modules: a responder R that generates task outputs and an evaluator E that grades those outputs. E is a base LLM (GPT-3.5) whose prompt is optimized, via DSPy or TextGrad, on a small set of human-labeled (input, output, score) triplets; R's prompt is then optimized with the same PO framework using E as the metric. The authors claim that this requires only a single round of human feedback and that PLHF outperforms prior output-grading strategies, including using GPT-4o as an evaluator, on three public datasets (SGD, AES-ASAP, AES-2.0) and one industrial SQL-QA dataset. The industrial SQL-QA results are judged by real human experts; the public-dataset results are judged by a 'pseudo-human judge' (prompt-optimized GPT-4o).

Significance. The problem addressed—optimizing prompts for generative tasks without a golden metric and with scarce human feedback—is timely and practically important. The two-module architecture is simple and modular, and it can wrap existing PO frameworks. The industrial deployment with real expert labels provides positive evidence that the framework can work in a product setting. However, the public-dataset evidence does not currently establish the central claim, because the pseudo-human judge is an unvalidated LLM and the paper's own motivation is that LLM judges misalign with humans. If the authors add a human-agreement validation or real human evaluation for the public datasets, the paper could make a solid contribution.

major comments (3)
  1. [Section 3.4, Table 2, Appendix D] The central claim in Section 3.4 that PLHF achieved the best performance 'for all the four datasets' rests on the public-dataset rows of Table 2, but those rows are scored by a pseudo-human judge that is a prompt-optimized GPT-4o model. Section 3.3 states: 'we use GPT-4o with prompt optimizations via DSPy as the pseudo-human judge.' The paper provides no validation that this judge agrees with the actual human raters on the newly generated outputs, even though the original human raters are unavailable. This is not a minor concern: Figure 1 motivates PLHF precisely by showing that GPT-4o's verdicts can disagree with human preferences, and the responder in PLHF is optimized against a GPT-3.5 evaluator trained on only a few human labels. The observed gains on SGD and AES-ASAP, for example +8.71% and +8.68% with TextGrad, could reflect reward hacking toward shared LLM biases rather than genuine improvement in human-perceived output quality. I request a human-agreement study (e.g., correlation and agreement metrics between the pseudo-human judge and human raters on a held-out sample of generated outputs), or a re-evaluation with real human labels for at least the main comparisons. Without this, the cross-dataset generalization claim is unsupported for three of the four datasets.
  2. [Section 3.4, Table 2, Appendix D] Table 2 reports only point estimates (relative improvements) for the responder task, with no error bars, confidence intervals, or significance tests. The only variance information appears in Appendix D, and it is limited to PLHF on SGD and AES-ASAP; no baseline variance is reported. For AES-2.0, the advantage of PLHF over 'PO with GPT-4o' is +2.62% (DSPy) and +4.30% (TextGrad), which may be within noise. To support the claim that PLHF 'outperforms' the baselines, the authors should report multiple independent runs, standard deviations or confidence intervals, and paired significance tests for all methods and datasets.
  3. [Abstract, Section 2, Algorithm 1] The paper's contribution statement in the abstract claims PLHF 'requires only a single round of human feedback to complete the entire prompt optimization process,' but Algorithm 1's outer loop is 'while there are new training samples added into D,' which suggests the framework can consume additional human feedback over time. Section 2 also says 'human grading no greater than linear (with respect to the number of training samples) times.' These statements are not mutually consistent. Please clarify whether the intended setting is a single, fixed training set D or an iterative data-collection process, and state precisely how many human labels are used in the main experiments for each dataset.
minor comments (5)
  1. [Section 3.1.2] The abstract contains the typo 'näive' (should be 'naive').
  2. [Section 3.1.2] The sentence in Section 3.1.2, 'The dataset, named as AES-2.0, contains 24,000 student-written argumentative essays. Each essay was scored on a scale of 1 to 6 as the holistic rating 1 judged by human experts,' has an awkward and unclear phrasing ('as the holistic rating 1 judged by human experts').
  3. [References] The reference list entry for the ASAP dataset is malformed: 'Hamner Ben, Morgan Jaison, lynnvandev, Shermis Mark, and Ark Tom Vander. 2012. The hewlett foundation: Automated essay scoring.' The author names are inconsistently ordered and some appear to be GitHub usernames; please fix the citation.
  4. [Figure 2] Figure 2's caption contains a garbled phrase: 'where PO can be PO arbitrary prompt optimization method.' This should read 'where PO can be an arbitrary prompt optimization method.'
  5. [Section 3.2, Appendix D] For the public datasets, the paper does not state how many training samples (human-labeled triplets) are used in the main experiments; Appendix D sweeps the number of samples up to 100, but the relationship between those curves and the numbers in Table 2 is unclear. Please state the exact sample sizes and the positive-sample thresholds (e.g., 'Labeled Score ≥ 8' in the toy example) for each dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PLHF's reported gains come from an external pseudo-human judge, not from re-using its own fitted evaluator.

full rationale

The derivation chain is linear and externally grounded: (i) a few human-labeled triplets are used to prompt-optimize an evaluator E via a conventional loss L (RMSE/accuracy); (ii) E's scores are used as the reward signal to prompt-optimize the responder R; (iii) final outputs are scored by a separate pseudo-human judge (GPT-4o with DSPy) for public datasets and by real human experts for SQL-QA. No step sets the reported gain equal to the fitted quantity. E is fitted to human labels, R is fitted to E, and the reported numbers are produced by a different judge (GPT-4o or humans); the pseudo-human judge is not the same function as E and is not trained on PLHF's outputs, so the superiority in Table 2 does not hold by construction. The pseudo-human judge is an unvalidated surrogate — the paper does not show it agrees with the original human raters on newly generated outputs, and Fig. 1 itself warns that LLM judges can disagree with humans — but that is an external-validity weakness, not a circularity. Moreover, if anything, the GPT-4o-based judge should favor the GPT-4o-evaluator baseline, yet PLHF wins, which is evidence the result is not forced. DSPy and TextGrad are external PO frameworks; no load-bearing self-citation or imported uniqueness theorem appears. Score 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

PLHF introduces no new physical or ontological entities; it is a software framework composed of two existing LLM modules. The main unstated inputs are the positivity threshold and the sample counts, which are free choices that affect the results. The most consequential axiom is that the GPT-4o pseudo-human judge is a valid stand-in for human preference, which the paper does not verify.

free parameters (2)
  • Positive sample threshold
    Algorithm 1 uses training samples with 'positive rating ri', but the threshold for positivity is not specified for any dataset. Appendix B uses 'Labeled Score >= 8' as the condition. This choice determines which samples train the responder and can materially affect the final prompt and the reported performance.
  • Number of training samples
    The main tables do not state how many labeled samples are used for the reported results. Appendix D shows curves for 5 to 100 samples, but the exact number used in Tables 1 and 2 is not given. This is an experimental design choice that affects all numbers.
assumptions (3)
  • domain assumption The human-labeled scores in the SGD, AES-ASAP, AES-2.0, and SQL-QA datasets accurately reflect the true output quality for those tasks.
    The entire framework assumes these labels are reliable enough to train an evaluator that then guides prompt optimization. If the labels are noisy or biased, the evaluator inherits those biases.
  • domain assumption A prompt-optimized GPT-3.5 evaluator trained on a few labeled samples generalizes accurately enough to grade unseen outputs and provide a valid training signal for the responder.
    Section 2.1 assumes the evaluator E, with an optimized prompt, can estimate output quality for the responder. If the evaluator overfits to the few labels or fails to generalize, the responder prompt optimization will be misguided.
  • domain assumption GPT-4o with DSPy prompt optimization is a valid pseudo-human judge for evaluating generated outputs on the public datasets.
    Section 3.3 introduces the pseudo-human judge without validating that its scores agree with real human raters on the newly generated outputs. Since the paper's motivation is that LLM judges can diverge from human preferences, this assumption is load-bearing for the reported empirical claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PLHF: Prompt Optimization with Few-Shot Human Feedback." pith.science (2026). https://pith.science/paper/QCP2XNI7

@misc{pith2026250507886,
  author       = {Pith},
  title        = {Pith review of: PLHF: Prompt Optimization with Few-Shot Human Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QCP2XNI7}},
  note         = {Machine review of arXiv:2505.07886}
}
read the original abstract

Automatic prompt optimization frameworks are developed to obtain suitable prompts for large language models (LLMs) with respect to desired output quality metrics. Although existing approaches can handle conventional tasks such as fixed-solution question answering, defining the metric becomes complicated when the output quality cannot be easily assessed by comparisons with standard golden samples. Consequently, optimizing the prompts effectively and efficiently without a clear metric becomes a critical challenge. To address the issue, we present PLHF (which stands for "P"rompt "L"earning with "H"uman "F"eedback), a few-shot prompt optimization framework inspired by the well-known RLHF technique. Different from naive strategies, PLHF employs a specific evaluator module acting as the metric to estimate the output quality. PLHF requires only a single round of human feedback to complete the entire prompt optimization process. Empirical results on both public and industrial datasets show that PLHF outperforms prior output grading strategies for LLM prompt optimizations.

Figures

Figures reproduced from arXiv: 2505.07886 by the authors.

Figure 1
Figure 1. Demonstrations of the actual failure cases that the evaluations from pre-trained LLMs have different [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Workflow framework of PLHF. The entire LLM program contains two modules, Responder R and Evaluator E, where PO can be PO arbitrary prompt optimization method. pert inquiries acting as the metric, once the prompt is updated with any modifications, we have to ask human experts for their judgment repeatedly for each sample — this causes serious efficiency bot￾tleneck. To address the aforementioned issues, we present PL… view at source ↗
Figure 3
Figure 3. A toy example to illustrate the subtask designs of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance curves of PLHF on the datasets SGD and AES-ASAP. For the plots, we consider DSPy as the PO method for PLHF. The x-values are the number of (randomly selected) training samples. The y-values are mean values of the RMSE losses for E and the output scores for …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 12 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Hamner Ben, Morgan Jaison, lynnvandev, Shermis Mark, and Ark Tom Vander. 2012. https://kaggle.com/competitions/asap-aes The hewlett foundation: Automated essay scoring

    work page 2012

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  4. [4]

    Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1--27

    work page 2011

  5. [5]

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369--3391

    work page 2022

  6. [6]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. 2023. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797

    arXiv 2023

  7. [7]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556--6576

    2024

  8. [8]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816--3830

    2021

Show all 39 references
  1. [9]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2023. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532

    2023 arXiv

  2. [10]

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2024. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems, 36

    2024

  3. [11]

    Kaggle. 2024. https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/ Automated essay scoring 2.0

    2024

  4. [12]

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In The Twelfth Internatio...

    2024

  5. [13]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations

    2023

  6. [14]

    Zihao Li, Zhuoran Yang, and Mengdi Wang. 2023. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438

    2023 arXiv

  7. [15]

    Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology

    1932

  8. [16]

    Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low. 2024. Prompt optimization with human feedback. arXiv preprint arXiv:2405.17346

    2024 arXiv

  9. [17]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  10. [18]

    Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730

    2018 arXiv

  11. [19]

    OpenAI. 2024. https://platform.openai.com/docs/guides/embeddings/embedding-models Openai platform: Embeddings --- embedding models

    2024

  12. [20]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

    2022

  13. [21]

    Marius-Constantin Popescu, Valentina E Balas, Liliana Perescu-Popescu, and Nikos Mastorakis. 2009. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems, 8(7):579--588

    2009

  14. [22]

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687--5711

    2023

  15. [23]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957--7968

    2023

  16. [24]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/index/language-unsupervised/ Improving language understanding by generative pre-training

    2018

  17. [25]

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), ...

    2020

  18. [26]

    Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. 2021. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. In Proceedings of the 44rd International ACM SIGIR Conference on Research and Develop...

    2021

  19. [27]

    Ting Fang Tan, Kabilan Elangovan, Liyuan Jin, Yao Jie, Li Yong, Joshua Lim, Stanley Poh, Wei Yan Ng, Daniel Lim, Yuhe Ke, et al. 2024. Fine-tuning large language model (llm) artificial intelligence chatbots in ophthalmology and llm-based evaluation using gpt-4. arXiv preprint ...

    2024 arXiv

  20. [28]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In Proceedings of the 30th ACM joint European software engineering conference and symposium on the...

    2022

  21. [29]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048

    2023 arXiv

  22. [30]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36

    2024

  23. [31]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    2024

  24. [32]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations

    2023

  25. [33]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. https://arxiv.org/abs/2406.07496 Textgrad: Automatic "differentiation" via text

    2024 arXiv

  26. [34]

    Gonzalez

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2023. https://openreview.net/forum?id=gSHyqBijPFO TEMPERA : Test-time prompt editing via reinforcement learning . In The Eleventh International Conference on Learning Representations

    2023

  27. [35]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  28. [36]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. https://arxiv.org/abs/2211.01910 Large language models are human-level prompt engineers . Preprint, arXiv:2211.01910

    2023 arXiv

  29. [37]

    Rubens Zimbres. 2024. https://medium.com/google-cloud/evaluating-llms-with-langchain-using-gpt-4-to-evaluate-googles-open-model-gemma-2b-it-eb7555e3bdeb Evaluating llms with langchain: Using gpt-4 to evaluate google’s open model gemma-2b-it

    2024

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.