Pith. sign in

REVIEW 4 major objections 5 minor 33 references

SEO: Stochastic Experience Optimization for Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An LLM can search over natural-language rule lists, keep only edits that pass a fresh 50-example check, and thereby improve its own task performance without any parameter update.

desk verdict Useful new twist on prompt optimization with an honest ablation, but the headline claim outruns the statistics. read the letter →

arxiv 2501.04393 v1 pith:UJAGU4JM submitted 2025-01-08 cs.CL

Jitao Xu , Hongyun Zhou , Lei Shen , Conghui Zhu , Jin Huang , Yitao Duan This is my paper · ORCID
classification cs.CL
keywords stochasticexperienceoptimizationLLMinference-timeadaptationnatural-languagerulepromptvalidationmodel-specificmulti-hopquestionansweringmachinetranslation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEO is an iterative procedure for finding a short list of natural-language rules—an 'experience'—that makes a given large language model perform better at a given task, without changing the model's weights. The paper claims that the resulting experiences reliably beat both the no-experience baseline and the initial experience across three tasks, three generator LLMs, and seven datasets. The authors argue that the method works because every candidate rewrite is screened by a stochastic validation set before it is kept, giving each optimization step a concrete direction rather than relying on the optimizer's guess about what should help. If correct, this makes inference-time prompting a tunable asset, so frozen or API-only models can be adapted to a task by growing a better instruction list.

What carries the argument

The load-bearing object is the 'experience': a compact natural-language rule list inserted into the generator model's prompt. The mechanism that carries the argument is the stochastic validation step, which simulates a descent direction for an optimization problem where no true gradient exists: a candidate experience is accepted only if its score on a freshly resampled 50-example validation set exceeds the current experience's score on that same set, and the training loop otherwise keeps the old experience. This acceptance rule is what distinguishes SEO from earlier reflection-based experience learners, which update experiences without checking whether the update actually helps.

What would settle it

Run the same SEO loop on one task and record, at every accepted update, whether the held-out dev-set score also improves; if accepted updates regularly lower the dev score or perform no better than randomly selected candidate experiences on the held-out test set, the 50-example stochastic validation is not identifying a reliable update direction.

Watch

Extended reading notes

Core claim

The paper's central claim is that an 'experience'—a list of roughly ten general rules written in natural language—can be treated as the analogue of model parameters and optimized for a specific LLM on a specific task. At each step, the generator LLM answers the same training question with and without the current experience, and the optimizer LLM sees the question, the correct answer, both trial outputs, and the score difference $\delta$ between them, then samples $k$ candidate rewrites. Every candidate is scored on a freshly sampled 50-example validation subset of the training data, and the experience moves forward only if the best candidate beats the current experience's score on that subset; otherwise the old experience is kept. The authors report that this loop improves exact-match accuracy on HotpotQA, COMET scores across four WMT23 translation directions, and accuracy on SST-2 and CoLA for GPT-3.5, Llama-2-13b, and Llama-2-7b, and that the optimized experience transfers to an out-of-distribution multi-hop QA dataset and across translation directions.

Load-bearing premise

The method's steering depends on a randomly sampled 50-example validation set from the training data being a trustworthy proxy for what will help on the held-out test distribution.

Editorial extensions

If this is right

  • On HotpotQA, the SEO-optimized experience raises exact-match accuracy for GPT-3.5 from 37.2 to 45.2, for Llama-2-13b from 19.4 to 29.8, and for Llama-2-7b from 19.0 to 22.6, and it beats both the CLIN and ExpeL baselines.
  • On MT, SEO improves COMET over the direct, no-experience baseline in essentially all model/language-direction combinations; for the weaker Llama-2-7b model it also reduces failure-to-translate cases, for example from 143 to 8 in the En-Zh direction.
  • The optimized experience generalizes out of distribution: experience optimized on HotpotQA lifts exact-match on 2WikiMultiHopQA from 23.0 to 35.4 for GPT-3.5 and from 20.2 to 27.0 for Llama-2-13b.
  • When validation is removed, HotpotQA test accuracy drops from 45.2 to 37.6, and a fixed validation set performs worse than the stochastic one, which the paper reads as evidence that the random resampling and acceptance rule are doing the work.
  • Experience optimized for one translation direction typically improves other directions for Llama-2-7b, and in some cases beats the experience optimized for the target direction itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 50-example validation proxy is as reliable as the paper's results suggest, the same loop could be applied to other prompt attachments—few-shot example selection, chain-of-thought scaffolding, output-format instructions, or safety constraints—treating any text in the context window as a tunable parameter.
  • A testable consequence of the design is that SEO's advantage should shrink as the generator model approaches its own performance ceiling, a pattern already visible in the small MT gains for GPT-3.5; one could measure this directly by running the loop on increasingly strong generators.
  • The cross-lingual MT results imply that some learned rules are language-general; an extension the paper does not run is to test whether a HotpotQA-optimized experience transfers to other reasoning benchmarks, not just 2WikiMultiHopQA, and whether transfer decays with task distance.
  • Because the optimizer LLM is GPT-4 while all generators are weaker, an open question the paper leaves implicit is how much of SEO's success depends on the optimizer being stronger than the generator; a direct test would pair each generator with an optimizer of matched strength.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Stochastic Experience Optimization (SEO), an inference-time procedure that iteratively refines a natural-language 'experience' (a list of rules) for a frozen generator LLM. At each training step, an optimizer LLM samples k candidate updates conditioned on a single training example, its gold answer, the generator's trials with and without the current experience, and the current experience's effect; the candidates are then evaluated on a stochastically resampled, 50-example validation set, and the best candidate that beats the current experience on that set is kept as the next experience. Experiments on HotpotQA, WMT23 machine translation (four directions), and GLUE classification (SST-2, CoLA) with GPT-3.5 and Llama-2-7b/13b report performance improvements over direct prompting and over applying the initial experience, along with ablations, an error analysis, and an out-of-distribution transfer test to 2WikiMultiHopQA.

Significance. If the central claim were statistically established, SEO would be a practical, model-agnostic and task-agnostic inference-time optimization tool: it improves LLM performance without parameter updates, uses only a frozen optimizer LLM, and the described algorithm is clear enough to reimplement. The paper has genuine strengths: Algorithm 1 is explicit, the ablation in Table 8 isolates the validation step as a major contributor, the error analysis in Section 5.2 distinguishes format gains from correctness gains, and the OOD transfer result in Table 7 is a falsifiable prediction that goes beyond same-distribution tuning. However, the empirical evidence is currently under-powered: there are no error bars, no significance tests, no repeated runs, and the reported numbers are selected maxima over many candidate evaluations and over two optimization runs. The one acknowledged exception (En-Zh Llama-2-13b) is precisely the kind of outcome expected if part of the reported gains are selection noise. The method is plausible and worth further investigation, but the headline claim of 'consistently improved performance' is not yet supported by the experiments as presented.

major comments (4)
  1. [Section 4, Tables 1-3] The central claim of consistent improvement is not statistically established. The paper reports no repeated runs, no error bars, and no significance tests. For the MT results in Table 2, many reported gains are 0.1-0.5 COMET points (e.g., GPT-3.5 En-De: SEO 84.23 vs Direct 83.27; De-En: 85.47 vs 85.36), which are likely within metric noise for a 200-example test set. The one acknowledged exception, En-Zh Llama-2-13b (SEO 73.23 vs Direct 73.41), is exactly the kind of result expected under selection noise, and it undermines the abstract's unqualified phrase 'consistently improved performance.'
  2. [Section 2.3, Eq. (4)] The stochastic validation step does not, by itself, guarantee a reliable update direction. With n=3, k=3, and up to 200 training steps, the algorithm compares up to 1,800 candidate scores against the current experience score on 50-example validation sets. For EM and accuracy, a 50-example binary metric has a standard error of roughly 7 points; under the null hypothesis that candidates are equivalent to Et, the maximum of many candidate scores will frequently exceed St by chance. No multiple-comparison correction, calibration, or stability analysis is provided. The statement in Section 2.3 that validation 'make[s] sure that an updated experience actually improves the performance of Mgen on Dt' is only true for the particular sampled Dt, not for the test distribution, and the paper should be revised to replace 'ensure' with a claim that is actually supported.
  3. [Section 2.2, Figure 2] Candidate experiences are generated using the ground-truth answer Y of a single training example, and the optimizer prompt only softly instructs the model to avoid sentence-specific content. The 50-example stochastic validation filter is therefore the only objective defense against single-example overfitting, but its reliability is not independently tested. The paper should report, for example, the correlation between validation-set scores and held-out dev scores for accepted candidates, or the fraction of accepted candidates that fail to generalize to a fresh validation sample. Without such an analysis, it is unclear whether the validation step is selecting for genuine general rules or for experiences that happen to score well on the small sampled set.
  4. [Section 2.3 and Section 3.2] The final reported performance is the result of an additional selection stage: the best experience is chosen from all valid training steps using a 200-example held-out dev set, and the best of two runs (batch size 1 and batch size 3) is selected. The tables therefore report the maximum over many candidate trajectories, not the expected performance of a single SEO run. This inflates the apparent improvement and further justifies the need for repeated runs or a nested evaluation protocol. At minimum, the paper should report the number of valid steps, the dev scores of the selected runs, and the performance variance across runs.
minor comments (5)
  1. [Abstract and Section 4] The abstract states that experiences optimized by SEO 'can achieve consistently improved performance,' but Section 4 acknowledges 'except for one case' and Table 2 shows SEO below Direct for En-Zh Llama-2-13b. Please align the wording (e.g., 'generally' or 'in all but one case').
  2. [Table 8] The difference between the stochastic-validation setting (45.2 test, 51.0 dev) and the fixed-validation setting (44.4 test, 47.0 dev) is small, and the text says fixed validation 'may lead to less effective experience.' Without repeated runs, this conclusion is not supported; please soften the claim or provide variance estimates.
  3. [Figure 3] The matrix in Figure 3 reports differences in COMET scores when an experience optimized for one direction is applied to another direction, but the row and column labels are somewhat confusing as printed. Please make explicit whether rows correspond to the experience's training direction and columns to the tested target direction, and clarify the color scale.
  4. [Section 5.2, Table 5] The error-type analysis reports differences between SEO and Direct (e.g., Delta Correct = 10, 14, 6) without giving the base counts of each error type in the Direct condition. Please include the total number of dev examples and the per-type counts for each method, so the reader can assess the magnitude of these differences.
  5. [Section 3.2] The paper says code and optimized experiences will be released 'upon acceptance.' Given that the experiments depend on specific API model versions (gpt-3.5-turbo-0613, gpt-4-0613), releasing the optimized experiences and, where possible, the exact prompts and sampling seeds as supplementary material would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified: SEO is an empirical search procedure with external test sets and no self-citation load-bearing chain.

full rationale

SEO is an empirical search procedure rather than a derivation, and its claimed improvements are evaluated on held-out or external test sets. Candidate experiences are generated using the current experience and a single training example, then filtered by performance on a stochastically resampled validation subset of the training set (Eq. 3-4); the final experience is selected using a held-out dev set, and reported results are on separate test sets (HotpotQA test split, WMT23 test set, and the official GLUE dev sets for classification). This is a standard train/validation/test protocol, not a case where a prediction is equivalent to its fitted input by construction. The reference answer Y is visible to the optimizer during candidate generation, but the validation step is precisely the guard against single-example overfitting, and the final numbers are not fitted to the test labels. No load-bearing claim rests on self-citation: the cited work on prompt optimization and experience learning is background context, and none of the authors' own prior results is invoked as a premise or uniqueness theorem. The acknowledged En-Zh Llama-2-13b exception is an empirical robustness concern, not evidence of circularity. Therefore the central claim that SEO finds experiences that improve performance does not reduce to the paper's own equations or inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; 'experience' is a natural-language rule list. The method relies on standard assumptions about validation-set reliability and optimizer capability.

free parameters (6)
  • sample size k = 3
    Number of candidate experiences sampled per training example per step. Chosen by hand; ablation not reported for k.
  • validation set size m = 50
    Size of stochastic validation set sampled from training data at each step. Set to 50, similar to prior prompt optimization work.
  • batch size n = 1 and 3
    The paper says the best experience is selected from runs with batch size 1 and 3 using the held-out dev set. This adds an extra model-selection layer.
  • maximum steps = 200
    Optimization runs at most 200 steps due to computational budget.
  • optimizer temperature = 0.5
    Sampling temperature for Mopt during candidate generation.
  • rule count limit = around ten
    Soft constraint on number of rules in the experience, found by preliminary experiments to avoid context overflow and over-specification.
assumptions (4)
  • domain assumption Performance on a random 50-example subset of training data is a reliable proxy for generalization to the test distribution.
    The stochastic validation set D_t is sampled from D and used to select candidate experiences (Section 2.3). If this proxy is noisy, the optimization direction is unreliable.
  • domain assumption The optimizer Mopt (GPT-4) can generate candidate experiences that are general, non-sentence-specific, and sometimes better than the current experience.
    Section 2.2 relies on the optimizer following the prompt constraints. The ablation shows weaker optimizers hurt, confirming dependence on this assumption.
  • domain assumption The per-example effect delta_E_t = Eval(Yhat_Et) - Eval(Yhat) is a helpful signal for the optimizer.
    Section 2.1 introduces delta as input to Mopt; the ablation in Table 8 shows removing it degrades test performance.
  • domain assumption Decoder temperature 0 for Mgen gives deterministic enough trials for reliable evaluation.
    Section 3.2 sets temperature 0 for all generators; the paper does not assess trial variability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SEO: Stochastic Experience Optimization for Large Language Models." pith.science (2026). https://pith.science/paper/UJAGU4JM

@misc{pith2026250104393,
  author       = {Pith},
  title        = {Pith review of: SEO: Stochastic Experience Optimization for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJAGU4JM}},
  note         = {Machine review of arXiv:2501.04393}
}
read the original abstract

Large Language Models (LLMs) can benefit from useful experiences to improve their performance on specific tasks. However, finding helpful experiences for different LLMs is not obvious, since it is unclear what experiences suit specific LLMs. Previous studies intended to automatically find useful experiences using LLMs, while it is difficult to ensure the effectiveness of the obtained experience. In this paper, we propose Stochastic Experience Optimization (SEO), an iterative approach that finds optimized model-specific experience without modifying model parameters through experience update in natural language. In SEO, we propose a stochastic validation method to ensure the update direction of experience, avoiding unavailing updates. Experimental results on three tasks for three LLMs demonstrate that experiences optimized by SEO can achieve consistently improved performance. Further analysis indicates that SEO-optimized experience can generalize to out-of-distribution data, boosting the performance of LLMs on similar tasks.

Figures

Figures reproduced from arXiv: 2501.04393 by the authors.

Figure 1
Figure 1. Overview of a training step of the SEO process. We first gather trials with and without experience ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt used for the optimizer model to gen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Differences of COMET scores when evaluat [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 5 canonical work pages

  1. [1]

    Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ond r ej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espa \ n a-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, ...

    2021

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  3. [3]

    Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, and Yanghua Xiao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.215 MAPO : Boosting large language model performance with model-adaptive prompt optimization . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3279--3304, Singapore....

    doi:10.18653/v1/2023.findings-emnlp.215 2023

  4. [4]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    arXiv 2022

  5. [5]

    Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.644 Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9465--9480, Abu Dhabi, United Arab Emir...

    work page doi:10.18653/v1/2022.emnlp-main.644 2022

  6. [6]

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023. http://arxiv.org/abs/2309.16797 Promptbreeder: Self-referential self-improvement via prompt evolution

    arXiv 2023

  7. [7]

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. https://doi.org/10.18653/v1/2020.coling-main.580 Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps . In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609--6625, Barcelona, Spain (Online). International Committee ...

    doi:10.18653/v1/2020.coling-main.580 2020

  8. [8]

    Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popovi \'c , and Mariya Shmatova. 2023. http...

    doi:10.18653/v1/2023.wmt-1.1 2023

Show all 33 references
  1. [9]

    Tom Kocmi, Rachel Bawden, Ond r ej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Nov \'a k, Ma...

    2022

  2. [10]

    Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łu...

    2023

  3. [11]

    Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.183 Memory-assisted prompt editing to improve GPT -3 after deployment . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2...

    2022 doi

  4. [12]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://proceed...

    2023

  5. [13]

    Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Peter Jansen, Oyvind Tafjord, Niket Tandon, Li Zhang, Chris Callison-Burch, and Peter Clark. 2023. http://arxiv.org/abs/2310.10134 CLIN : A continually learning language agent for rapid task adaptation and generalization

    2023 arXiv

  6. [14]

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Bel...

    2023 arXiv

  7. [15]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  8. [16]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.494 Automatic prompt optimization with `` gradient descent '' and beam search . In Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023 doi

  9. [17]

    Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pro...

    2022

  10. [18]

    Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

    2020 doi

  11. [19]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. https://openreview.net/forum?id=vAElhFcKW6 Reflexion: language agents with verbal reinforcement learning . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  12. [20]

    Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Siyuan Lu, Yaliang Li, and Ji-Rong Wen. 2024. http://arxiv.org/abs/2402.17564 Unleashing the potential of large language models as prompt optimizers: An analogical analysis with gradient-based model optimizers

    2024 arXiv

  13. [21]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  14. [22]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. https://doi.org/10.18653/v1/2023.acl-long.557 Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions . In Proceedings of the 61st Annual Meeting of th...

    2023 doi

  15. [23]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://openreview.net/forum?id=rJ4km2R5t7 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In International Conference on Learning Representations

    2019

  16. [24]

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.775 S cience W orld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pr...

    2022 doi

  17. [25]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    2024

  18. [26]

    Zeyuan Yang, Peng Li, and Yang Liu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.109 Failures pave the way: Enhancing large language models through tuning-free rule accumulation . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p...

    2023 doi

  19. [27]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/v1/D18-1259 H otpot QA : A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  20. [28]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. https://openreview.net/forum?id=WE_vluYUL-X React: Synergizing reasoning and acting in language models . In The Eleventh International Conference on Learning Representations

    2023

  21. [29]

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2023 a . http://arxiv.org/abs/2308.10144 Expe L : LLM agents are experiential learners

    2023 arXiv

  22. [30]

    Jiachen Zhao, Zonghai Yao, Zhichao Yang, and Hong Yu. 2023 b . https://openreview.net/forum?id=nN8pCTVQZD SELF - EXPLAIN : Teaching large language models to reason complex questions by themselves . In R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models

    2023

  23. [31]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. https://openreview.net/forum?id=92gvk82DE- Large language models are human-level prompt engineers . In The Eleventh International Conference on Learning Representations

    2023

  24. [32]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.