Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

RiOT: Efficient Prompt Refinement with Residual Optimization Tree

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic prompt optimization can be made more diverse and drift-resistant by organizing it as a tree that branches into several candidates, keeps the most informative one, and fuses it back with its parent.

desk verdict A sensible but underspecified prompt-optimization recipe: the diversity mechanism needs evidence before the empirical claims hold. read the letter →

arxiv 2506.16389 v1 pith:PGJHWAOI submitted 2025-06-19 cs.CL

Chenyi Zhou , Zhengyan Shi , Yuan Yao , Lei Liang , Huajun Chen , Qiang Zhang This is my paper · ORCID
classification cs.CL
keywords promptoptimizationresidualconnectionsperplexitytreesearchchain-of-thoughtpromptingreasoningbenchmarkssemanticdrifttextgradients
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic prompt optimization has a two-sided failure mode: iterative edits converge on a narrow phrasing, and each rewrite can erase useful pieces of the previous prompt. RiOT claims to fix both at once by treating optimization as a tree: at every step an optimizer LLM proposes several semantically different candidate rewrites, the method keeps the one with the highest perplexity, then merges the survivor with its parent via sentence-level similarity before moving down the tree. On five benchmarks spanning commonsense, math, logic, dates, and object counting, this procedure yields the best weighted-average accuracy of 77.2 percent, beating the strongest prior automatic method by 2.7 points and beating manual chain-of-thought prompting. The implication is that black-box prompt search can be made broader and more stable without parameter access or extra training.

What carries the argument

The load-bearing object is the optimization tree itself: each node is a prompt, each level is one refinement step, and the tree width K is the number of candidate rewrites per parent. Two mechanisms drive it. Perplexity-informed node selection scores each child by its perplexity under the target LLM and keeps the maximum, treating high uncertainty as a proxy for semantic informativeness. Text residual connection then fuses parent and child: it tokenizes both into sentences, embeds them, keeps parent sentences whose best similarity to a child sentence is at least 1-b1, adds child sentences whose best similarity to a parent sentence is below 1-b2, and unions the two sets into the next prompt. The fusion thresholds b1=0.25 and b2=0.5 control how much old content is preserved and how much new content enters, so each step is a bounded edit instead of a fresh rewrite.

What would settle it

Log the K=3 candidate prompts produced at a single optimization step with temperature 0: if repeated invocations return identical text, the tree has no width and the perplexity selection and fusion steps have nothing to act on. A second check is to replace perplexity selection with random selection among the K candidates and compare GSM8K accuracy; if the gap vanishes, the claimed diversity mechanism is not doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two known failure modes of iterative prompt optimization—low diversity and semantic drift—are jointly curable by a tree whose nodes are prompts and whose edges are residual fusions. Starting from a root prompt, each iteration produces K child candidates through text-gradient feedback, selects the child with the highest perplexity as the most informative branch, and then composes the selected child with its parent by embedding both at sentence level and retaining parent sentences that resemble the child and child sentences that are new. This step is repeated down a single path of the tree for 15 iterations with K=3. On the paper's five task benchmarks, the method reaches 81.2 percent on GSM8K, improves over zero-shot CoT on all five tasks, and reaches the highest weighted average of 77.2 percent; ablations attribute 13.0 and 12.4 points on GSM8K to the diversity-based selection and the residual connection respectively.

Load-bearing premise

The method assumes that calling the same optimizer model several times with the same input produces several genuinely different candidate prompts; the paper never explains where that variation comes from, since it runs the model with its randomness-level fixed at zero.

Editorial extensions

If this is right

  • If the claim holds, prompt optimization remains fully black-box: only LLM APIs and an embedding model are needed, with no parameter gradients, logit access, or reward-model training.
  • A single fixed path through the tree yields the gains, so the framework can later be expanded into beam search over multiple lineages without changing the fusion step.
  • Perplexity-maximizing selection outperforms entropy- or length-based selection on GSM8K, suggesting the diversity signal is semantic rather than statistical verbosity.
  • The optimized prompts transfer to a different target model in three of five tasks, and optimizing directly for that model improves all five, implying the gains are not tied to one LLM.
  • On the harder AMC12 set, the method reaches 46.0 percent, 5.1 points over zero-shot CoT, pointing to usefulness where data is scarce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to make K's diversity explicit: sampling with temperature above zero or with varied optimization instructions could make the three candidates genuinely distinct, which would let the perplexity and residual components be tested on their own terms.
  • The same sentence-level fusion may apply beyond prompts, to instructions, code scaffolding, or reasoning templates, where the parent-child edit is discrete text and drift is the main failure mode.
  • The paper's own limitations—text-only tasks and uneven per-task gains—suggest the strongest case is for textual reasoning; whether RiOT's fusion generalizes to multimodal instructions is untested and would be the most direct next experiment.
  • Because K=3 is optimal and larger K hurts, more candidates is not automatically better; the interaction between candidate count and fusion thresholds is worth a dedicated study.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes RiOT, a tree-based prompt optimization framework built on the TextGrad backbone. At each optimization step, RiOT generates K candidate prompts from text gradients, selects one child node by a perplexity-based criterion, and applies a text residual connection that fuses parent and child sentences using an embedding model and two similarity thresholds. The method is evaluated on five reasoning benchmarks against manual CoT prompting and four automatic prompt optimization baselines, reporting a weighted average accuracy of 77.2% and additional experiments on generalization, ablations, selection metrics, and computational cost.

Significance. If the mechanism works as described, the contribution is of practical value: it targets two real limitations of black-box prompt optimization, candidate diversity and semantic drift, with a modular tree structure and a lightweight selection scheme. The paper includes a promised code release, held-out evaluation with validation-based prompt selection, and a reasonable range of baselines. The residual fusion idea and the tree formulation are simple and potentially useful for practitioners. However, the current evidence does not yet establish the central claim, because the temperature-0 diversity mechanism is unsupported, the perplexity formula in Eq. (6) is incorrect, and the abstract overstates the per-dataset results relative to Table 1.

major comments (5)
  1. [§4 and Implementation Details (Section 5)] The mechanism for generating K diverse candidates assumes stochasticity that the experimental configuration does not provide. Implementation Details set both GPT-3.5-turbo (target) and GPT-4o (optimizer) temperature to 0, yet §4 states that 'LLMs exhibit inherent variability in their outputs' without identifying the source of nondeterminism. With greedy decoding, K calls with identical inputs normally produce identical outputs, which would collapse the tree to a chain and make the perplexity-based selection and the Table 3 diversity ablation vacuous. The paper should either report the actual distinct candidate strings for a representative iteration or specify the sampling mechanism (e.g., API-level nondeterminism, top-p sampling, or varied inputs). This also bears on Table 1: with temperature fixed at 0, the nonzero standard deviations over five runs are unexplained.
  2. [§4.1, Eq. (6)] Equation (6) defines PPL(p) = exp{-(1/J) * sum p_theta(x_j|x_<j)}, omitting the logarithm that is essential to the standard perplexity definition. As written, the quantity is the exponential of a negative average token probability, not a perplexity; it is at most 1, and maximizing it is equivalent to minimizing the average token probability, which is the opposite of the uncertainty-based interpretation in the text. If the implementation follows Eq. (6) literally, the selection criterion is not perplexity-based. Please correct the formula to exp{-(1/J) * sum log p_theta(x_j|x_<j)} and confirm that the code and reported rankings correspond to the corrected definition.
  3. [Abstract and Table 1] The abstract claims that RiOT 'outperforms both previous prompt optimization methods and manual prompting,' but Table 1 shows that TextGrad reaches 88.3% on Object Counting while RiOT reaches 86.9%, so RiOT does not beat all automatic optimization baselines on every benchmark. The main text correctly describes the Object Counting result as 'suboptimal.' The abstract and the contribution framing should be revised to state that RiOT achieves the best weighted average accuracy and leads on four of five tasks, rather than outperforming all baselines across all five benchmarks.
  4. [Table 3 and §6.2] The ablation labeled 'w/o Perplexity-Informed Node Selection' is implemented by setting K=1, which simultaneously removes candidate diversity and the selection rule. This design cannot attribute the 13.0% drop specifically to the perplexity criterion; the drop could equally reflect the loss of multiple candidates or the absence of any selection among them. To isolate the contribution, the authors should compare perplexity-based selection against random or first-candidate selection at the same K (e.g., K=3) and report the resulting accuracy.
  5. [Table 1 and §6.1] Several headline per-task gains are within one standard deviation of the closest baseline. For example, LogiQA 2.0 shows 61.4±1.5 vs. TextGrad's 60.0±1.2, and StrategyQA shows 74.6±1.5 vs. DSPy's 73.4±1.5. Because the central empirical claim depends on small differences, the paper should report paired significance tests or confidence intervals over the five runs, or explicitly limit the claim of per-task superiority to the weighted average and to those tasks where the difference is statistically credible.
minor comments (6)
  1. [Appendix E] In the Date Understanding entry, the sentence 'For our study, For our study, we sample 329 test samples' contains a duplicated phrase; please fix it.
  2. [Table 6] The rows 'RiOT (Single)' and 'RiOT (Multi)' are not defined in the main text; clarify what the two threading settings are and how the reported runtimes were measured.
  3. [Figure 2] The axis label 'Token' in Figure 2 is ambiguous; use 'Prompt length (tokens)' or define the secondary variable explicitly in the caption.
  4. [Table 7 caption] The caption contains a typo: 'Comparion' should be 'Comparison.'
  5. [§4.1 and Appendix B] The definitions of perplexity, entropy, and length do not specify which model supplies the token probabilities p_theta(x_j|x_<j); state whether this is the target model or the optimizer model, and how those probabilities are obtained in the implementation.
  6. [§4] The notation alternates among p_t, p*_t, and p^{(i)}_{t+1} without a formal definition of the tree level and the parent-child relationship; a short formalization of the tree data structure would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are evaluated on held-out test samples with validation-tuned hyperparameters, and no load-bearing step reduces to its own inputs.

full rationale

The paper's central claim is an empirical accuracy comparison across five benchmarks. The test sets are fixed and reported with standard deviations over five runs, while hyperparameters (K, b1, b2) are selected on a validation set and the optimal prompt is identified on validation before testing. No fitted parameter is renamed as a prediction, and no derived quantity is equal to an input by construction. The perplexity-informed node selection and text residual connection are introduced as heuristic components and are evaluated through ablations; they are not definitions that presuppose the performance gain. The paper adopts TextGrad as an external backbone and cites prior prompt-optimization work without invoking any uniqueness theorem or author-specific ansatz as the load-bearing justification. Self-citations in the introduction (e.g., Jiang et al., 2024a, 2025b; Xu et al., 2025) are incidental references to general LLM work and do not support the central derivation. Potential technical concerns, such as the stated temperature-0 sampling in tension with the claim of K diverse candidates, and the nonstandard form of Eq. (6) for perplexity, are correctness or evidence issues rather than circularity: the paper does not define its outputs in terms of its inputs, and no quoted equation or procedure reduces a claimed result to a fitted value or self-citation. Under the rule that only explicit reductions count as circularity, the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The core method depends on several hand-set thresholds and heuristic assumptions rather than a closed-form derivation; these are the main sources of fragility. No new physical or ontological entities are introduced.

free parameters (5)
  • b1 = 0.25
    Similarity threshold for retaining parent sentences in the Content Fusion Algorithm (Algorithm 1). Selected based on GSM8K sensitivity analysis (Figure 2b), which shows accuracy decreasing as b1 increases.
  • b2 = 0.5
    Similarity threshold for selecting novel child sentences. Chosen as the trade-off point between performance and prompt length in Figure 2c.
  • K = 3
    Tree width: number of candidate prompts generated per iteration. Figure 2a shows best accuracy on GSM8K at K=3, with decline for larger K.
  • Optimization iterations = 15
    Number of tree levels, derived from batch size 4 and 3 training epochs; not independently justified.
  • Embedding model = text-embedding-3-large
    Choice of embedding model for sentence similarity in residual connection; Table 5 shows a 21% accuracy drop with a weaker embedding model.
assumptions (4)
  • ad hoc to paper The optimizer LLM produces distinct candidate prompts across K calls despite temperature set to 0
    Section 4 says LLM variability yields K different candidates, but Implementation Details set temperature 0 for both models, which typically removes stochasticity.
  • domain assumption Perplexity of a prompt is a good proxy for semantic diversity or informativeness
    Section 4.1 justifies selection by analogy to information theory and Bayesian optimization, without direct evidence that high-perplexity prompts improve downstream task accuracy.
  • domain assumption Cosine similarity between sentence embeddings captures semantic redundancy well enough for residual fusion
    Algorithm 1's thresholds assume that sentences with high embedding similarity are redundant and low similarity are novel; Table 5 shows sensitivity to embedding quality.
  • domain assumption The sampled test subsets (e.g., 100 samples for GSM8K and StrategyQA) are representative of the full benchmark
    Small test sets inflate variance and no sampling seeds are reported, so exact replication is difficult.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RiOT: Efficient Prompt Refinement with Residual Optimization Tree." pith.science (2026). https://pith.science/paper/PGJHWAOI

@misc{pith2026250616389,
  author       = {Pith},
  title        = {Pith review of: RiOT: Efficient Prompt Refinement with Residual Optimization Tree},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGJHWAOI}},
  note         = {Machine review of arXiv:2506.16389}
}
read the original abstract

Recent advancements in large language models (LLMs) have highlighted their potential across a variety of tasks, but their performance still heavily relies on the design of effective prompts. Existing methods for automatic prompt optimization face two challenges: lack of diversity, limiting the exploration of valuable and innovative directions and semantic drift, where optimizations for one task can degrade performance in others. To address these issues, we propose Residual Optimization Tree (RiOT), a novel framework for automatic prompt optimization. RiOT iteratively refines prompts through text gradients, generating multiple semantically diverse candidates at each step, and selects the best prompt using perplexity. Additionally, RiOT incorporates the text residual connection to mitigate semantic drift by selectively retaining beneficial content across optimization iterations. A tree structure efficiently manages the optimization process, ensuring scalability and flexibility. Extensive experiments across five benchmarks, covering commonsense, mathematical, logical, temporal, and semantic reasoning, demonstrate that RiOT outperforms both previous prompt optimization methods and manual prompting.

Figures

Figures reproduced from arXiv: 2506.16389 by the authors.

Figure 1
Figure 1. Overview of RIOT. (i) At each step, RIOT generates multiple candidate prompts (child nodes). (ii) Then, the optimal child node is selected based on perplexity. The perplexity-informed selection process is shown in the region within the blue block . (iii) Finally, the parent node is connected to the optimal child node based on semantic similarity. This residual connection process is highlighted within the yellow bloc… view at source ↗
Figure 2
Figure 2. Hyperparametric Senstivity Analysis of RIOT. Our results show that (a): Increasing the tree width K leads to a unimodal curve, with peak accuracy at K = 3; (b): The parameter b1 is negatively correlated with performance; and (c): The parameter b2 plays a critical role in balancing performance and computational overhead. 6 Results 6.1 Main Results Finding #1: Scaling the number of demonstra￾tion examples in few-shot … view at source ↗
Figure 3
Figure 3. List of optimized prompts on GSM8K by different prompt optimization methods, categorized according [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Weighted Average Accuracy (WAA) of differ [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: An example of few-shot prompts from GSM8K. The purple section represent the [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The prompt template for the LLM-based Implementation of Text Residual Connection. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination

    cs.LG 2026-07 conditional novelty 6.0 of 10

    HERO optimizes programs by generating atomic edits without score feedback and selecting the highest-scoring subset of those edits, avoiding the 'weakest-link' failure of accepting or rejecting whole edit bundles.

Reference graph

Works this paper leans on

58 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Xavier Amatriain. 2024. Prompt design and engineering: Introduction and advanced methods. arXiv preprint arXiv:2401.14423

    arXiv 2024

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  6. [6]

    Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69--72

    work page 2006

  7. [7]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    arXiv 2021

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

Show all 58 references
  1. [9]

    Huajun Chen. 2024. https://doi.org/10.3724/2096-7004.di.2024.0001 Large knowledge model: Perspectives and challenges . Data Intelligence, 6(3):587--620

    2024

  2. [10]

    Justin Chih-Yao Chen, Swarnadeep Saha, and Mohit Bansal. 2023. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. arXiv preprint arXiv:2309.13007

    2023 arXiv

  3. [11]

    Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang, and Yajun Du. 2024. https://doi.org/10.1162/dint_a_00251 Llama-lora neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains . Data Intellig...

    2024 doi

  4. [12]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  5. [13]

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.222 RLP rompt: Optimizing discrete text prompts with reinforcement learning . In Proceedings of the 2022 Confe...

    2022 doi

  6. [14]

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.106 L o RAM o E : Alleviating world kn...

    2024 doi

  7. [15]

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325

    2023 arXiv

  8. [16]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. https://doi.org/10.18653/v1/2021.acl-long.295 Making pre-trained language models better few-shot learners . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint...

    2021 doi

  9. [17]

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. https://doi.org/10.1162/TACL\_A\_00370 Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies . Trans. Assoc. Comput. Linguistics, 9:346--361

    2021 doi

  10. [18]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778

    2016

  11. [19]

    Songtao Jiang, Yan Zhang, Yeying Jin, Zhihang Tang, Yangyang Wu, Yang Feng, Jian Wu, and Zuozhu Liu. 2025 a . Hscr: Hierarchical self-contrastive rewarding for aligning medical vision language models. arXiv preprint arXiv:2506.00805

    2025 arXiv

  12. [20]

    Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, and Zuozhu Liu. 2024 a . Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv preprint arXiv:2404.04514

    2024 arXiv

  13. [21]

    Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, and Zuozhu Liu. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.221 M ed- M o E : Mixture of domain-specific experts for lightweight medical vision-language models . In Findings of the Association for Computati...

    2024 doi

  14. [22]

    Songtao Jiang, Chenyi Zhou, Yan Zhang, Yeying Jin, and Zuozhu Liu. 2025 b . Fast or slow? integrating fast intuition and deliberate thinking for enhancing visual question answering. arXiv preprint arXiv:2506.00806

    2025 arXiv

  15. [23]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. https://openreview.net/forum?id=sY5N0zY5Od DSP y: Compil...

    2024

  16. [24]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017 a . Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy...

    2017

  17. [25]

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017 b . Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy...

    2017

  18. [26]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  19. [27]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.243 The power of scale for parameter-efficient prompt tuning . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045--3059, Online a...

    2021 doi

  20. [28]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  21. [29]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092--1097

    2022

  22. [30]

    Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935--2947

    2017

  23. [31]

    Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. 2023. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing

    2023

  24. [32]

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2024. Gpt understands, too. AI Open, 5:208--215

    2024

  25. [33]

    Davide Maltoni and Vincenzo Lomonaco. 2019. Continuous learning in single-incremental-task scenarios. Neural Networks, 116:56--73

    2019

  26. [34]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  27. [35]

    OpenAI. 2022. Introducing chatgpt. https://openai.com/index/chatgpt/. Accessed: 2025-01-21

    2022

  28. [36]

    OpenAI. 2023. New embedding models and api updates. https://openai.com/index/new-embedding-models-and-api-updates/. Accessed: 2025-01-21

    2023

  29. [37]

    OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-01-21

    2024

  30. [38]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.494 Automatic prompt optimization with gradient descent and beam search . In Proceedings of the 2023 Conference on Empirical Methods in Natural Languag...

    2023 doi

  31. [39]

    Guanghui Qin and Jason Eisner. 2021. https://doi.org/10.18653/v1/2021.naacl-main.410 Learning how to ask: Querying LM s with mixtures of soft prompts . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2021 doi

  32. [40]

    Weijieying Ren, Xinlong Li, Lei Wang, Tianxiang Zhao, and Wei Qin. 2024. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. arXiv preprint arXiv:2402.18865

    2024 arXiv

  33. [41]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1--7

    2021

  34. [42]

    Greene Ryan, Sanders Ted, Weng Lilian, and Neelakantan Arvind. 2022. New and improved embedding model. https://openai.com/index/new-and-improved-embedding-model/. Accessed: 2025-01-21

    2022

  35. [43]

    Claude Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379--423

    1948

  36. [44]

    Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.733 Toward human readable prompt tuning: Kubrick`s the shining is a good movie, and a good prompt too? In Findings of the Associati...

    2023 doi

  37. [45]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.346 A uto P rompt: E liciting K nowledge from L anguage M odels with A utomatically G enerated P rompts . In Proceedings of the 2020 Conference o...

    2020 doi

  38. [46]

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25

    2012

  39. [47]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating t...

    2023

  40. [48]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Sch \" a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-ACL.824 Challenging big-bench tasks and whether chain-of-tho...

    2023 doi

  41. [49]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

    2024 arXiv

  42. [50]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  43. [51]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36

    2024

  44. [52]

    Qinwei Xu, Xingkun Xu, Chenyi Zhou, Zuozhu Liu, Feiyue Huang, Shaoxin Li, Lifeng Zhu, Zhian Bai, Yuchen Xu, and Weiguo Hu. 2025. https://doi.org/10.1016/j.eswa.2025.126585 Towards normalized clinical information extraction in chinese radiology report with large language models...

    2025

  45. [53]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, ...

    2024

  46. [54]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  47. [55]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496

    2024 arXiv

  48. [56]

    Gonzalez

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2023. https://openreview.net/forum?id=gSHyqBijPFO TEMPERA : Test-time prompt editing via reinforcement learning . In The Eleventh International Conference on Learning Representations

    2023

  49. [57]

    Xilin Zhang, Zhixin Mao, Ziwen Chen, and Shen Gao. 2024. https://doi.org/10.3724/2096-7004.di.2024.0013 Effective tool augmented multi-agent framework for data analysis . Data Intelligence, 6(4):923--945

    2024

  50. [58]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. https://openreview.net/forum?id=92gvk82DE- Large language models are human-level prompt engineers . In The Eleventh International Conference on Learning Representation...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.