Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

LLM-AutoDiff: Auto-Differentiate Any LLM Workflow

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-AutoDiff claims that every textual input in a multi-component LLM pipeline, including retrieval functions and agent loops, can be optimized by natural-language gradients propagated backward through the workflow graph.

desk verdict Useful engineering for multi-node prompt tuning, but the headline claim that it beats textual-gradient baselines is only tested on single-node tasks. read the letter →

arxiv 2501.16673 v2 pith:FL6SGFKA submitted 2025-01-28 cs.CL

Li Yin , Zhangyang Wang (Atlas) This is my paper · ORCID
classification cs.CL
keywords automaticpromptengineeringtextualgradientsoptimizationretrieval-augmentedgenerationmulti-agentworkflowsgraph-basedcycle-awarebackpropagationLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that automatic prompt optimization can be extended from a single LLM call to arbitrarily structured LLM workflows: pipelines with several LLM nodes, retrieval and formatting functions, and loops that call the same node repeatedly. Its proposal is to treat every textual input to every node as a trainable parameter, run the pipeline forward once, and let a frozen backward-engine LLM generate textual feedback, analogous to gradients, that flows backward along the dependency graph and tells a second optimizer LLM how to rewrite each sub-prompt. The paper argues that this graph-centric design fixes two gaps in prior textual-gradient methods: errors can be traced through functional nodes that have no prompt of their own, and repeated calls in loops get feedback in the correct chronological order. If correct, the result would make prompt engineering for compound systems, such as multi-hop retrieval and agent loops, an automated, gradient-like learning process rather than manual trial and error. The experiments compare against textual-gradient and few-shot baselines on classification, multi-hop retrieval QA, and agent pipelines, reporting higher accuracy at lower training cost.

What carries the argument

The central object is the textual gradient: a natural-language critique, produced by a backward-engine LLM, that says how a node's input or prompt contributed to a low score and what to change. The paper builds a dynamic parameter graph during the forward pass, recording every intermediate input, output, and sub-prompt as a node, and then runs a textual backward pass in which a frozen backward LLM reads the conversation context and the accumulated downstream feedback to emit a textual gradient for each node. An optimizer LLM proposes rewritten sub-prompts, and validation decides whether to keep them. The three named mechanisms, pass-through, time-sequential, and peer-aware gradients, are what let this loop operate on cyclic, multi-component graphs with non-trainable functional nodes.

What would settle it

Run the framework on a multi-hop RAG pipeline in two modes: with genuine backward-engine gradients and with deliberately shuffled or wrong attributions that always blame the final generator; if the wrong-attribution mode improves validation accuracy at the same rate, the claim that node-level root-cause feedback drives the gains would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that auto-differentiation in the textual domain is not limited to a single LLM call: every textual input in a compound LLM system can be optimized by propagating natural-language feedback backward through a directed computation graph. It introduces three mechanisms: pass-through gradients that carry feedback across non-trainable functional nodes such as retrievers or deduplicators; time-sequential gradients that accumulate feedback for the same node called multiple times in a loop and keep the calls in order; and peer-aware gradients that split a prompt into separate sub-prompts, such as instructions, few-shot examples, and output format, so the backward engine can attribute errors to the precise sub-prompt. On top of this, the framework computes gradients only for error samples and uses a two-stage validation to cut cost. The author would state that, across single-step classification, multi-hop retrieval QA, and agent-driven pipelines, the approach consistently beats prior textual-gradient baselines in both accuracy and training cost.

Load-bearing premise

The load-bearing premise is that a frozen backward LLM can reliably attribute each error to the right sub-prompt or predecessor from the aggregated conversation and feedback, a premise the paper does not independently calibrate or human-verify.

Editorial extensions

If this is right

  • If the framework works as claimed, prompt optimization for RAG and agent systems no longer requires hand-tuning each module's prompt; a single backward pass locates the responsible sub-prompt.
  • The time-sequential gradient rule makes cycles optimizable, so multi-hop retrievers and iterative agents can be trained with the same loop as a single LLM call.
  • Error-only gradient computation should make training cheaper than full-batch textual methods, with the paper reporting higher token efficiency and faster convergence.
  • Splitting prompts into peer sub-prompts should reduce the lost-in-the-middle effect, because feedback reaches the exact instruction, format, or example that caused the error.
  • The framework generalizes the single-node textual-gradient paradigm, so future compound LLM applications can be designed as auto-differentiable graphs from the start.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to measure gradient quality directly: have human raters or a held-out judge score whether each textual gradient names the true error source, and correlate that score with downstream accuracy gains.
  • The same graph formalism could co-optimize non-textual knobs such as retrieval depth, chunk size, or temperature by representing them as parameter nodes whose updates are proposed in natural language; the paper lists hyperparameter co-optimization as future work.
  • The error-only gradient rule implies a curriculum-like effect, because the optimizer sees mostly hard samples; this might improve sample efficiency but could also overfit the optimizer to a narrow error distribution, which fresh error types would reveal.
  • If the framework's claims hold, it offers a practical recipe for turning prompt engineering into a declarative optimization problem, which could lower the barrier to building reliable multi-agent systems.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces LLM-AutoDiff, a framework built on the AdalFlow library that treats every textual input in an LLM pipeline as a trainable parameter and uses a frozen backward-engine LLM to generate textual gradients, which are backpropagated through a computation graph to iteratively refine prompts. The method is designed for multi-component, potentially cyclic workflows, with algorithmic contributions including pass-through gradients for functional nodes, time-sequential gradients for repeated calls, peer-aware sub-prompt decomposition, selective gradient computation on error samples, and a gradient-driven prompt optimizer. Experiments cover single-LLM tasks (ObjectCount, TREC-10) and four HotPotQA RAG variants (Vanilla RAG, Multi-hop RAG, Multi-hop RAG(Cycle), Agentic RAG), with comparisons against Text-Grad, DsPy MIPROv2, and OPRO variants. The paper claims consistent outperformance over textual gradient baselines in both accuracy and training cost.

Significance. If the central claims hold, the paper offers a genuinely useful generalization of textual-gradient prompt optimization to compound and cyclic LLM systems, with a concrete implementation and public code. The ablations provide evidence that node-level gradients matter more in multi-node pipelines, and the proposed efficiency techniques (error-only gradients, two-stage validation, multi-proposal optimization) are practically motivated. However, the strongest advertised comparison---outperforming textual-gradient baselines in multi-node settings---is not actually demonstrated, and the training-cost claim is anecdotal. The framework's value is plausible but the evidence presented is incomplete, so the contribution is not yet established at the level claimed.

major comments (5)
  1. [Section 4.2, Table 2] The central claim that LLM-AutoDiff 'consistently outperforms existing textual gradient baselines' is not tested in the multi-node experiments that constitute the paper's novelty. Text-Grad, the primary textual-gradient baseline named in the abstract and related work, appears only in the one-LLM rows (ObjectCount and TREC-10). All four HotPotQA RAG pipelines compare only against DsPy MIPROv2 and OPRO variants, which are not textual-gradient methods. Without running Text-Grad (or an equivalent textual-gradient method) on Vanilla RAG, Multi-hop RAG, Multi-hop RAG(Cycle), and Agentic RAG, the distinctive claim of extending and beating textual-gradient baselines in multi-component or cyclic architectures is unsupported.
  2. [Section 4.2, Table 2, Multi-hop RAG row] The abstract and Section 4.2 claim consistent outperformance, but the reported numbers contradict this on Multi-hop RAG: DsPy(2+2) achieves 50.63% test EM while Ours achieves 48.25%. The text explicitly cites only Vanilla RAG and Multi-hop RAG(Cycle) as wins, yet the wording 'consistently outperforms' and 'average performance improvement of 10%' overstates the multi-node results. The paper should confront this negative result directly, either by discussing statistical significance or by restricting the claim to the settings where it holds.
  3. [Section 4.2, 'Efficiency Gains'] The training-cost superiority claim is not backed by measured data. The only evidence is a textual statement that On ObjectCount, Ours consumes fewer tokens over 12 steps than Text-Grad's 12 proposals; no token counts, wall-clock times, or cost breakdowns are given for any pipeline, and no multi-node cost comparison is provided at all. Since the abstract asserts superiority 'in both accuracy and training cost' across diverse tasks, the paper should report actual training cost metrics (e.g., tokens consumed, number of LLM calls, wall-clock time) for all compared methods and all evaluated pipelines.
  4. [Section 4.2, Table 2, Agentic RAG] The Agentic RAG baseline is DsPy*, a number cited from the original DsPy paper with 'possibly different dataset splits' and no direct run. This is not a controlled comparison. The paper either needs to run DsPy MIPROv2 on the same split for the Agentic RAG pipeline, or clearly label the comparison as a reference point and not use it in the claim of outperformance.
  5. [Section 3.2-3.3, Eqs. (5)-(7), (10)] The entire framework rests on the assumption that the frozen backward-engine LLM can produce accurate root-cause attributions for errors in complex pipelines, yet the paper provides no calibration or control for the quality of these textual gradients. A control replacing the backward-engine feedback with a generic 'improve your output' message---while keeping the same optimizer and selection procedure---would isolate whether node-level attribution, rather than merely the iterative search process, drives the observed gains. The ablation in Table 3 compares with and without gradients, but 'without gradients' refers to OPRO-style input-output-score pairing, not to a lower-quality or random gradient signal; thus the contribution of attribution quality is not directly measured.
minor comments (5)
  1. [Section 3.5] The text reads 'We collectively name our novel enhancements of ORPO as Gradient-Driven Prompt Optimizer (GDPO)'; 'ORPO' should presumably be 'OPRO', since OPRO is the optimizer being extended.
  2. [Section 3.3.3] The text states 'LLM-Diff avoids confusion in multi-subprompt prompts'; the framework is named LLM-AutoDiff elsewhere, so this is likely an inconsistent abbreviation.
  3. [Equation (1)] The notation L({Pi}) = [ i LTi(...) uses an undefined bracket symbol; clarify whether this is a union, summation, or concatenation of losses, and define the index range.
  4. [Table 2 header] The column 'Start Acc(%)(v, t)' is unclear; define what v and t refer to, since the same notation is later used for time steps in Eq. (8).
  5. [Section 4.1, Data splits] The description of the HotPotQA split says '50 hard queries from training' and '7405-test subset into 100-sample validation and 200-sample test'; clarify whether the 50 training queries are distinct from the 100 validation and 200 test samples, and whether the selection is random or fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's equations are definitions, and performance is checked against held-out external metrics.

full rationale

The paper's derivation chain is descriptive rather than deductive. Equations (4)-(10) define how textual gradients are generated and propagated through a computation graph, and no claimed result is assumed inside those definitions. Prompts are optimized on training batches, selected by validation accuracy, and evaluated on held-out test sets with external EM/F1 metrics (Sections 4.1-4.2), so the reported accuracies are not fitted inputs renamed as predictions. The self-citation in Section 3.3.4 (reference [5], whose author list includes co-author Zhangyang Wang) is used only as inspiration for optional skip connections and is not load-bearing: the paper explicitly states these connections must be manually declared and are not automatically created. The most substantive weakness is empirical rather than circular: the abstract claims "consistently outperforms existing textual gradient baselines in both accuracy and training cost," but Table 2 does not run Text-Grad on the multi-node RAG or agent pipelines, so that claim is under-supported. This is a baseline coverage gap, not an equation-level reduction or a fitted-input-as-prediction pattern. No circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

The ledger shows the framework depends on a handful of hand-set training hyperparameters and on unverified assumptions that LLM-generated feedback is a reliable attribution signal. The only invented entities are conceptual bookkeeping devices, textual gradients, time-stamped tuples, and peer decomposition; none is validated in isolation.

free parameters (6)
  • Training steps S_m = 12
    Set to 12 training steps in Section 4.1, mirroring Text-Grad; the optimization algorithm in Algorithm 2 runs to a fixed step count, so the final reported accuracy depends on this stopping point.
  • Minibatch size B = 4
    Batch size 4 for all LLM-AutoDiff runs (Section 4.1); small batches over 50 training samples produce noisy validation decisions.
  • Error threshold tau = not reported
    Eq. (12) and Algorithm 2 use tau to decide which samples receive true backward passes; the value is never given, so token efficiency and training dynamics cannot be exactly reproduced.
  • Retriever top-k = 3 for Vanilla RAG, 2 for others
    Section 4.1 sets top-3 retrieval for Vanilla RAG and top-2 for other RAG pipelines; retrieval depth changes what the generator sees and therefore which prompt is optimal.
  • Backward and optimizer temperature = 1.0
    Appendix B sets temperature=1 for GPT-4o backward and optimizer engines; this randomness affects proposal quality and run-to-run variance.
  • HotPotQA training set composition = 50 hard queries plus 100 validation and 200 test
    Section 4.1 says the 50 training queries are sampled as 'hard' queries from the official set; this is a data-selection choice that influences the magnitude and direction of all HotPotQA results.
assumptions (5)
  • domain assumption A frozen LLM (GPT-4o) can generate reliable root-cause textual gradients from conversation and feedback context.
    Eqs. (4), (5), (7), and (10) all use LLMbackward as the oracle for attribution; the paper provides no calibration or human evaluation showing that these textual gradients correctly identify which sub-prompt caused an error.
  • domain assumption The dynamic parameter graph Gp captures every dependency that influences the final loss.
    Section 3.2 states Gp traces how each parameter affects subsequent computations; if an unmodeled dependency or a user-defined function breaks the trace, backward attribution is incomplete.
  • ad hoc to paper Validation accuracy on a fixed 100-sample set is a reliable selection signal for prompt updates.
    Algorithm 2 accepts or rejects every prompt based on minibatch then full validation accuracy; with 100 validation samples and 12 steps, selection noise could account for some reported gains.
  • ad hoc to paper The pass-through and LLM-backward rules for functional nodes (Eq. 7) preserve the error information needed to update predecessors.
    The rule is a design choice rather than a derived identity; it can either drop or duplicate error signals in graphs with many predecessors.
  • domain assumption EM and F1 evaluation metrics are the correct training signals for these pipelines.
    The loss component uses EM or F1; other properties, such as faithfulness, format compliance, or safety, are not optimized or measured.
invented entities (3)
  • Textual gradient (natural-language feedback used as partial derivative of loss with respect to a variable)
    purpose: Serves as the optimization signal that updates prompts in Eq. (6).
    The concept is defined in this paper; its usefulness is evidenced only by the paper's own benchmark results and ablations, with no external validation of the attribution quality.
  • Time-stamped gradient tuple (t, gradient at invocation t)
    purpose: Preserves temporal order of multiple invocations of the same node in cycles, used in Eq. (8).
    This is internal bookkeeping introduced for cyclic graphs; no ablation isolates its contribution from the rest of the system.
  • Peer decomposition of prompt parameters
    purpose: Splits instructions, output format, and few-shot examples into separately optimizable peers to reduce confounding, Eqs. (9)-(10).
    The paper argues for peer awareness but does not ablate peer versus monolithic prompts directly, so its standalone value is not independently evidenced.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-AutoDiff: Auto-Differentiate Any LLM Workflow." pith.science (2026). https://pith.science/paper/FL6SGFKA

@misc{pith2026250116673,
  author       = {Pith},
  title        = {Pith review of: LLM-AutoDiff: Auto-Differentiate Any LLM Workflow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FL6SGFKA}},
  note         = {Machine review of arXiv:2501.16673}
}
read the original abstract

Large Language Models (LLMs) have reshaped natural language processing, powering applications from multi-hop retrieval and question answering to autonomous agent workflows. Yet, prompt engineering -- the task of crafting textual inputs to effectively direct LLMs -- remains difficult and labor-intensive, particularly for complex pipelines that combine multiple LLM calls with functional operations like retrieval and data formatting. We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE) that extends textual gradient-based methods (such as Text-Grad) to multi-component, potentially cyclic LLM architectures. Implemented within the AdalFlow library, LLM-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine LLM to generate feedback-akin to textual gradients -- that guide iterative prompt updates. Unlike prior single-node approaches, LLM-AutoDiff inherently accommodates functional nodes, preserves time-sequential behavior in repeated calls (e.g., multi-hop loops), and combats the "lost-in-the-middle" problem by isolating distinct sub-prompts (instructions, formats, or few-shot examples). It further boosts training efficiency by focusing on error-prone samples through selective gradient computation. Across diverse tasks, including single-step classification, multi-hop retrieval-based QA, and agent-driven pipelines, LLM-AutoDiff consistently outperforms existing textual gradient baselines in both accuracy and training cost. By unifying prompt optimization through a graph-centric lens, LLM-AutoDiff offers a powerful new paradigm for scaling and automating LLM workflows - mirroring the transformative role that automatic differentiation libraries have long played in neural network research.

Figures

Figures reproduced from arXiv: 2501.16673 by the authors.

Figure 1
Figure 1. Benchmark Performance of LLM-AutoDiff in AdalFlow. reduce a textual “loss.” Current approaches such as Text-Grad [13, 24] have validated the viability of this technique, demon￾strating performance gains in tasks where a single LLM node is trained on labeled data via an iterative prompt-update loop. However, these methods do not fully address the intricacies of multi-component or cyclic pipelines, where prompts for d… view at source ↗
Figure 2
Figure 2. Representing an LLM application as an auto-differentiable computation graph. We illustrate how each node in the graph can be one of three types: an LLM component, a functional component, or a loss component. By coupling this graph with a gradient-driven LLM prompt optimizer, the traditionally labor-intensive task of manually crafting prompts is automated. During training, a forward pass traces every intermediate inp… view at source ↗
Figure 3
Figure 3. Example of a ReACT Auto-Differentiable Graph. This configuration extends our multi-hop RAG paradigm by incorporating a ReACT planner and functional nodes for tool usage and final output assembly. Here, ExecuteAction calls the retriever module when needed, while Finish is a simple function—governed by its own function docstring—that merges the accumulated step history into a coherent answer. The Combine node is a fun… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Gradient-Driven Prompt Optimizer. The optimizer LLM is prompted with the textual gradients and the previous prompt, along with a multi-node “system view.” It proposes new subprompt texts aligned with the identified errors. 4. Experiments In this section, we evaluate LL…
Figure 5
Figure 5. Figure 5: Object count computation graph Trec-10 The runtime computation graph: 26 [PITH_FULL_IMAGE:figures/full_fig_p026_5.png]
Figure 6
Figure 6. Figure 6: Trec-6 computation graph Vanilla RAG The runtime computation graph [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Vanilla RAG computation graph. Multi-hop RAG The runtime computation graph: 27 [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: Multi-hop RAG computation graph. We see query generator 0 output and query generator 1 output are outputs from two different genertor components. Multi-hop RAG(Cycle) The runtime computation graph: 28 [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 9
Figure 9. Figure 9: Multi-hop RAG(Cycle) computation graph. We see [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VeRO: A Harness for Agents to Optimize Agents

    cs.AI 2026-02 unverdicted novelty 7.0 of 10

    VeRO supplies a versioned harness, benchmark suite, and empirical comparison of optimizer configurations for coding agents that improve other agents.

  2. HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A new benchmark, HarnessOpt-Bench, measures how well frontier LLMs can improve the prompts, tools, and control flow of target agents under a fixed evaluation budget with a held-out test set.

  3. LAD-VF: LLM-Automatic Differentiation Enables Fine-Tuning-Free Robot Planning from Formal Methods Feedback

    cs.RO 2025-09 conditional novelty 5.0 of 10

    LAD-VF iteratively rewrites LLM prompts using formal verification failure rates as a loss, lifting safety-spec compliance from about 60% to over 90% without model fine-tuning.

  4. Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems

    cs.AI 2025-05 conditional novelty 5.0 of 10

    The paper shows that evaluating prompts by the semantic similarity of repeated LLM outputs, and refining prompts toward higher similarity, improves task success in general-purpose multi-agent systems.

Reference graph

Works this paper leans on

32 extracted references · 16 canonical work pages · cited by 4 Pith papers

  1. [1]

    {TensorFlow}: a system for {Large-Scale} machine learning

    Mart ´ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. {TensorFlow}: a system for {Large-Scale} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283, 2016. 3, 5

    work page 2016

  2. [2]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1, 4, 5

    arXiv 2005

  3. [3]

    Rlprompt: Optimizing discrete text prompts with reinforcement learning

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205.12548, 2022. 5

    arXiv 2022

  4. [4]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 11

    2016

  5. [5]

    Training your sparse neural network better with any mask

    Ajay Kumar Jaiswal, Haoyu Ma, Tianlong Chen, Ying Ding, and Zhangyang Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pages 9833–9844. PMLR, 2022. 11

    work page 2022

  6. [6]

    Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensive nlp

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024, 2022. 1, 2, 4, 5, 6, 10, 15, 16

    arXiv 2022

  7. [7]

    An llm compiler for parallel function calling

    Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. An llm compiler for parallel function calling. In Forty-first International Conference on Machine Learning. 1

    work page

  8. [8]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Infor- mation Processing Systems, 33:9459–9474, 2020. 2, 4

    work page 2020

Show all 32 references
  1. [9]

    Learning question classifiers

    Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics, 2002. 14

    2002

  2. [10]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. 2, 4, 12

    2024

  3. [11]

    Optimizing instructions and demonstrations for multi-stage language model programs

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. arXiv preprint arXiv:2406.11695, 2024. 5, 15, 16

    2024 arXiv

  4. [12]

    Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, 2019

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library.Advances in neural information processing systems, 32, ...

    2019

  5. [13]

    Automatic prompt optimization with” gradient descent” and beam search

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023. 1, 2, 5

    2023 arXiv

  6. [14]

    Evaluating web-based question answering systems

    Dragomir R Radev, Hong Qi, Harris Wu, and Weiguo Fan. Evaluating web-based question answering systems. In LREC. Citeseer,

  7. [15]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 1, 4

    2024

  8. [16]

    Fine-tuning and prompt optimization: Two great steps that work better together

    Dilara Soylu, Christopher Potts, and Omar Khattab. Fine-tuning and prompt optimization: Two great steps that work better together. arXiv preprint arXiv:2407.10930, 2024. 5

    2024 arXiv

  9. [17]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri `a Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint...

    2022 arXiv

  10. [18]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Sch ¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. 12, 14

    arXiv

  11. [19]

    How to correctly do semantic backpropagation on language-based agentic systems

    Wenyi Wang, Hisham A Alyahya, Dylan R Ashley, Oleg Serikov, Dmitrii Khizbullin, Francesco Faccio, and J ¨urgen Schmidhuber. How to correctly do semantic backpropagation on language-based agentic systems. arXiv preprint arXiv:2412.03624, 2024. 5

    2024 arXiv

  12. [20]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022. 1, 4

    2022

  13. [21]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023. 1, 4, 5, 12, 16

    2023 arXiv

  14. [22]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. 14, 15

    2018 arXiv

  15. [23]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. 1, 2, 4, 6, 10, 15

    2022 arXiv

  16. [24]

    Textgrad: Automatic” differentiation” via text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic” differentiation” via text. arXiv preprint arXiv:2406.07496, 2024. 1, 2, 5, 8, 11, 15

    2024 arXiv

  17. [25]

    Tempera: Test-time prompting via reinforce- ment learning

    Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforce- ment learning. arXiv preprint arXiv:2211.11890, 2022. 5

    2022 arXiv

  18. [26]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36:46595–46623, 2023. 7, 19

    2023

  19. [27]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022. 5 A. Meta Prompts Meta Prompts for Backward Engines(GradComponent) This includ...

    2022 arXiv

  20. [28]

    From <CONVERSATION></CONVERSATION> section, you can find how the variable is obtained and used

  21. [29]

    As there might be multiple precedessors, and multi-components, it is possible that the feedback/error is not directly related to the variable itself

  22. [30]

    When you reason, really think about the variable’s role in the component(infer from the CONVERSATION section) and the VARIABLE section before you provide feedback

  23. [31]

    Be specific, concise, critical, and direct

  24. [32]

    "" Listing 2. Intermediate objective template(General GradComponent). OBJECTIVE_INSTRUCTION_CHAIN = r

    Maximum 3 sentences. [Cycle]: If the same DataID has multiple gradients, it means this component/variable is called multiple times in the compound system(with a cycle) in the same order as it appears in the gradient list. Ensure the feedback is aware of all sets of inputs and ...

    2000

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.