{"id":"686a06e5-2fd1-4fb2-94f6-ac50e8728681","arxiv_id":"2501.16673","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An automatic differentiation-style framework optimizes prompts across multi-component and cyclic LLM workflows, outperforming single-node textual-gradient baselines on several small benchmarks.","lead":"LLM-AutoDiff treats every prompt in a multi-step LLM pipeline as a trainable parameter and uses a second LLM to generate feedback, called textual gradients, that improves those prompts. It reports accuracy gains on single-node, multi-hop retrieval, and agent tasks, but key comparisons to textual-gradient baselines on multi-node tasks are missing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-node outperformance claim lacks the key baseline: Text-Grad is absent from all RAG/agent comparisons, so \"consistently outperforms textual gradient baselines\" is only tested on single-node tasks.","rationale":"The reader's verdict was CONDITIONAL, and the missing Text-Grad baseline in multi-node experiments is already noted in the reader's rationale, though the reader's stated weakest assumption focuses on the reliability of textual gradients. I agree with the overall conditional assessment, but I identify the missing baseline as the more load-bearing concern: even if the backward-engine gradients are noisy, the empirical comparisons to DsPy and OPRO could still support a weaker claim about multi-node prompt optimization; however, the paper's headline claim specifically asserts superiority over textual-gradient baselines, and that comparison is absent exactly where the method's novelty lies. My concern does not require changing the verdict: the framework may still be a valid contribution, but acceptance should be conditional on adding Text-Grad baselines (or softening the claim). I therefore recommend keeping the reader's CONDITIONAL verdict.","tokens_in":20190,"tokens_out":2916,"duration_ms":29573,"concrete_test":"Re-run the HotPotQA Multi-hop RAG(Cycle) experiment with Text-Grad as a baseline, using the same 50/100/200 train/validation/test split, GPT-3.5-turbo forward engine, GPT-4o backward/optimizer model, and the same evaluation metric. If Text-Grad cannot express the cyclic graph, run it in its supported multi-variable form on the (query-generator, generator) prompts, or on the final generator prompt only, and compare final test accuracy and total token usage over the same 12 training steps. If Text-Grad matches or exceeds LLM-AutoDiff, the claimed consistent outperformance over textual-gradient baselines in multi-node settings is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LLM-AutoDiff \"consistently outperforms existing textual gradient baselines in both accuracy and training cost.\" In the multi-node experiments that constitute the novelty--Vanilla RAG, Multi-hop RAG, Multi-hop RAG(Cycle), and Agentic RAG in Table 2--the only baselines are DsPy MIPROv2 and OPRO variants; Text-Grad is not run. Text-Grad appears only in the one-LLM rows (ObjectCount, TREC-10). Consequently, the distinctive claim that the method extends textual gradients to multi-component and cyclic architectures and beats textual-gradient baselines there is not empirically supported. The Agentic RAG row compares only to a reported DsPy* number from another paper, with possibly different splits and no direct run. Likewise, the training-cost claim rests on an anecdotal token comparison on ObjectCount, not on measured cost across the multi-node workflows. A framework can still be useful if it only beats non-gradient APE methods, but the abstract and conclusions assert a stronger, unsupported comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLM-AutoDiff, a framework built on the AdalFlow library that treats every textual input in an LLM pipeline as a trainable parameter and uses a frozen backward-engine LLM to generate textual gradients, which are backpropagated through a computation graph to iteratively refine prompts. The method is designed for multi-component, potentially cyclic workflows, with algorithmic contributions including pass-through gradients for functional nodes, time-sequential gradients for repeated calls, peer-aware sub-prompt decomposition, selective gradient computation on error samples, and a gradient-driven prompt optimizer. Experiments cover single-LLM tasks (ObjectCount, TREC-10) and four HotPotQA RAG variants (Vanilla RAG, Multi-hop RAG, Multi-hop RAG(Cycle), Agentic RAG), with comparisons against Text-Grad, DsPy MIPROv2, and OPRO variants. The paper claims consistent outperformance over textual gradient baselines in both accuracy and training cost.","tokens_in":20482,"tokens_out":3492,"duration_ms":31694,"significance":"If the central claims hold, the paper offers a genuinely useful generalization of textual-gradient prompt optimization to compound and cyclic LLM systems, with a concrete implementation and public code. The ablations provide evidence that node-level gradients matter more in multi-node pipelines, and the proposed efficiency techniques (error-only gradients, two-stage validation, multi-proposal optimization) are practically motivated. However, the strongest advertised comparison---outperforming textual-gradient baselines in multi-node settings---is not actually demonstrated, and the training-cost claim is anecdotal. The framework's value is plausible but the evidence presented is incomplete, so the contribution is not yet established at the level claimed.","major_comments":[{"comment":"The central claim that LLM-AutoDiff 'consistently outperforms existing textual gradient baselines' is not tested in the multi-node experiments that constitute the paper's novelty. Text-Grad, the primary textual-gradient baseline named in the abstract and related work, appears only in the one-LLM rows (ObjectCount and TREC-10). All four HotPotQA RAG pipelines compare only against DsPy MIPROv2 and OPRO variants, which are not textual-gradient methods. Without running Text-Grad (or an equivalent textual-gradient method) on Vanilla RAG, Multi-hop RAG, Multi-hop RAG(Cycle), and Agentic RAG, the distinctive claim of extending and beating textual-gradient baselines in multi-component or cyclic architectures is unsupported.","section":"Section 4.2, Table 2"},{"comment":"The abstract and Section 4.2 claim consistent outperformance, but the reported numbers contradict this on Multi-hop RAG: DsPy(2+2) achieves 50.63% test EM while Ours achieves 48.25%. The text explicitly cites only Vanilla RAG and Multi-hop RAG(Cycle) as wins, yet the wording 'consistently outperforms' and 'average performance improvement of 10%' overstates the multi-node results. The paper should confront this negative result directly, either by discussing statistical significance or by restricting the claim to the settings where it holds.","section":"Section 4.2, Table 2, Multi-hop RAG row"},{"comment":"The training-cost superiority claim is not backed by measured data. The only evidence is a textual statement that On ObjectCount, Ours consumes fewer tokens over 12 steps than Text-Grad's 12 proposals; no token counts, wall-clock times, or cost breakdowns are given for any pipeline, and no multi-node cost comparison is provided at all. Since the abstract asserts superiority 'in both accuracy and training cost' across diverse tasks, the paper should report actual training cost metrics (e.g., tokens consumed, number of LLM calls, wall-clock time) for all compared methods and all evaluated pipelines.","section":"Section 4.2, 'Efficiency Gains'"},{"comment":"The Agentic RAG baseline is DsPy*, a number cited from the original DsPy paper with 'possibly different dataset splits' and no direct run. This is not a controlled comparison. The paper either needs to run DsPy MIPROv2 on the same split for the Agentic RAG pipeline, or clearly label the comparison as a reference point and not use it in the claim of outperformance.","section":"Section 4.2, Table 2, Agentic RAG"},{"comment":"The entire framework rests on the assumption that the frozen backward-engine LLM can produce accurate root-cause attributions for errors in complex pipelines, yet the paper provides no calibration or control for the quality of these textual gradients. A control replacing the backward-engine feedback with a generic 'improve your output' message---while keeping the same optimizer and selection procedure---would isolate whether node-level attribution, rather than merely the iterative search process, drives the observed gains. The ablation in Table 3 compares with and without gradients, but 'without gradients' refers to OPRO-style input-output-score pairing, not to a lower-quality or random gradient signal; thus the contribution of attribution quality is not directly measured.","section":"Section 3.2-3.3, Eqs. (5)-(7), (10)"}],"minor_comments":[{"comment":"The text reads 'We collectively name our novel enhancements of ORPO as Gradient-Driven Prompt Optimizer (GDPO)'; 'ORPO' should presumably be 'OPRO', since OPRO is the optimizer being extended.","section":"Section 3.5"},{"comment":"The text states 'LLM-Diff avoids confusion in multi-subprompt prompts'; the framework is named LLM-AutoDiff elsewhere, so this is likely an inconsistent abbreviation.","section":"Section 3.3.3"},{"comment":"The notation L({Pi}) = [ i LTi(...) uses an undefined bracket symbol; clarify whether this is a union, summation, or concatenation of losses, and define the index range.","section":"Equation (1)"},{"comment":"The column 'Start Acc(%)(v, t)' is unclear; define what v and t refer to, since the same notation is later used for time steps in Eq. (8).","section":"Table 2 header"},{"comment":"The description of the HotPotQA split says '50 hard queries from training' and '7405-test subset into 100-sample validation and 200-sample test'; clarify whether the 50 training queries are distinct from the 100 validation and 200 test samples, and whether the selection is random or fixed.","section":"Section 4.1, Data splits"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is promising, but the headline claim of consistently outperforming textual-gradient baselines in multi-node settings is not supported because Text-Grad is absent from all multi-node experiments. The authors should either run Text-Grad on the RAG and agentic pipelines (which is likely feasible since AdalFlow already supports Text-Grad on single-node cases), or substantially revise the claim to only assert superiority over the non-gradient baselines tested. The training-cost claim likewise needs quantitative support. The ablations are a positive feature and suggest the framework has value even if the comparison is narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper has a real engineering contribution: a graph-based framework that propagates textual feedback through functional nodes, preserves temporal order in cyclic calls, splits sub-prompts into peers, and skips backward passes on correct samples. Implemented in AdalFlow with code links, it's the kind of thing that would help practitioners tune RAG and agent pipelines. The ablations show gradients matter most in the multi-component setups, which supports the core design intuition.\n\nThe soft spots are mostly about what is claimed versus what is actually tested. The abstract says 'consistently outperforms existing textual gradient baselines.' In the multi-node experiments that constitute the novelty—Vanilla RAG, Multi-hop RAG, Cycle, Agentic RAG—Text-Grad is never run. Text-Grad appears only on the single-LLM tasks. So the distinctive claim that the method extends textual gradients to multi-component systems and beats those baselines there is empirically unbacked. The one place a solid baseline beats it is in their own table: DsPy(2+2) hits 50.63 on Multi-hop RAG vs 48.25 for Ours. That undercuts 'consistently outperforms' even against non-gradient baselines.\n\nOther concerns: the training split is 50 hard queries for training; validation is 100 and test 200 from HotPotQA. No significance tests, and the standard deviations overlap for several key comparisons. The Agentic RAG row uses a reported DsPy* number from another paper without a direct run, so it's not a controlled comparison. The training-cost claim rests on an anecdotal token count on ObjectCount, not measured cost across multi-node workflows. And the assumption that a frozen GPT-4o gives reliable root-cause textual gradients is untested—no human eval or calibration of gradient quality, which matters because the whole loop depends on it.\n\nThat said, the novelty relative to GASO is modest but real: the specific combination of pass-through gradients, time-stamped cycles, peer sub-prompts, and error-only backprop isn't in the cited work, and the paper does cite GASO honestly. The engineering and framing are clear. A serious referee should engage with this; it's a useful tool paper, but the authors need to fix the missing baseline, add significance testing, and tone down the abstract.","headline":"Useful engineering for multi-node prompt tuning, but the headline claim that it beats textual-gradient baselines is only tested on single-node tasks.","tokens_in":20987,"tokens_out":2285,"would_cite":false,"duration_ms":21373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-AutoDiff claims that every textual input in a multi-component LLM pipeline, including retrieval functions and agent loops, can be optimized by natural-language gradients propagated backward through the workflow graph.","keywords":["automatic prompt engineering","textual gradients","prompt optimization","retrieval-augmented generation","multi-agent workflows","graph-based optimization","cycle-aware backpropagation","LLM workflows"],"falsifier":"Run the framework on a multi-hop RAG pipeline in two modes: with genuine backward-engine gradients and with deliberately shuffled or wrong attributions that always blame the final generator; if the wrong-attribution mode improves validation accuracy at the same rate, the claim that node-level root-cause feedback drives the gains would be refuted.","tokens_in":19951,"feed_emoji":"🔁","tokens_out":5976,"duration_ms":56033,"temperature":0.7,"pith_summary":"This paper tries to establish that automatic prompt optimization can be extended from a single LLM call to arbitrarily structured LLM workflows: pipelines with several LLM nodes, retrieval and formatting functions, and loops that call the same node repeatedly. Its proposal is to treat every textual input to every node as a trainable parameter, run the pipeline forward once, and let a frozen backward-engine LLM generate textual feedback, analogous to gradients, that flows backward along the dependency graph and tells a second optimizer LLM how to rewrite each sub-prompt. The paper argues that this graph-centric design fixes two gaps in prior textual-gradient methods: errors can be traced through functional nodes that have no prompt of their own, and repeated calls in loops get feedback in the correct chronological order. If correct, the result would make prompt engineering for compound systems, such as multi-hop retrieval and agent loops, an automated, gradient-like learning process rather than manual trial and error. The experiments compare against textual-gradient and few-shot baselines on classification, multi-hop retrieval QA, and agent pipelines, reporting higher accuracy at lower training cost.","feed_headline":"Whole LLM pipelines get auto-differentiated prompts","feed_subtitle":"Graph-based textual gradients let multi-hop RAG and agent loops refine every sub-prompt automatically.","key_machinery":"The central object is the textual gradient: a natural-language critique, produced by a backward-engine LLM, that says how a node's input or prompt contributed to a low score and what to change. The paper builds a dynamic parameter graph during the forward pass, recording every intermediate input, output, and sub-prompt as a node, and then runs a textual backward pass in which a frozen backward LLM reads the conversation context and the accumulated downstream feedback to emit a textual gradient for each node. An optimizer LLM proposes rewritten sub-prompts, and validation decides whether to keep them. The three named mechanisms, pass-through, time-sequential, and peer-aware gradients, are what let this loop operate on cyclic, multi-component graphs with non-trainable functional nodes.","core_discovery":"The paper's central claim is that auto-differentiation in the textual domain is not limited to a single LLM call: every textual input in a compound LLM system can be optimized by propagating natural-language feedback backward through a directed computation graph. It introduces three mechanisms: pass-through gradients that carry feedback across non-trainable functional nodes such as retrievers or deduplicators; time-sequential gradients that accumulate feedback for the same node called multiple times in a loop and keep the calls in order; and peer-aware gradients that split a prompt into separate sub-prompts, such as instructions, few-shot examples, and output format, so the backward engine can attribute errors to the precise sub-prompt. On top of this, the framework computes gradients only for error samples and uses a two-stage validation to cut cost. The author would state that, across single-step classification, multi-hop retrieval QA, and agent-driven pipelines, the approach consistently beats prior textual-gradient baselines in both accuracy and training cost.","pith_inferences":["A testable extension is to measure gradient quality directly: have human raters or a held-out judge score whether each textual gradient names the true error source, and correlate that score with downstream accuracy gains.","The same graph formalism could co-optimize non-textual knobs such as retrieval depth, chunk size, or temperature by representing them as parameter nodes whose updates are proposed in natural language; the paper lists hyperparameter co-optimization as future work.","The error-only gradient rule implies a curriculum-like effect, because the optimizer sees mostly hard samples; this might improve sample efficiency but could also overfit the optimizer to a narrow error distribution, which fresh error types would reveal.","If the framework's claims hold, it offers a practical recipe for turning prompt engineering into a declarative optimization problem, which could lower the barrier to building reliable multi-agent systems."],"forward_implications":["If the framework works as claimed, prompt optimization for RAG and agent systems no longer requires hand-tuning each module's prompt; a single backward pass locates the responsible sub-prompt.","The time-sequential gradient rule makes cycles optimizable, so multi-hop retrievers and iterative agents can be trained with the same loop as a single LLM call.","Error-only gradient computation should make training cheaper than full-batch textual methods, with the paper reporting higher token efficiency and faster convergence.","Splitting prompts into peer sub-prompts should reduce the lost-in-the-middle effect, because feedback reaches the exact instruction, format, or example that caused the error.","The framework generalizes the single-node textual-gradient paradigm, so future compound LLM applications can be designed as auto-differentiable graphs from the start."],"supporting_citations":[{"why":"Supplies the single-node textual-gradient baseline and the mini-batch training loop that this work extends.","marker":"[24]"},{"why":"Provides the LLM-as-optimizer method that the paper's Gradient-Driven Prompt Optimizer builds on and ablates against.","marker":"[21]"},{"why":"Describes the multi-node instruction-and-demonstration optimization baseline used for comparison on RAG and agent pipelines.","marker":"[11]"},{"why":"Provides the demonstrate-search-predict pipeline structure and the retrieval conventions adopted in the RAG benchmarks.","marker":"[6]"},{"why":"Motivates the time-sequential gradient mechanism by contributing the ReAct agent-loop structure.","marker":"[23]"},{"why":"Establishes the lost-in-the-middle position-bias effect that motivates splitting prompts into peer sub-prompts.","marker":"[10]"},{"why":"Supplies the multi-hop QA dataset and the F1/EM evaluation used in the HotPotQA experiments.","marker":"[22]"},{"why":"Provides an earlier textual-gradient prompt optimization approach that the paper generalizes to multi-node graphs.","marker":"[13]"}],"fun_headline_variants":["Auto-diff for all LLM calls in your workflow","Backprop for LLM pipelines: optimize every sub-prompt","Peer-aware textual gradients fix lost-in-the-middle","One backward engine to autotune all prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen backward LLM can reliably attribute each error to the right sub-prompt or predecessor from the aggregated conversation and feedback, a premise the paper does not independently calibrate or human-verify.","fun_headline_variants_meta":{"raw":{"variants":["Auto-diff for all LLM calls in your workflow","Backprop for LLM pipelines: optimize every sub-prompt","Peer-aware textual gradients fix lost-in-the-middle","One backward engine to autotune all prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000435,"raw_usage":{"total_tokens":2255,"prompt_tokens":1027,"completion_tokens":1228,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1164}},"tokens_in":643,"tokens_out":1228,"duration_ms":11746,"temperature":1.0,"reasoning_tokens":1164,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:30:24.021083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the framework on a multi-hop RAG pipeline in two modes: with genuine backward-engine gradients and with deliberately shuffled or wrong attributions that always blame the final generator; if the wrong-attribution mode improves validation accuracy at the same rate, the claim that node-level root-cause feedback drives the gains would be refuted.","supporting_citations":[{"cited_title":"Lost in the middle: How language models use long contexts","cited_arxiv_id":null,"evidence_quote":"Establishes the lost-in-the-middle position-bias effect that motivates splitting prompts into peer sub-prompts."}],"review_version":1}