{"id":"85342db0-b047-4521-bbf9-a6a67949fc0f","arxiv_id":"2412.03624","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semantic backpropagation sends improvement messages backward through an AI agent's computation graph, conditioning each message on neighboring nodes, and improves accuracy on BIG-Bench Hard and GSM8K compared with TextGrad, OptoPrime, and COPRO.","lead":"This paper presents a method for automatically improving the instructions that drive AI agent systems. The method computes textual improvement messages for each component, using neighboring components' outputs to make the messages more relevant, and it outperforms prior text-gradient methods on reasoning and math benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Update gate is evaluated on the same training samples used to generate gradients, so the reported gains may reflect selection on the training set rather than neighbor-aware credit assignment; a held-out validation/test evaluation is needed.","rationale":"The paper's core contribution is empirical: neighbor-aware semantic backpropagation is claimed to improve credit assignment and beat existing GASO solvers. The theoretical framework is permissive, so the evidence must carry the argument. The update gate is explicitly called essential in the Remark of Section 3.2.2, and the ablation that isolates the central mechanism—full SGD versus 'Gradient without Neighborhood' in Table 2—uses the same 50 samples as query distribution, validation set, and evaluation set. Because Algorithm 2 accepts only updates that improve LVal on those same samples, the reported gap may be driven by selecting prompts that fit those 50 samples, not by genuinely better credit assignment. The main Table 1 results inherit the same issue because the gate is computed on the training set. Lack of error bars and reliance on externally reported OptoPrime/COPRO numbers are real but secondary concerns; they would be less decisive if a held-out evaluation still showed the neighbor-aware variant winning. The reader's conditional verdict is appropriate, pending a controlled evaluation with a disjoint validation set and test set.","tokens_in":19598,"tokens_out":6294,"duration_ms":64547,"concrete_test":"On the LIAR setup, split the data: use 50 disjoint samples for query distribution and gradient generation, a separate held-out set (e.g., 50 additional samples) for LVal in the update gate, and the LIAR test split for reporting accuracy. Rerun full SGD and the 'Gradient without Neighborhood' variant for 5 seeds and compare test accuracy. If the gap collapses or reverses, the neighbor-conditioning claim is an artifact of validation-on-training. Also rerun one BBH task (e.g., Web of Lies) with the gate on a held-out split and check whether the test accuracy in Table 1 moves outside noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that the update gate in Algorithm 2 can be evaluated on the training set without invalidating the comparison. Section 5.1 states that the gate 'is computed using the training samples,' and Section 5.2 uses the same 50 LIAR samples 'as both the query distribution D and the validation set.' Algorithm 2 accepts a proposed parameter update only when LVal on that set improves. Thus the same data are used to (i) generate the semantic gradients, (ii) decide whether to accept the update, and (iii) compute the reported accuracy in Table 2. This is a selection-on-the-evaluation-set procedure. In the ablation, the comparison that is supposed to isolate neighbor-aware gradients—full SGD (71.2±3.2) versus 'Gradient without Neighborhood' (63.2±4.1)—could partly reflect the update gate overfitting to the 50 samples rather than better credit assignment. For the main BBH/GSM8K results, the gate is likewise computed on the training set, so the advantage over TextGrad/OptoPrime/COPRO in Table 1 is not established independently of this training-set selection; there are also no error bars.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes graph-based agentic system optimization (GASO) and proposes semantic backpropagation, which propagates 'semantic gradients' (directional natural-language feedback) through a computational graph while conditioning on neighboring nodes. It introduces semantic gradient descent with an update gate, shows that TextGrad is a special case of the framework, and reports experiments on GSM8K, BIG-Bench Hard, and BigCodeBench comparing with TextGrad, OptoPrime, and COPRO, plus an ablation on LIAR. A public implementation is provided.","tokens_in":19823,"tokens_out":11217,"duration_ms":97141,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it unifies reverse-mode automatic differentiation and TextGrad under a single message-passing view, and the neighbor-aware backward pass is a plausible improvement for credit assignment in language-agent graphs. The public code, the explicit algorithm listings, and the prompt-rephrasing robustness check in Appendix C.4 are strengths. However, the empirical support is weakened by the evaluation-protocol issues detailed below, so the significance of the reported gains is not yet established.","major_comments":[{"comment":"The evaluation protocol for the LIAR ablation is not specified. The text says the same 50 samples are used as the query distribution D and as the validation set for the update gate, but it never states the set on which the classification accuracy in Table 2 is computed. If that accuracy is on the same 50 samples, then the gate is selecting on the evaluation set and the reported 71.2±3.2 is a training-set number, which would invalidate the ablation as evidence for generalization and may explain why removing the gate drops accuracy to 49.2±5.0. Please state the evaluation split explicitly; if it is the training set, re-run the ablation on a held-out split.","section":"Section 5.2, Table 2"},{"comment":"No variance or significance information is provided for any entry in Table 1. The headline differences (e.g., 93.2 versus 83.9 on GSM8K, 82.5 versus 48.7 on BBH NLP) come from single runs of stochastic LLM-based optimizers and could change substantially across seeds. Report multiple seeds with standard errors or confidence intervals for every row, including BCB, where 27.8 versus 27.6 is effectively a tie and the abstract's 'outperforms' claim should be qualified.","section":"Section 5.1, Table 1"},{"comment":"The update gate is evaluated on the training samples, so the same data are used to generate semantic gradients, to accept or reject candidate updates, and (in the LIAR experiment) to compute the reported accuracy. This is a selection-on-training procedure. Because the Remark in Section 3.2.2 identifies the gate as essential and Section 5.1 states that the gate 'is computed using the training samples,' the method's advantage over TextGrad and OptoPrime may reflect the availability of a labeled training set for model selection rather than neighbor-aware credit assignment. The authors should re-run the main comparisons with a held-out validation set for the gate, or provide evidence that training-set gating does not bias the comparison.","section":"Section 3.2.2, Algorithm 2, Section 5.1"},{"comment":"The claim that TextGrad is a special case of semantic backpropagation is central to the paper's motivation, but the formal statement is incomplete. Equation (3) quantifies u ambiguously ('for all u ∈ V, w ∈ Successors(v), and u ∈ Predecessors(w)'), and no derivation is given showing that TextGrad's backward pass satisfies these equations. Please provide a precise statement of the conditions and a mapping from TextGrad's actual algorithm to the notation.","section":"Section 3.3, Eqs. (3) and (4)"}],"minor_comments":[{"comment":"The sentence 'This proposal may be integrated or rejected depending on the (as introduced in Section 3.2iteration regardless.' is garbled; please rewrite it.","section":"Section 5.1, paragraph after Table 1"},{"comment":"Algorithm 2 uses a strict inequality in the gate condition while the text defines the condition with '≤'; please make the two consistent.","section":"Section 3.2.2"},{"comment":"The citation 'APE (Zhou et al., 2023)' points to the reference 'Large language models are human-level prompt engineers' (OPRO), not to the APE paper; please correct the citation.","section":"Section 4.1"},{"comment":"The rows marked with † are external results from Cheng et al. (2024); since the authors state they could not reproduce them, these rows should be clearly labeled as reported values and the potential differences in setup should be discussed.","section":"Table 1"},{"comment":"The abstract and conclusion say the method 'outperforms' existing methods, but Table 1 shows 27.8 vs 27.6 on BCB, which the text itself calls 'roughly equal'; please qualify the summary claims accordingly.","section":"Abstract and Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning venue and the conceptual framework is interesting. The main barrier is the evaluation protocol: the LIAR ablation appears to report training-set accuracy, and the main table lacks error bars. If the authors supply held-out evaluation and variance estimates, the contribution could be accepted; otherwise the empirical claims are not supported. I would not base a rejection on the informality of the semantic-gradient definition, as the paper's contribution is primarily conceptual and empirical."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper makes a real conceptual point: semantic gradients as directional information generalize both RMAD and TextGrad, and the explicit reduction of TextGrad to a special case via Equations (3)-(4) is a genuinely useful framing. The neighbor-aware conditioning is the concrete new ingredient, and the LIAR ablation gives credible evidence that it matters: full SGD (71.2) beats the no-neighborhood variant (63.2) with both using the same update gate, so that comparison is internally clean. The code is public, the ablation covers several variants, and the paper is honest about the gate being essential.\n\nThe soft spots are mostly empirical. The LIAR table reports accuracy on the same 50 samples used as both the query distribution and the validation set, so those numbers are essentially training accuracy, not a holdout evaluation. The main results in Table 1 have no error bars or multiple seeds, and the baseline comparisons are not fully controlled: TextGrad needed prompt modifications to work, and the OptoPrime/COPRO numbers are lifted from another paper. The stress-test note about the update gate being computed on training samples is partially right but overstated for Table 1: the test sets for GSM8K and BBH are held out, so the gate is doing model selection on the training set, which is not the same as evaluating on the test set. That said, it can still inflate reported accuracy if the training set is small, and the paper should show that the advantage persists with a held-out validation set for the gate.\n\nThe conceptual contribution and the internal ablation are strong enough that this deserves a serious referee. The authors need to add held-out evaluation for the LIAR study, report standard errors for Table 1, and provide a more careful baseline comparison, ideally running OptoPrime and COPRO under the same conditions. I would not desk-reject this.","headline":"A useful formalization of semantic backprop with a convincing neighbor-conditioning ablation, but the evaluation protocol needs tightening before the headline numbers can be trusted.","tokens_in":20355,"tokens_out":2678,"would_cite":true,"duration_ms":27075,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Semantic backpropagation that conditions backward feedback on neighbors outperforms TextGrad and related optimizers on BIG-Bench Hard and GSM8K, the paper shows.","keywords":["semantic backpropagation","semantic gradients","graph-based agentic system optimization","prompt optimization","credit assignment","large language models","TextGrad","semantic gradient descent"],"falsifier":"Run semantic gradient descent and TextGrad on a fresh multi-component LLM task with identical forward and backward language models, iteration counts, and optimizers, but with the backward messages forced to ignore neighbor nodes; if the no-neighbor variant matches or exceeds the neighbor-conditioned variant's accuracy on a held-out test set, the central claim that neighbor conditioning improves credit assignment is refuted.","tokens_in":19405,"feed_emoji":"🤖","tokens_out":8692,"duration_ms":65758,"temperature":0.7,"pith_summary":"The paper claims that credit assignment in language-based agentic systems—multi-component programs that call LLMs—has been done incorrectly by existing backpropagation-inspired optimizers. It formalizes semantic backpropagation, which passes backward messages that depend on the receiving node and on its neighbors, and introduces semantic gradient descent, an optimizer that uses these messages to update prompt parameters. On BIG-Bench Hard and GSM8K, the method outperforms TextGrad, OptoPrime, and COPRO, suggesting that neighborhood-conditioned feedback is a genuine improvement for optimizing such systems. If correct, this would substantially reduce the manual effort needed to tune agentic systems and would establish that reverse-mode-style credit assignment carries over to natural-language computational graphs.","feed_headline":"Neighbor-aware feedback beats TextGrad on LLM agent optimization","feed_subtitle":"Semantic gradient descent scores 93.2 on GSM8K and 82.5 on BBH NLP, topping existing optimizers.","key_machinery":"The central object is the semantic gradient $\\nabla_v l_Q$, a string-valued directional message indicating how variable $v$ should change to improve the system output for query $Q$. Semantic backpropagation computes it by applying backward functions $\\hat{h}^v_w$ to each successor $w$, where $\\hat{h}^v_w$ receives $\\mathrm{Predecessors}(w)$, $w$, and $\\nabla_w l_Q$, and thus depends on both the target variable $v$ and its neighbors; the aggregator $A_v$ then combines the messages from all successors. This machinery generalizes the chain rule in reverse-mode automatic differentiation, and reduces to TextGrad when the backward functions ignore $v$ and the other predecessors. The argument's force comes from making this neighbor conditioning explicit: it allows complementary and synergistic roles of neighboring nodes to shape each variable's update direction.","core_discovery":"The central claim is that the correct generalization of backpropagation to language-based computational graphs is semantic backpropagation, which computes a semantic gradient for each variable by aggregating backward messages from its successors, where each message explicitly conditions on the variable in question and on the other predecessors of the successor. This differs from TextGrad, whose backward functions ignore the receiving variable and its neighbors; the paper argues these independence assumptions are unjustified and empirically harmful. Formally, semantic gradients are directions for how to change a variable to improve the system output, and semantic gradient descent applies them through an LLM-based update function gated by a validation-set comparison. The paper reports that this method reaches 93.2 on GSM8K, 82.5 on BBH NLP, and 85.6 on BBH Algorithmic, outperforming TextGrad and OptoPrime, while ablation on LIAR shows that removing neighborhood conditioning, gradients, or the update gate each degrades performance.","pith_inferences":["If neighbor conditioning is the operative mechanism, then the same principle should transfer to other credit-assignment schemes for LLM pipelines, such as edge optimization or structural changes, where local context also matters.","The success of the gate suggests that a lightweight predictor of validation performance might replace full evaluation and preserve the gains at lower cost.","The formalization invites a unified view of automatic differentiation and LLM-as-optimizer methods, in which any semantically interoperable message passing in reverse topological order counts as semantic backpropagation.","If the method generalizes, it could make end-to-end optimization of full agentic systems beyond question answering feasible, including code-generation pipelines and tool-using agents where multi-component credit assignment is currently manual."],"forward_implications":["Backpropagation-style credit assignment transfers to natural-language computational graphs, provided backward messages are conditioned on the full local neighborhood.","The independence assumptions in TextGrad—messages independent of the receiving variable and of sibling predecessors—are not just theoretically questionable but empirically harmful.","Automatic optimization of multi-component LLM systems becomes feasible with modest compute, since forward passes dominate the cost while backward passes and optimizer calls are few.","The update gate is essential: accepting every LLM-proposed prompt change can cause the system to drift to worse regions, which is why semantic gradient descent rejects updates that fail to improve on the validation set."],"supporting_citations":[{"why":"Baseline TextGrad that the paper critiques and compares against on GSM8K, BBH, and BCB.","marker":"Yuksekgonul et al., 2024"},{"why":"Baseline OptoPrime (Trace) that uses subgraph feedback; the paper compares against it and reproduces its reported results.","marker":"Cheng et al., 2024"},{"why":"Provides the COPRO baseline from DSPy for comparison on BBH.","marker":"Khattab et al., 2024"},{"why":"Reverse-mode automatic differentiation that semantic backpropagation generalizes to arbitrary natural-language functions.","marker":"Linnainmaa, 1970; 1976"},{"why":"Origin of textual gradients for prompt optimization, which the paper extends to full computational graphs.","marker":"Pryzant et al., 2023"},{"why":"GSM8K dataset used to evaluate the method against baselines.","marker":"Cobbe et al., 2021"},{"why":"BIG-Bench Hard dataset used to measure performance across 23 tasks and 27 subtasks.","marker":"Suzgun et al., 2023"},{"why":"LIAR dataset used in the ablation study to show that removing each key component degrades performance.","marker":"Wang, 2017"}],"fun_headline_variants":["Semantic backprop with neighbor context beats TextGrad","Neighbor-aware semantic gradients outperform TextGrad on agents","Semantic gradient descent with neighbor info tops TextGrad","New semantic backprop conditions on neighbors, bests TextGrad"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's gains rest on the assumption that evaluating proposed updates on the same training samples used for optimization—rather than on a held-out validation set—is enough to prevent the update gate from accepting harmful prompt changes and to keep the reported results representative.","fun_headline_variants_meta":{"raw":{"variants":["Semantic backprop with neighbor context beats TextGrad","Neighbor-aware semantic gradients outperform TextGrad on agents","Semantic gradient descent with neighbor info tops TextGrad","New semantic backprop conditions on neighbors, bests TextGrad"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3200,"prompt_tokens":992,"completion_tokens":2208,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2139}},"tokens_in":608,"tokens_out":2208,"duration_ms":15280,"temperature":1.0,"reasoning_tokens":2139,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:25:11.974850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run semantic gradient descent and TextGrad on a fresh multi-component LLM task with identical forward and backward language models, iteration counts, and optimizers, but with the backward messages forced to ignore neighbor nodes; if the no-neighbor variant matches or exceeds the neighbor-conditioned variant's accuracy on a held-out test set, the central claim that neighbor conditioning improves credit assignment is refuted.","supporting_citations":[],"review_version":1}