Pith. sign in

REVIEW 3 major objections 6 minor 16 references

Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a two-step locate-then-optimize feedback loop can improve a role-based multi-agent system for software development on all five evaluated quality dimensions, and that the online group one-pass setting is the…

desk verdict A well-specified locator/optimizer case study whose headline gains rest on metrics scored by the same model being optimized; plausible but needs stronger evaluation. read the letter →

arxiv 2505.16086 v2 pith:SUYJZBNP submitted 2025-05-22 cs.AI cs.CL

Ming Shen , Raphael Shu , Anurag Pratik , James Gung , Yubin Ge , Monica Sunkara , Yi Zhang This is my paper
classification cs.AIcs.CL
keywords multi-agentsystemsLLM-basedagentspromptoptimizationtextualfeedbacksoftwaredevelopmentLLM-as-judgegrouprole-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that textual feedback alone can steer a role-based multi-agent system toward better software development output, without hand-tuning every agent prompt. The proposed remedy is a two-step loop: an LLM locator reads global feedback on the final code, decides which agents are responsible, and explains their failures; an LLM optimizer then rewrites only those agents' system prompts based on the explanations. The paper confronts this loop with a solver-and-reviewers agent team and evaluates it on five software-quality dimensions: functionality, robustness, test coverage, documentation, and PEP 8 style violations. Across settings, the method improves on the unoptimized system on nearly every dimension, and the best variant—online group optimization with one-pass prompting—outperforms both a one-shot baseline and a direct prompt-rewriting baseline on all five dimensions. The practical payoff is that prompt engineering for agent teams could become an automated, feedback-driven process rather than a manual one.

What carries the argument

The machinery is a two-step feedback pipeline wrapped around a critic mechanism. The critic takes a task description and the system's final code and produces both a scalar utility score and natural-language feedback for a chosen evaluation dimension; for functionality, robustness, and test coverage the critic is GPT-4-as-judge, while documentation is scored by comment and docstring line count and code style by the pycodestyle tool. A locator then takes the task, the global feedback, and the full collaboration details (role descriptions and communication trajectories) and outputs the set of underperforming agents plus a specific failure explanation for each. An optimizer rewrites the system prompt of each identified agent, keeping the rest of the prompt fixed, so that the failure explanation is addressed. The comparison settings vary where feedback comes from (online real-time vs offline pre-collected), how many agents are updated per step (individual vs group), and whether group updates occur in one joint inference call or separate calls. The system being optimized is a vertical multi-agent team in which a solver writes code and two reviewers critique it.

What would settle it

Re-run the pipeline on the same SRDD tasks with functionality, robustness, and test-coverage critics replaced by execution-based unit tests or independent human expert ratings; if the optimized prompts do not beat the unoptimized baseline under those critics, the reported improvements are artifacts of GPT-4's evaluation preferences rather than gains in software quality.

Watch

Extended reading notes

Core claim

The central claim is that the location-plus-optimization decomposition is what makes prompt optimization work in a genuine role-based multi-agent system. Instead of feeding global natural-language feedback directly into every agent prompt, the pipeline first attributes negative feedback to specific underperforming agents and generates fine-grained failure explanations for them, then updates only those agents' system prompts. Evaluated on the SRDD software requirement set with GPT-4 as the underlying model, this method improves the unoptimized system on all five evaluation dimensions in almost every optimization setting; the only miss is offline individual optimization on functionality, which trails by 0.1. The strongest configuration, online group optimization with one-pass prompting, beats both the one-shot baseline and the direct prompt-optimization baseline on all five dimensions. The paper also finds that optimizing all underperforming agents together at each step works better than optimizing one at a time, that online feedback collection is generally at least as good as offline collection, and that one-pass and multi-pass prompting perform comparably.

Load-bearing premise

The load-bearing premise is that the critic's textual feedback is an objective and reliable measure of the evaluation dimension, because for the first three dimensions that feedback comes from GPT-4 judging code, and the locator is explicitly told to assume such feedback is objective and robust, even though the optimizing agents and the critic share the same model.

Editorial extensions

If this is right

  • If the central claim holds, an LLM-based multi-agent system for software development can be tuned automatically from natural-language feedback, removing the need for manual prompt engineering and allowing non-experts to improve agent teams.
  • The best setting, online group optimization with one-pass prompting, is also the cheapest among the group variants, so practitioners can adopt the strongest configuration without extra inference calls.
  • Group optimization outperforming individual optimization suggests that simultaneous prompt updates do not cause harmful interference or overfitting in systems of this size, so groupwise update should be the default.
  • Because offline optimization is also effective, feedback can be collected in advance and reused, which opens the door to human-annotated feedback without requiring agent-environment interaction during training.
  • Starting optimization from nearly empty prompts recovers most of the performance gained from default prompts, which the paper interprets as evidence that the pipeline learns task-relevant instructions rather than merely polishing given ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because three of the five critics are GPT-4 judging code, the pipeline may be optimizing for what GPT-4 calls quality, and the portion of the gain that transfers to human judgment or execution-based tests is untested.
  • The paper does not isolate the locator's contribution; a random-agent-selection ablation would separate the benefit of attribution from the benefit of rewriting prompts.
  • A natural extension is to run the same loop on code-generation benchmarks with executable unit tests, turning the feedback into verifiable signals and likely making the optimization more stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a case study of optimizing the prompts of a role-based multi-agent system (solver and reviewer agents) for software development tasks. The authors propose a two-step pipeline: a locator that uses natural-language feedback from a critic to identify underperforming agents and explain their failures, followed by an optimizer that rewrites the system prompts of those agents. They compare online versus offline feedback collection, individual versus group optimization, and one-pass versus multi-pass prompting for group optimization. Evaluation is performed on five dimensions: functionality, robustness, test-case coverage, documentation, and code-style violation. The headline result is that online group optimization with one-pass prompting performs best, and the proposed pipeline generally outperforms a one-shot baseline and a TextGrad-style direct-optimization baseline on the tested dimensions.

Significance. If the reported gains reflect genuine software-quality improvements, this is a useful contribution to prompt optimization for role-based multi-agent systems. The paper is concrete and reproducible: the appendix includes full prompts for all agents, the locator, the optimizer, and the model-based critics. The two rule-based dimensions (documentation via comment/docstring line count, code style via pycodestyle) provide objective checks that are not subject to judge-based circularity, and the improvements on those dimensions are independently verifiable. The comparison of online/offline and individual/group settings is practically informative. However, the headline claims are weakened by the use of GPT-4 as both the judge and the optimized agent for the three most substantive dimensions, the tiny training set, the single 100-task test sample, and the absence of error bars or significance tests. The significance therefore depends on additional evaluation that anchors the three model-judged dimensions to an independent ground truth.

major comments (3)
  1. [§5.2, Figure 9, Figure 11, §5.3] The evaluation for functionality, robustness, and test-case coverage is performed by GPT-4 (Section 5.2, Figure 9), and the same gpt-4-0613 model is the backbone of all agents being optimized, as well as of the locator and the optimizer (Section 5.3). The locator prompt in Figure 11 instructs the locator to assume that 'the external feedback to the final solution is objective and robust,' but this feedback is not external: it is generated by the same model family whose prompts are being changed. The reported improvements on these three dimensions may therefore reflect optimization toward GPT-4's scoring tendencies rather than toward code that objectively satisfies the task requirements. The two rule-based dimensions provide partial grounding, but the central claim of effectiveness across all dimensions is not anchored to an independent measure. Please add external validation, such as human-annotated scores on a subset of test outputs, execution-based functional tests, or at least an evaluation with a different judge model plus a judge-human agreement analysis, and explain how the locator's assumption of objectivity can be justified without such validation.
  2. [§5.3, Table 1] All claims of improvement are based on point estimates from a single run: 5 training task descriptions and one randomly sampled 100-task test set, with no repeated seeds, confidence intervals, or significance tests. Because LLM outputs are stochastic and the training set is extremely small, the differences that drive the conclusions in Section 5.5, such as 'online and group optimizations are more effective' and 'one-pass prompting is preferred for efficiency,' may be within noise. Please report variances over multiple runs and paired significance tests for the main comparisons, or at least provide bootstrap confidence intervals for the values in Table 1, so that the comparative claims are statistically supportable.
  3. [§5.6.1, Figure 2] The conclusion that 'there is no over-fitting happening during training' is based on qualitative visual inspection of training and development curves over only 5 training examples and 30 sampled development examples, with no quantitative measure of the train-dev gap or its evolution. Please provide numerical train-dev gaps (e.g., at each optimization step) and state the development sample size explicitly in the main text; as written, the claim cannot be assessed from the figure alone.
minor comments (6)
  1. [§5.3] The random sampling of 5 training task descriptions and 100 test task descriptions should be fixed with a seed for reproducibility, and the paper should state whether the same test sample is used across all optimization settings.
  2. [Table 1] The note says the last two dimensions are 'simply positive integers,' but documentation and violation are on different scales (counts of lines and counts of violations, respectively); the table would benefit from an explicit statement that cross-dimension numerical comparisons are not meaningful.
  3. [§5.6.2, Figure 4] The column header 'Comp.' in Figure 4 is undefined; it presumably refers to code style violation or compliance, and it should be renamed to match the terminology used in the rest of the paper.
  4. [Section 3 title] The section title 'Software Developemt Task' contains a typo and should read 'Software Development Task.'
  5. [Figure 7 caption] The caption says 'online group with one-pass promoting optimization'; this should be 'prompting optimization.'
  6. [§5.6.1] The paper notes in the analysis of Figure 3 that the code style violation dimension is chosen to 'avoid potential bias raised by model-based evaluation'; this is a valuable caveat, but it should be elevated to a limitations discussion that acknowledges the same bias applies to the main results for the three model-judged dimensions.

Circularity Check

2 steps flagged · score 6.0 of 10

Closed optimization loop: prompts are optimized against the GPT-4 critic whose scores are then reported for functionality, robustness, and coverage; only the two rule-based dimensions provide independent grounding.

  1. fitted input called prediction [Section 4.1 (Problem Formulation); implemented in Sections 5.2 and 5.3]
    "Our goal is to find an optimal prompt group P∗ drawn from the natural language space such that the expectation of utility Ui is maximized over Dtrain utilizing Ii: P∗= arg max P Exi∼Dtrain[Ui]. Finally, we use the optimized agent prompt group P∗ to perform evaluation on a testing set based on utility scores generated by the same critic mechanism: Exi∼Dtest[Ui]."

    For functionality, robustness, and coverage, the utility Ui and feedback Ii are produced by GPT-4 as judge (Section 5.2), and gpt-4-0613 is also the model powering the agents being optimized (Section 5.3). The prompt group P* is selected by maximizing this same critic's utility on training tasks, and the headline improvement is this same critic's utility on test tasks. Since no execution-based or human-validated ground truth anchors these three dimensions, the reported gains show optimization toward the judge's preferences rather than independent evidence of the semantic properties named in Table 3. The held-out test set and the two rule-based dimensions limit the forcedness, so the circularity is partial.

  2. other [Appendix Figure 11 (locator prompt); Sections 5.2 and 5.3]
    "For functionality, robustness, and test case coverage dimension, we generate scores on a scale from 0 to 10 and natural language feedback at the same time using GPT-4 as judge. ... We usegpt-4-0613 version of GPT-4 as the LLM everywhere in our study. ... You can assume all agents can possibly make mistakes when writing solution code or providing feedback to code, but the external feedback to the final solution is objective and robust."

    The locator is instructed to treat the final-solution feedback as 'external' and 'objective,' but that feedback is generated by GPT-4, the same model family whose agent prompts are being optimized. The failure explanations produced by the locator therefore inherit the judge's own criteria and biases, and the optimizer then rewrites prompts to better satisfy those criteria. This closes the loop before final evaluation: for the three model-judged dimensions, the locator feedback, the prompt updates, and the reported scores all arise from the same GPT-4 judgment function, so improvements on those dimensions reflect alignment with that judge rather than an externally verified property of the generated software.

full rationale

The paper's two-step locator-optimizer pipeline is a genuine algorithmic proposal, and the held-out test split, the training/development curve analysis, and the starting-from-empty control are reasonable empirical safeguards. However, for three of the five evaluation dimensions (functionality, robustness, test coverage), the optimization signal and the reported metric are both GPT-4 as judge, with the same gpt-4-0613 model also used as the agent backbone. This is the key circular component: the objective is defined as maximizing the critic's utility, the prompts are updated using that critic's textual feedback, and the headline results are that same critic's utility on a test set. Such a loop measures alignment with the judge rather than independently verified code quality, so gains on those three dimensions are partly by construction. The documentation and code-style-violation dimensions are rule-based (comment/docstring line counts and pycodestyle), and they do improve, which gives the pipeline some independent content; this is why the score is 6 rather than higher. The overlapping-author citations (Ge et al. 2023, 2025) are background only and are not load-bearing, so they do not affect the score. Overall: partial circularity concentrated in the model-judged dimensions, with independent evidence from the rule-based dimensions.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on several hand-chosen experimental settings (5 training tasks, 5 optimization steps, 2 reviewers, temperature 0.1, 100 test samples) and on the unvalidated assumption that GPT-4 judge scores measure software quality. No external benchmark anchors the three model-based evaluation dimensions.

free parameters (5)
  • Number of training task descriptions = 5
    Randomly sampled from the SRDD training split (Section 5.3); no ablation or sensitivity analysis is reported for this choice.
  • Number of optimization steps = 5
    Hand-picked (Section 5.3); Appendix C reports that 10 steps are not beneficial, but no sweep over intermediate step counts is given.
  • Number of reviewer agents = 2
    Architecture choice in Appendix A; this value is never varied.
  • Agent temperature = 0.1
    Fixed for all LLM calls (Section 5.3); no sensitivity analysis is reported.
  • Test set sample size = 100
    Randomly sampled from the test split due to budget constraints (Section 5.3); results are point estimates without confidence intervals.
assumptions (3)
  • domain assumption GPT-4-as-judge scores are a valid and objective measure of software functionality, robustness, and test coverage.
    This is the definition of the utility function in Sections 4.1 and 5.2. The locator prompt in Figure 11 calls this model-generated feedback 'accurate external feedback', but it is not external and is not validated against human judgment or execution results.
  • domain assumption An LLM locator can correctly attribute negative global feedback to specific underperforming agents based on role descriptions and communication trajectories.
    The whole pipeline depends on the locator's attributions being correct (Section 4.2.1). No human evaluation of locator accuracy is provided.
  • domain assumption Prompt modifications learned from 5 training tasks generalize to held-out software tasks.
    The training set has only 5 task descriptions (Section 5.3). The paper assumes that the prompt edits derived from these few examples produce improvements on a separate 100-task test set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development." pith.science (2026). https://pith.science/paper/SUYJZBNP

@misc{pith2026250516086,
  author       = {Pith},
  title        = {Pith review of: Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SUYJZBNP}},
  note         = {Machine review of arXiv:2505.16086}
}
read the original abstract

We have seen remarkable progress in large language models (LLMs) empowered multi-agent systems solving complex tasks necessitating cooperation among experts with diverse skills. However, optimizing LLM-based multi-agent systems remains challenging. In this work, we perform an empirical case study on group optimization of role-based multi-agent systems utilizing natural language feedback for challenging software development tasks under various evaluation dimensions. We propose a two-step agent prompts optimization pipeline: identifying underperforming agents with their failure explanations utilizing textual feedback and then optimizing system prompts of identified agents utilizing failure explanations. We then study the impact of various optimization settings on system performance with two comparison groups: online against offline optimization and individual against group optimization. For group optimization, we study two prompting strategies: one-pass and multi-pass prompting optimizations. Overall, we demonstrate the effectiveness of our optimization method for role-based multi-agent systems tackling software development tasks evaluated on diverse evaluation dimensions, and we investigate the impact of diverse optimization settings on group behaviors of the multi-agent systems to provide practical insights for future development.

Figures

Figures reproduced from arXiv: 2505.16086 by the authors.

Figure 1
Figure 1. A high-level workflow of our two-step optimization pipeline. Please refer to § [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Training and development evaluation score curves analysis for all evaluation dimensions [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Training curve analy￾sis for code style violation eval￾uation dimension under major optimization settings. In this section, we plot the optimization curves of the average evaluation scores on the training and development set with respect to each optimization step. Due to budget limits, we are not able to plot curves for all development examples, so we randomly sample 30 examples from the development set [PITH_FULL_… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The effect of starting op￾timization from an "empty" agent prompt instead of a default prompt. Instead of starting optimization from a default prompt, we study whether our optimization pipeline is still ef￾fective when the prompt to optimize starts from empty. Note tha…
Figure 5
Figure 5. Figure 5: System and user prompt for solver agent whose responsibility is to write the main solution [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: System and user prompt for reviewer agent whose responsibility is to review the solution [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Curve analysis for doubling the training steps for functionality and code style violation [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: System and user prompt we use for the role selection agent for selecting concrete roles for [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Evaluation system and user prompt for generating utility scores and textual feedback for [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Evaluation system and user prompt for generating textual feedback for documentation [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: System and user prompt for the locator. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: System and user prompt for the optimizer. [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 12 canonical work pages

  1. [1]

    Chief executive officer whose main responsibilities include being an active decision-maker on users’ demands and other key policy issues, leader, manager, and executor

    work page

  2. [2]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Cer’on Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike

    URL https://api.semanticscholar.org/CorpusID:267413214. Nat McAleese, Rai Michael Pokorny, Juan Felipe Cer’on Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. ArXiv, abs/2407.00215, 2024. URL https://api.semanticscholar.org/CorpusID:270844127. OpenAI. Gpt-4 technical report. 2023. URL https://api.semanticscholar....

    arXiv 2024

  3. [3]

    Counselor whose main responsibilities include asking what users and customers think and providing valuable suggestions

    work page

  4. [4]

    Programmer who can write/create computer software or applications with extensive computing and coding experience

    URL https://api.semanticscholar.org/CorpusID:270380429. Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=...

    arXiv 2024

  5. [5]

    Chief human resource officer who oversees all aspects of human resource management and industrial relations policies, practices and operations for an organization

    work page

  6. [6]

    Chief product officer who is responsible for all product-related matters, including product design, product strategy, product vision, product innovation, project management, and product marketing

    work page

  7. [7]

    Code reviewer who can help programmers assess source codes for software troubleshooting, fix bugs to increase code quality and robustness, and offer proposals to improve the source codes

    work page

  8. [8]

    Chief technology officer who is very familiar with information technology and will make high-level decisions for the overarching technology infrastructure that closely aligns with the organization’s goals

    work page

Show all 16 references
  1. [9]

    [/Experts pool] You need to recruit a total of ${number of agents} experts from the above expert pool to collaboratively solve the given task

    Chief creative officer who directs the company’s creative software and develops the artistic design strategy that defines the company’s brand. [/Experts pool] You need to recruit a total of ${number of agents} experts from the above expert pool to collaboratively solve the giv...

    2025

  2. [10]

    Programmer who can write/create computer software or applications with extensive computing and coding experience in many varieties of programming languages and platforms, such as Python, Java, C, C++, HTML, CSS, JavaScript, XML, SQL, PHP, etc

  3. [12]

    Software test engineer who can use the software as intended to analyze its functional properties, design manual and automated test procedures to evaluate each software product, build and implement software evaluation test programs, and run test programs to ensure that testing ...

  4. [14]

    Also, make sure that you only give a high score when the solution code is really good at satisfying the definition of the evaluation dimension

    ${dimension}: a score (from 0 to 10) + <Reasons> detailed reasons </Reasons> Make sure your score and explanations of the score are only based on the evaluation dimension, not anything else. Also, make sure that you only give a high score when the solution code is really good ...

    2025

  5. [15]

    Agent 1: detailed explanations of why programmer leads to negative feedback

  6. [16]

    Figure 11: System and user prompt for the locator

    Agent 3: detailed explanations of why code reviewer leads to negative feedback If the feedback is completely positive, you can return a string of "None", meaning no agent is making mistakes. Figure 11: System and user prompt for the locator. 22 AI Agents: Capabilities and Safe...

    2025

  7. [2023]

    gradient descent

    URL https://api.semanticscholar.org/CorpusID:258040990. Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. GrIPS: Gradient-free, edit- based instruction search for prompting large language models. In Andreas Vlachos and Isabelle Augenstein (eds.), Proceedings of the 17t...

    2023

  8. [2024]

    Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, and Xuanjing Huang

    URL https://openreview.net/forum?id=XII0Wp1XA9. Ruotian Ma, Xiaolei Wang, Xin Zhou, Jian Li, Nan Du, Tao Gui, Qi Zhang, and Xuanjing Huang. Are large language models good prompt optimizers? ArXiv, abs/2402.02101,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.