Pith. sign in

REVIEW 4 major objections 5 minor 6 cited by

ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ScoreFlow claims that LLM agent workflows can be automated by fine-tuning a code-writing generator with score-weighted preference pairs, reaching 85.3% average solve rate and letting smaller models beat larger ones at lower cost.

desk verdict A useful incremental method with a serious unresolved data-split question; the theory is overclaimed but the empirical comparison is the real issue. read the letter →

arxiv 2502.04306 v1 pith:V4DEETW7 submitted 2025-02-06 cs.CL

Yinjie Wang , Ling Yang , Guohao Li , Mengdi Wang , Bryon Aragam This is my paper · ORCID
classification cs.CL
keywords LLMmulti-agentworkflowsworkflowoptimizationscore-basedpreferencedirectcoderepresentationautomaticgenerationmathematicalreasoningquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ScoreFlow claims that the labor of designing multi-agent LLM workflows can be automated by treating workflow generation as code generation and optimizing the generator with a score-aware version of direct preference optimization. Rather than searching over discrete workflow structures, the method samples several workflows per task, executes them, scores the results, and fine-tunes the generator on preference pairs weighted by score gaps and rescaled rewards. On six benchmarks covering question answering, coding, and mathematical reasoning, ScoreFlow reports an average solve rate of 85.3%, which the paper reports as an 8.2% improvement over existing baselines. The practical payoff would be that smaller, cheaper models can outperform larger ones while lowering inference and optimization cost.

What carries the argument

The mechanism is the Score-DPO loss $$L_{\mathrm{Score-DPO}} = -\mathbb{E}_{(w,l)\sim P^*}\left[\log\$\sigma$\left(f(s_w)r_w - (1-f(s_l))r_l\right)\right],$$ with $r_y=\beta\log(\pi_\theta(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))$ and sampling distribution $P^*(w,l)\propto d(s_w,s_l)P(w,l)$, where $f$ and $d$ are strictly increasing in the score and score gap respectively (defaults $f(x)=x$, $d(x,y)=(x-y)^3$). The loss is what carries the argument: it up-weights confident preference pairs and scales each item's reward by its score, so that Theorem 3.2 can show per-sample influence grows with score under the condition $-(1-f(s_z))^{-1}\le r_z\le f^{-1}(s_z)$. The surrounding pipeline iteratively generates $k=8$ workflows per task, executes them with an independent executor, and fine-tunes the code-writing generator on the resulting scored pairs.

What would settle it

Log the pair $(s_z, r_z)$ for every sampled workflow during a ScoreFlow run; if the measured per-sample influence is not increasing in $s_z$ whenever the exact condition $-(1-f(s_z))^{-1}\le r_z\le f^{-1}(s_z)$ fails, while $|r_z|\le1$ still holds, then the paper's sufficient condition is shown not to guarantee the mechanism on those samples.

Watch

Extended reading notes

Core claim

Score-DPO is the paper's central proposal: it takes the standard DPO objective and injects the quantitative evaluation score $s$ into both the sampling distribution and the ranking target. Preference pairs $(w,l)$ are sampled with probability proportional to $d(s_w,s_l)$, and the implicit reward gap becomes $f(s_w) r_w - (1-f(s_l)) r_l$, so pairs with decisive scores dominate the gradient. Theorem 3.2 states that, under a range condition on $r_z$, the per-sample influence of a workflow on the loss is strictly increasing in its score, which the paper argues reduces the harm of noisy evaluation scores that slows DPO. Empirically, the full ScoreFlow pipeline, which generates eight workflows per problem per iteration, executes them, and runs three Score-DPO fine-tuning iterations, outperforms the baselines and, on HumanEval, lets small executors beat the chain-of-thought outputs of a larger model.

Load-bearing premise

The load-bearing premise is that the range condition in Theorem 3.2 holds during optimization so that each sample's influence on the loss rises with its evaluation score; the paper verifies only the sufficient proxy $|r_z|\le1$, which it reports holds with probability about 91.1% before convergence, leaving the remaining samples without a proven monotonicity guarantee.

Editorial extensions

If this is right

  • Automated workflow optimization shifts from discrete search over structures to gradient-based fine-tuning of the generator, which the paper shows scales to a combined dataset of math, coding, and question-answering problems better than the discrete-search baseline.
  • Score-DPO can be applied wherever a scalar score, not just a binary preference, is available, so any evaluation metric that can be computed per output can feed the loss.
  • Because the generator is an open-source 8B model and converges within two to three iterations, the optimization and inference costs are below those of methods that repeatedly call a large optimizer model.
  • Smaller executors such as GPT-4o-mini and DeepSeek-V3, once wrapped in a ScoreFlow-optimized workflow, beat the chain-of-thought output of the larger GPT-4o on HumanEval, implying that workflow quality can matter more than raw model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The score-weighting recipe is not tied to workflow generation: the same loss could be applied to any preference-tuning task with continuous rewards, such as human quality ratings or verifier scores, though the paper only demonstrates it on six agentic benchmarks.
  • The approach suggests a natural curriculum: because pairs with larger score gaps are up-weighted first, later iterations could anneal $d(x,y)$, for example from cubic toward linear, to transition from coarse to fine preference learning.
  • Since the generator optimizes per-task workflows, the method could be extended to reuse trained workflows across similar tasks, amortizing the one-time optimization cost over a whole dataset; the paper does not test transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ScoreFlow proposes an automated framework that generates per-task LLM multi-agent workflows expressed as Python code. The workflow generator, a small open-source LLM (default Llama-3.1-8B-Instruct), is iteratively fine-tuned with Score-DPO, a variant of DPO that uses quantitative evaluation scores through an upweighting function d(x,y) and a score-scaling function f(x). The paper reports an average solve rate of 85.3% across six benchmarks, an 8.2% improvement over baselines, and shows that smaller executors can outperform larger ones after workflow optimization. It also contains a theoretical analysis claiming that Score-DPO makes per-sample influence monotonically increasing with the evaluation score under a stated condition.

Significance. If the empirical claims hold, the paper makes a practically valuable contribution: it reduces the cost of workflow optimization by using a small open-source generator, demonstrates gains across QA, coding, and math benchmarks, and includes comparisons against SFT, PPO, DPO, AFlow, and ADAS. The cost analyses in Figure 4 and the cross-model ablations in Table 3 are useful, and the idea of incorporating quantitative scores into DPO is likely to be of independent interest. However, two load-bearing points need attention before the claims can be accepted: the possibility that optimization used test data, and an inconsistency between the stated sufficient condition in Theorem 3.2 and the default choice f(x)=x. The absence of error bars further weakens the empirical claims.

major comments (4)
  1. [A.3.4, Table 5 and Algorithm 1] The caption of Table 5 reads '(optimization process on test data).' Section 4.1 says the data are split 1:4 into validation and test, and Tables 1-3 report test-set solve rates. Algorithm 1 consumes its input dataset D for preference collection and generator fine-tuning. If the optimization cost experiment was run on the test split, then the workflows used to compute the final test numbers were optimized on test queries, which would invalidate the headline results regardless of the theoretical analysis. The paper never states that only the validation split was used for optimization; please state explicitly which split feeds Algorithm 1 for every reported experiment, and if any optimization step used test data, recompute all test-set results with optimization confined to the validation split.
  2. [3.5, Theorem 3.2 and Appendix A.1] As printed, the theorem states the condition as -(1 - f(s_z))^{-1} <= r_z <= f^{-1}(s_z). With the default f(x)=x, f^{-1}(s_z)=s_z, so the condition is -(1 - s_z)^{-1} <= r_z <= s_z. This is not implied by |r_z| <= 1; for example, s_z=0.1 and r_z=0.5 satisfy |r_z|<=1 but violate r_z <= s_z. The proof's use of Lemma A.1 suggests the intended upper bound may be 1/f(s_z) rather than f^{-1}(s_z), and with that reciprocal interpretation |r_z|<=1 would be sufficient, but the notation in the theorem statement is then incorrect. As written, the theorem is not established for the default function, and Appendix A.3.3 verifies only P(|r_z|<=1), not the condition stated in the theorem. The authors should correct the condition and report the empirical frequency of the actual condition used.
  3. [4.2, Tables 1-3] All reported results are averages over three evaluations without standard deviations, confidence intervals, or significance tests. The central claim of an 8.2% average improvement and several per-dataset margins (for example, DROP at 86.2 vs 83.5 and MBPP at 84.7 vs 82.9) might be within run-to-run variation. The paper should provide variance estimates or significance tests for the main comparison against AFlow and against the DPO ablation; without these, the empirical strength of the claim is difficult to assess.
  4. [4.1, Metrics and Evaluation Scores] For MATH, DROP, and HotpotQA the final judge is GPT-4o-mini, which is also the model used as the executor during optimization. This creates a potential confound: part of the reported improvement may reflect agreement with the judge model rather than correctness on the underlying task. The paper should acknowledge this limitation, provide a small human-verified or rule-based evaluation for these datasets, or report judge-agreement statistics to show the result is not driven by judge bias.
minor comments (5)
  1. [4.1] There is a typo: 'validaton' should be 'validation.'
  2. [A.2.3, Algorithm 1] The Input list numbers the items incorrectly: there are two items labeled '5).' Please renumber the inputs.
  3. [Table 1 caption] The caption says 'All methods are executed using GPT-4o-mini,' but ScoreFlow uses Llama-3.1-8B-Instruct as the generator for the main results. Please clarify that GPT-4o-mini is the executor, not the generator for ScoreFlow, to avoid confusion.
  4. [A.2.3, Algorithm 1 and 4.1] Several training hyperparameters are not reported: LoRA rank and alpha, learning rate, batch size, and the number of Score-DPO update steps per iteration. These are needed to reproduce the method.
  5. [3.4] The notation P^*(w,l) is used both for the weighted sampling distribution and, in Algorithm 1, for the sampled preference loss; please define the relationship to P(w,l) explicitly in the algorithm listing.

Circularity Check

1 steps flagged · score 6.0 of 10

Appendix Table 5 states optimization was run on test data, so the reported HumanEval test-set results are potentially fitted rather than predicted; otherwise the Score-DPO derivation is not circular.

  1. fitted input called prediction [Appendix A.3.4, Table 5 caption; Section 4.1 data split; Table 1/Table 3 HumanEval; Algorithm 1]
    "The detailed cost value ($) in Figure 4b (optimization process on test data). ... We split the data into validaton and test set using a 1:4 ratio."

    The central empirical claim is a test-set solve rate (Table 1: HumanEval 95.9; Table 3: GPT-4o-mini Ours 95.7) obtained after Algorithm 1's preference-collection and Score-DPO fine-tuning loop. Section 4.1 presents the test split as the held-out evaluation split, but Table 5's caption states that the optimization process was run on test data, at least for the Figure 4b/4a cost experiments on HumanEval. The paper never states that these test-optimized runs were excluded from the reported solve rates. If they were not, the generator was fitted to the same test queries on which pass@1 is reported, so the HumanEval prediction is statistically forced rather than held out, and the headline 8.2% improvement is partially an artifact of fitting to the evaluation split.

full rationale

Aside from the test-data issue, ScoreFlow's Score-DPO claim is not circular: the loss is a heuristic objective, the theorem is an algebraic property of the chosen weighting, and the main comparisons use separate baselines and held-out splits as described. The theorem's stated sufficient condition |rz| <= 1 is not actually sufficient for the condition - (1-f(sz))^{-1} <= rz <= f^{-1}(sz) with f(x)=x, since that requires rz <= sz, but this is a correctness flaw, not a circularity. There are no load-bearing self-citations; references to the authors' other work are general capability citations. The Appendix Table 5 caption, however, explicitly says optimization was performed on test data. If that applies to the models whose test-set scores are reported, the empirical prediction reduces to fitting the evaluation data, warranting the elevated circularity-contamination score. Since the paper does not disambiguate, this must be resolved before accepting the headline numbers.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. Score-DPO is a training objective; the operators are inherited from AFlow. The free parameters are the scoring and sampling functions plus stop/search hyperparameters, all chosen by hand or by small ablations.

free parameters (5)
  • f(x): score scaling function = f(x)=x
    Used to scale winner and loser implicit rewards in Score-DPO. Chosen as identity; only monotonicity is required, and results may vary with other monotone functions.
  • d(x,y): preference upweighting function = (x-y)^3
    Weights sampling probability of preference pairs by score difference. Ablation with exponents 2, 3, 100 shows performance depends strongly on this choice.
  • Number of iterations M = 3
    Stops after three generator updates; experiments show convergence by iteration 2, but this is not rigorously justified.
  • Workflows per problem k = 8
    Number of candidate workflows sampled per problem to construct preference pairs; affects both data diversity and optimization cost.
  • Preference samples per iteration S = 2000 (600 for HumanEval)
    Number of sampled pairs for each Score-DPO update; set based on dataset size, not ablated.
assumptions (4)
  • standard math Bradley-Terry model of pairwise preferences
    Underlies DPO and Score-DPO objectives; assumes the probability of preferring one response over another is logistic in reward difference.
  • domain assumption Evaluation scores are comparable across generated workflows
    Preference pairs are built by comparing scores; if scores are noisy, inconsistent, or biased, the preference data is corrupted and the optimization signal degrades.
  • domain assumption The provided operator set and code template are sufficient to express high-quality workflows
    The search space is restricted to predefined operators; methods requiring different agent types or communication patterns are outside the space.
  • ad hoc to paper The condition in Theorem 3.2 holds sufficiently often
    The theorem requires -1/(1-f(s_z)) <= r_z <= f^{-1}(s_z); the paper only verifies the claimed sufficient condition |r_z|<=1 on three datasets with 91.1% probability, but this sufficiency is not correct for f(x)=x.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization." pith.science (2026). https://pith.science/paper/V4DEETW7

@misc{pith2026250204306,
  author       = {Pith},
  title        = {Pith review of: ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V4DEETW7}},
  note         = {Machine review of arXiv:2502.04306}
}
read the original abstract

Recent research has leveraged large language model multi-agent systems for complex problem-solving while trying to reduce the manual effort required to build them, driving the development of automated agent workflow optimization methods. However, existing methods remain inflexible due to representational limitations, a lack of adaptability, and poor scalability when relying on discrete optimization techniques. We address these challenges with ScoreFlow, a simple yet high-performance framework that leverages efficient gradient-based optimization in a continuous space. ScoreFlow incorporates Score-DPO, a novel variant of the direct preference optimization method that accounts for quantitative feedback. Across six benchmarks spanning question answering, coding, and mathematical reasoning, ScoreFlow achieves an 8.2% improvement over existing baselines. Moreover, it empowers smaller models to outperform larger ones with lower inference costs. Project: https://github.com/Gen-Verse/ScoreFlow

Figures

Figures reproduced from arXiv: 2502.04306 by the authors.

Figure 1
Figure 1. Pipeline of ScoreFlow. First, for each problem in the dataset, multiple workflows are gen￾erated. Next, an executor is employed to execute these workflows for corresponding problems, resulting in evaluation scores. Based on these scores, preference data is collected. Subsequently, incorporating the score information, the Score-DPO algorithm is used to fine-tune the generator. This process is iterated until the maxim… view at source ↗
Figure 2
Figure 2. Illustration of the inference process: Two distinct workflows are generated for two GSM8K [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison between ScoreFlow and Aflow across various datasets. The y-axis [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: API Cost in Inference and Optimization processes. We analyze the API cost during both [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Solve rate during iteration process. 5 Conclusion In this work, we propose ScoreFlow, an automated, high-performance, and adaptive framework for optimizing multi-agent workflows. The framework leverages the generalizable Score-DPO to achieve robust and efficient optimi…
Figure 6
Figure 6. Figure 6: The distribution of sample implicit reward during optimization process before convergence [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: The distribution of sample implicit reward during optimization process before convergence [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]
Figure 8
Figure 8. Figure 8: The distribution of sample implicit reward during optimization process before convergence [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CONTRA: Red-Teaming Configurations of Personalizable Agents

    cs.CR 2026-07 conditional novelty 7.0 of 10

    An LLM-assisted configuration tree search finds benign agent setups that trigger malicious actions for 75% of popular skills, showing personalization itself is a major safety risk.

  2. A Workflow-Aware Serving Layer for Agentic Applications

    cs.DC 2026-07 conditional novelty 6.5 of 10

    A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.

  3. Rethinking Query Optimization for Multi-Agent Systems [Vision]

    cs.DB 2025-12 conditional novelty 6.0 of 10

    Agentic data pipelines are built by hand today; this paper sets a research agenda for automatically optimizing their structure, model choices, and execution engines jointly as a new query-optimization problem.

  4. Reward-Free Evolving Agents via Pairwise Validator

    cs.AI 2026-07 conditional novelty 5.0 of 10

    A frozen LLM making binary parent-vs-child comparisons can replace the scalar reward in self-evolving agent loops, matching or beating reward-gated evolution on most settings.

  5. FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills

    cs.AI 2026-04 conditional novelty 5.0 of 10

    FlowEvo compiles successful agent workflows into executable skill records and reuses them at inference time, reporting the best accuracy-cost tradeoff across ALFWorld, HumanEval, and GSM8K among tested baselines.

  6. MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Encoding LLM agent workflows as statically checked Mermaid graphs and evolving them with constrained mutations improves solve rates on GSM8K, MATH, HumanEval, and MBPP.

Reference graph

Works this paper leans on

56 extracted references · 21 canonical work pages · cited by 6 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  4. [4]

    A general theoretical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics , pages 4447–4455. PMLR, 2024

    work page 2024

  5. [5]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952

    1952

  6. [6]

    Preference learning algorithms do not learn preference rankings

    Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. arXiv preprint arXiv:2405.19534, 2024

    arXiv 2024

  7. [7]

    Autoagents: A framework for automatic agent generation

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, B¨ orje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 , 2023

    arXiv 2023

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

Show all 56 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021

    arXiv

  2. [10]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019

    2019

  3. [11]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt¨ aschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023

    2023 arXiv

  4. [12]

    Data interpreter: An llm agent for data science

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, et al. Data interpreter: An llm agent for data science. arXiv preprint arXiv:2402.18679, 2024

    2024 arXiv

  5. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  6. [14]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. In NeurIPS 2024 Workshop on Open-World Agents , 2024

    2024

  7. [15]

    The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization

    Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl; dr summarization. arXiv preprint arXiv:2403.17031 , 2024. 10

    2024 arXiv

  8. [16]

    Self-[in] correct: Llms struggle with refining self-generated responses

    Dongwei Jiang, Jingyu Zhang, Orion Weller, Nathaniel Weir, Benjamin Van Durme, and Daniel Khashabi. Self-[in] correct: Llms struggle with refining self-generated responses. arXiv preprint arXiv:2404.04298, 2024

    2024 arXiv

  9. [17]

    Dspy: Compil- ing declarative language model calls into state-of-the-art pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, Heather Miller, et al. Dspy: Compil- ing declarative language model calls into state-of-the-art pipelines. In The Twelfth International ...

    2024

  10. [18]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  11. [19]

    Autoflow: Automated workflow generation for large language model agents

    Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. arXiv preprint arXiv:2407.12821, 2024

    2024 arXiv

  12. [20]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  13. [21]

    A dynamic llm-powered agent network for task-oriented agent collaboration

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling , 2024

    2024

  14. [22]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems , 36, 2024

    2024

  15. [23]

    Simpo: Simple preference optimization with a reference- free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. arXiv preprint arXiv:2405.14734 , 2024

    2024 arXiv

  16. [24]

    Can generalist foundation models outcompete special-purpose tuning? case study in medicine

    Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 , 2023

    2023 arXiv

  17. [25]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  18. [26]

    Disentangling length from quality in direct preference optimization

    Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics (ACL 2024), 2024

    2024

  19. [27]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024

    2024

  20. [28]

    Code generation with alphacodium: From prompt engineering to flow engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500 , 2024

    2024 arXiv

  21. [29]

    Archon: An architecture search framework for inference-time techniques

    Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher R´ e, et al. Archon: An architecture search framework for inference-time techniques. arXiv preprint arXiv:2409.15254 , 2024

    2024 arXiv

  22. [30]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017

    2017 arXiv

  23. [31]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024. 11

    2024

  24. [32]

    Adaptive in-conversation team building for language model agents

    Linxin Song, Jiale Liu, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, and Chi Wang. Adaptive in-conversation team building for language model agents. arXiv preprint arXiv:2405.19425, 2024

    2024 arXiv

  25. [33]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023

    2023 arXiv

  26. [34]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. The Eleventh International Conference on Learning Representations , 2022

    2022

  27. [35]

    Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self-collaboration

    Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- persona self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter o...

    2024

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  29. [37]

    Is dpo superior to ppo for llm alignment? a comprehensive study

    Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. In Proceedings of the 41st International Conference on Machine Learning (ICML) , 2024

    2024

  30. [38]

    Lemur: Harmonizing natural language and code for language agents

    Yiheng Xu, SU Hongjin, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representations , 2024

    2024

  31. [39]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayi- heng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  32. [40]

    Large language models as optimizers

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409 , 2023

    2023 arXiv

  33. [41]

    Buffer of thoughts: Thought-augmented reasoning with large language models

    Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonza- lez, and Bin Cui. Buffer of thoughts: Thought-augmented reasoning with large language models. Advances in Neural Information Processing Systems , 2024

    2024

  34. [42]

    Supercorrect: Supervising and correcting language models with error-driven insights

    Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Supervising and correcting language models with error-driven insights. arXiv preprint arXiv:2410.09008, 2024

    2024 arXiv

  35. [43]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhiyu Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- ...

    2018

  36. [44]

    Textgrad: Automatic ”differentiation” via text

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic ”differentiation” via text. arXiv preprint arXiv:2406.07496, 2024

    2024 arXiv

  37. [45]

    G-designer: Architecting multi-agent communication topologies via graph neural networks

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782 , 2024

    arXiv

  38. [46]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762 , 2024

    2024 arXiv

  39. [47]

    Achiev- ing¿ 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners.arXiv preprint arXiv:2404.14963, 2024

    Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. Achiev- ing¿ 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners.arXiv preprint arXiv:2404.14963, 2024. 12

    2024 arXiv

  40. [48]

    Symbolic learning enables self-evolving agents

    Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406.18532 , 2024

    2024 arXiv

  41. [49]

    "" This is a wor kf low graph

    Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, R´ obert Csord´ as, Anand Gopalakr- ishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17066 ...

    2023

  42. [50]

    Format MUST follow : custom ( i n s t r u c t i o n : str ) -> str You can modify the i n s t r u c t i o n prompt

    Custom : Usage : G e n e r a t e s a nyt hi ng based on fixed input problem and m o d i f i a b l e i n s t r u c t i o n . Format MUST follow : custom ( i n s t r u c t i o n : str ) -> str You can modify the i n s t r u c t i o n prompt . The output can serve as the input of...

  43. [51]

    C u s t o m C o d e G e n e r a t e : Usage : G e n e r a t e s code based on c u s t o m i z e d input i n s t r u c t i o n . Format MUST follow : c o d e _ g e n e r a t e ( i n s t r u c t i o n : str ) -> str The i n s t r u c t i o n should e n c o u r a g e op er ato r ...

  44. [52]

    Format MUST follow : a n s w e r _ g e n e r a t e () -> str For example : so lu tio n = await self

    A n s w e r G e n e r a t e : Usage : Di rec tl y ge ne rat e answer ( i n c l u d i n g thought ) to the given problem . Format MUST follow : a n s w e r _ g e n e r a t e () -> str For example : so lu tio n = await self . a n s w e r _ g e n e r a t e () The output can serve...

  45. [53]

    P r o g r a m m e r : Usage : A u t o m a t i c a l l y writes , ex ec ute s Python code , and returns the final sol ut ion based on the pro vi ded problem d e s c r i p t i o n and an aly si s . Format MUST follow : p r o g r a m m e r ( an al ysi s : str = ’ None ’) -> str T...

  46. [54]

    S c E n s e m b l e : Usage : Ev alu at e every solutions , then select the best so lu tio n in the s olu ti on list . Format MUST follow : s c _ e n s e m b l e ( s o l u t i o n s : List [ str ]) -> str You can en se mbl e few solutions , for example : e n s e m b l e d _ s ...

  47. [55]

    Review : Usage : Given p re vio us solution , Review op era to r reviews the p rev io us s ol uti on to r e g e n e r a t e the sol ut io n . Format MUST follow : review ( p r e _ s o l u t i o n : str ) -> str p r e _ s o l u t i o n should be s olu ti on from pr evi ou s ope...

  48. [56]

    "" This is a wor kf low graph

    Test : Usage : Modify the input s olu ti on by testing the sol ut ion using public test cases . Format MUST follow : test ( sol ut io n : str ) -> str t e s t e d _ s o l u t i o n = await self . test ( so lu ti on = p r e _ s o l u t i o n ) 17 A.2.3 The Detailed Algorithm We...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.