Pith. sign in

REVIEW 4 major objections 5 minor 34 references

A multi-agent workflow that treats execution as the sole acceptance signal aims to make LLM backend code synthesis reliable and measurable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 10:47 UTC pith:WEC72XF4

load-bearing objection A genuinely useful eval harness for multi-agent code synthesis whose headline scaling claim doesn't survive a missing control: the +22.5pp DeepSeek graph result lacks the single-retry row. the 4 major comments →

arxiv 2607.20499 v1 pith:WEC72XF4 submitted 2026-06-18 cs.AI cs.SE

ExecuGraph: A Multi-Agent, Execution-Grounded Framework for Reliable Backend Code Synthesis with Large Language Models

Sai Deekshith Lekkala , Jothi Prabha Appadurai , Rohith Reddy Bellibatlu , Manpreet Singh This is my paper
classification cs.AI cs.SE
keywords LLM code generationmulti-agent systemsexecution-grounded validationbackend code synthesissoftware reliabilityworkflow orchestrationscaling hypothesisretrieval-augmented generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that most LLM code-synthesis failures stem from the single-pass, single-agent paradigm, which provides no runtime verification. It presents ExecuGraph, a multi-agent workflow with six specialized roles where acceptance is decided solely by sandboxed execution, not by an LLM critic's textual verdict. The key contribution is a single codebase that can be configured to collapse into a one-shot baseline, an execution-retry baseline, or per-agent ablations, allowing the marginal value of each lever to be measured. Across internal and public benchmarks, aggregate differences are mostly small and statistically non-significant, but a cross-model comparison shows a strong improvement for the multi-agent condition when the base model is larger. This supports a scaling hypothesis: multi-agent decomposition pays off once the base model is capable enough to follow structured instructions.

Core claim

The paper's central claim is that execution-based validation should be the sole decision predicate in LLM code synthesis, and that multi-agent role decomposition is a separable lever whose contribution can be measured only when execution feedback and role separation are not conflated. ExecuGraph formalizes the synthesis workflow as a typed transition system over a shared state record, where the decision predicate is simply whether the sandboxed execution passed, and static reviewer output is advisory. The strongest positive evidence is a cross-model result: with a 16B-parameter Mixture-of-Experts code model, graph-category accuracy improves from 57.5% (one-shot) to 80.0% (multi-agent), a +22

What carries the argument

The load-bearing machinery is the typed transition system W = (S, A, δ, D, T) over a shared state record, where the decision predicate D(s) is purely execution-driven: D(s) ≡ s.evaluation.passed = True. Static review output is advisory and never gates acceptance. The routing function ρ sends the workflow to the Optimizer only after a pass, back to the Generator on failure while retries remain, and otherwise to the Explainer, guaranteeing termination with a bounded retry budget. This formalization is what allows the same codebase to reduce by configuration to a one-shot baseline, a single-agent execution-retry baseline, and per-agent ablations, making the marginal contribution of each role me

Load-bearing premise

The claim that the single-retry baseline isolates the contribution of multi-agent decomposition assumes that the multi-agent advantage comes from role separation rather than from the additional LLM calls and tokens used (5.1 calls and 3,715 tokens per problem vs 1.5 calls and 507 tokens for single-retry); if extra agents only add sampling or prompt surface, the central methodological interpretation is undermined.

What would settle it

Run a budget-matched comparison: give the single-agent execution-retry condition the same total LLM calls and tokens as the multi-agent workflow (e.g., allow more regeneration attempts and a richer failure prompt) and see whether the +22.5pp cross-model gain persists. If a single retrying agent matches multi-full accuracy at equal budget, the multi-agent decomposition is not the driver; if the gap remains, the scaling hypothesis is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If execution-grounded acceptance is the correct decision predicate, then systems that let an LLM critic veto code on textual grounds risk rejecting correct candidates; acceptance should be gated only on runtime results.
  • If the scaling hypothesis is correct, then at small base-model scale (7B-class) the most cost-effective configuration is the single-agent execution-retry workflow—comparable accuracy at roughly 4× lower wall-clock and 7× lower token cost than the full multi-agent setup.
  • Because the same codebase collapses by configuration, practitioners can directly measure the marginal contribution of planner, reviewer, optimizer, and retrieval in their own pipelines, converting debates about agentic coding from anecdote to numbers.
  • A per-agent ablation in the paper showed that removing the Planner alone yielded the highest aggregate accuracy (80.0% vs 76.7% for the full config at small N), suggesting that a weaker planner may mislead the generator; this is a concrete, testable risk for agentic coding systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A token-matched and call-matched ablation (giving the single-retry baseline the same LLM budget as the multi-agent config) is the cleanest way to falsify or confirm the scaling hypothesis; the paper itself did not run this control.
  • If the scaling threshold is real, the implication is that multi-agent frameworks should be evaluated across a range of base-model capacities, not just one model; the community may need to report 'gain curves' vs model size rather than single accuracy numbers.
  • The advisory-only reviewer design implies a checkable property: in the released per-trial logs, the reviewer's severity or potential_failures should have zero influence on the final accept/reject outcome; a reader can audit this directly from the JSON logs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. ExecuGraph is a LangGraph-based multi-agent framework for backend code synthesis with six specialized agents and execution-only acceptance, designed to collapse via configuration into single-agent one-shot, single-agent execution-retry, and multi-agent conditions. The paper evaluates the framework on a curated 30-problem DSA suite, a 64-problem sample of HumanEval, and a 50-problem APPS-introductory subset, using locally hosted 7B-16B open-weight models. The headline internal-30 results show no statistically significant pairwise differences among the three main conditions; the strongest reported signal is a +22.5pp graph-category gain for DeepSeek-Coder-V2-Lite when comparing multi-full to one-shot. The authors frame the primary contribution as methodological: a single reproducible codebase that enables controlled measurement of execution feedback, retries, and multi-agent decomposition, with all tables generated from released per-trial JSON logs.

Significance. If the methodological framing is taken at face value, the paper is a useful contribution: the same harness instantiates multiple baselines and ablations, the statistical reporting is unusually candid about null results, and the reproducibility artifact (scripts generating every table from per-trial logs, pinned model digests, and seeds) is a genuine asset. The cross-model scaling hypothesis, if supported by a proper control, would be an interesting and falsifiable claim about when multi-agent decomposition pays off. However, the current evidence does not establish that claim: the configuration intended to isolate multi-agent decomposition is confounded with token/call counts, and the cross-model comparison omits the one control that would attribute the gain to role decomposition. The paper's central methodological potential is real but its headline empirical interpretation needs revision.

major comments (4)
  1. [§3.2, §5.3, Table 5] The claim that single-retry 'isolates the contribution of multi-agent decomposition' is not supported. The difference between multi-full and single-retry includes Planner, Reviewer, Optimizer, and Explainer agents plus far more LLM calls and tokens: Table 5 reports 5.1 calls and 3,715 tokens for multi-full versus 1.5 calls and 507 tokens for single-retry. Thus the contrast conflates role decomposition with added prompt surface and compute. The paper should either reframe the comparison as 'full workflow versus retry-only' or add a budget-matched control (e.g., a single-agent condition with the same number of calls/tokens).
  2. [§6.2, Table 9] The DeepSeek graph-category improvement of +22.5pp (80.0% vs 57.5%) is presented as the 'strongest positive signal' and is used to support the scaling hypothesis. But Table 9 omits the single-retry condition, the very condition that Section 3.2 defines as the control for isolating multi-agent decomposition. Without a DeepSeek single-retry row, the gain could come entirely from execution feedback and retry budget. Additionally, the comparison uses only 3 trials per condition and no confidence interval or p-value. The scaling hypothesis should be withheld or the missing control added before drawing this conclusion.
  3. [§4.1, §5.6, §6.2] The HumanEval result (+3.1pp, 54.7% vs 57.8%) is used as a directional external-validity signal, but the 64-problem sample has no documented sampling procedure or seed, and no paired significance test is reported. Given that internal-30 pairwise differences all have confidence intervals spanning zero, a 3.1pp difference on a single trial per problem is not established. The authors should either report the sampling method and a paired test (e.g., McNemar or bootstrap CI) or explicitly label the HumanEval difference as exploratory and non-significant.
  4. [§5.4, Table 6] The per-agent ablation is described in Section 1.1 as 'quantifying the marginal contribution' of each component, but each ablation cell uses only n=2 trials per problem, and the aggregate differences (76.7–80.0%) are within the noise of the 5-trial multi-full baseline. The paper already acknowledges this is a qualitative signal in §6.1, but the contribution statement should be aligned with that caveat: the ablation can indicate directions, not quantitative marginal effects.
minor comments (5)
  1. [§4.5] The statement that HumanEval (n=64) and APPS (n=50) 'provide sufficient statistical power' is too strong for N=1 trials per problem; the power depends on effect size and the test used. Consider softening.
  2. [§4.3] The cross-model condition uses 3 trials while the headline internal-30 uses 5; the reason is not stated in the main text. A brief explanation would help readers interpret Table 9.
  3. [§5.5, Table 7] The row label 'Budget 0 (one-shot)' is confusing: the condition is multi-full with retry budget 0, not the single-oneshot baseline. Rename to avoid equivocation between 'one-shot' as a condition and 'zero retries'.
  4. [§6.4] JudgeSense [17] is used to motivate the advisory-only Reviewer, but [17] is co-authored by one of the authors of this paper. The text does not flag this self-citation; an explicit disclosure would improve transparency.
  5. [§1.1] The phrase 'the first controlled empirical measurement of each lever's marginal contribution' is stronger than the data support, given the confounds noted above. Suggest narrowing to 'a controlled measurement framework'.

Circularity Check

0 steps flagged

No circular derivation; one minor self-citation (JudgeSense) is motivational only.

full rationale

The paper's central claims are empirical and derived from per-trial JSON logs regenerated by a public script, not from an equation that maps fitted parameters to conclusions. The single-retry baseline is a configuration toggle of the same workflow and, at equal retry budget, differs from multi-full primarily by the presence of the additional agents; this is a controlled comparison, not a definitional identity. The cross-model scaling claim (Section 6.2) is weakened by the omission of the single-retry control from Table 9, but that is a missing control and an attribution gap, not circularity. The only author-overlapping citation is JudgeSense [17], co-authored by one of the present authors, used to motivate the advisory-only Reviewer design (Sections 2.2 and 6.4). That citation is not part of the measured pipeline, the decision predicate (Eq. 1) is independent of it, and the design is also grounded in a documented internal failure mode; hence it is a minor self-citation, not load-bearing circularity. The paper also explicitly acknowledges its limitations (contamination risk, no formal guarantees, test-source bias), further supporting an honest non-circular reading.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No data-fitted parameters are used; retry budgets, temperatures, and seeds are configuration choices, and the test-source audit shows the LLM-generated-test fallback was never invoked on the reported benchmarks. The main assumptions are about representativeness of samples and the interpretability of the ablation contrast.

axioms (4)
  • domain assumption Execution outcomes on the provided test suite fully determine correctness acceptance (D(s) ≡ s.evaluation.passed).
    Section 3.2 Eq. (1) makes execution the sole acceptance predicate; Section 6.7 admits programs passing tests may violate untested invariants.
  • domain assumption The 64-problem HumanEval sample is representative of the full 164-problem benchmark.
    Section 4.1 samples without specifying method or seed; Table 8 treats it as an external-validity anchor.
  • domain assumption Differences between multi-full and single-retry are attributable to role decomposition rather than to the larger number of LLM calls and tokens.
    Section 3.2 'Reduction to baselines' claims single-retry isolates multi-agent decomposition, but Table 5 shows multi-full uses 5.1 calls vs 1.5, so the premise is unverified.
  • domain assumption Internal-30 results are not systematically contaminated by pretraining memorization; HumanEval and APPS are the load-bearing external signal.
    Section 6.7 acknowledges contamination risk in internal-30 and designates external benchmarks as load-bearing.

pith-pipeline@v1.3.0-alltime-deepseek · 17380 in / 10546 out tokens · 89620 ms · 2026-08-02T10:47:12.026133+00:00 · methodology

0 comments
read the original abstract

Large Language Models generate plausible backend code, but a single-pass paradigm provides no guarantee of correctness or runtime reliability. We present ExecuGraph, a multi-agent framework that places execution-based validation at the center of backend code synthesis. Six specialized agents (Planner, Code Generator, Logical Reviewer, Evaluator, Optimizer, and Explainer) are coordinated by a typed directed workflow with a bounded retry budget, implemented on LangGraph with locally hosted models (Ollama) and an optional retrieval layer for algorithmic technique recall. A subprocess-isolated sandbox with a wall-clock timeout guards every evaluation. We evaluate on a curated 30-problem DSA suite (internal-30), HumanEval (n=64), and an APPS-introductory subset, contrasting ExecuGraph against a single-agent one-shot baseline and a single-agent execution-retry baseline (a Reflexion-style ablation that isolates the contribution of multi-agent decomposition). On internal-30, the three conditions are statistically indistinguishable (n=30; paired Wilcoxon p=0.59 MF vs. SO, p=0.08 SR vs. SO); 95% bootstrap confidence intervals on all pairwise mean differences include zero. On HumanEval, multi-full edges ahead by +3.1 pp. The strongest signal is cross-model: with DeepSeekCoder V2 Lite, graph-category accuracy improves from 57.5% (oneshot) to 80.0% (multi-full), a +22.5 pp jump that supports a scaling hypothesis: the value of multi-agent decomposition grows with base-model capability. The framework's primary contribution is methodological: a single codebase that collapses by configuration into one-shot, execution-retry, and per-agent ablation conditions, enabling controlled measurement of each lever's marginal contribution. A per-agent ablation, retry-budget sweep, error-class taxonomy, and test-source audit are reported.

Figures

Figures reproduced from arXiv: 2607.20499 by Jothi Prabha Appadurai, Manpreet Singh, Rohith Reddy Bellibatlu, Sai Deekshith Lekkala.

Figure 1
Figure 1. Figure 1: Five-layer architecture of ExecuGraph. User input flows top-to-bottom through the [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: ExecuGraph workflow DAG. Solid arrows show the main pipeline (S [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Category-level pass-rate by condition on internal-30 (values from Table [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Cost-correctness trade-off: mean wall-clock per problem (Table [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Cumulative pass-rate as a function of retries used. The slope at small budgets [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 19 linked inside Pith

  1. [1]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei,...

    2020

  2. [2]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374 (2021)

    Pith/arXiv arXiv 2021

  3. [3]

    Hendrycks, S

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, J. Steinhardt, Measuring coding challenge competence with APPS, in: Proc. Int. Conf. Learn. Representa- tions (ICLR), 2021. 36

    2021

  4. [4]

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, M. Zhou, CodeBERT: A pre-trained model for program- ming and natural languages, in: Findings of EMNLP, 2020

    2020

  5. [5]

    Y. Wang, W. Wang, S. Joty, S. C. H. Hoi, CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, arXiv preprint arXiv:2109.00859 (2021)

    Pith/arXiv arXiv 2021

  6. [6]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2022

    2022

  7. [7]

    X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency improves chain-of-thought rea- soning in language models, arXiv preprint arXiv:2203.11171 (2022)

    Pith/arXiv arXiv 2022

  8. [8]

    X. Chen, C. Liu, D. Song, Execution-guided neural program synthesis, OpenReview (2018)

    2018

  9. [9]

    L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, G. Neubig, PAL: Program-aided language models, arXiv preprint arXiv:2211.10435 (2022)

    Pith/arXiv arXiv 2022

  10. [10]

    N.Shinn, F.Cassano, E.Berman, A.Gopinath, K.Narasimhan, S.Yao, Re- flexion: Languageagentswithverbalreinforcementlearning, arXivpreprint arXiv:2303.11366 (2023)

    Pith/arXiv arXiv 2023

  11. [11]

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Ec- cles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. de Mas- son d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, O. Vinyals, Competition-level code gen- e...

    2022

  12. [12]

    Chowdhery, et al., PaLM: Scaling language modeling with pathways, arXiv preprint arXiv:2204.02311 (2022)

    A. Chowdhery, et al., PaLM: Scaling language modeling with pathways, arXiv preprint arXiv:2204.02311 (2022). 37

    Pith/arXiv arXiv 2022

  13. [13]

    Nijkamp, B

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, C. Xiong, CodeGen: An open large language model for code with multi- turn program synthesis, arXiv preprint arXiv:2203.13474 (2022)

    Pith/arXiv arXiv 2022

  14. [14]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettlemoyer, M. Lewis, InCoder: A generative model for code infilling and synthesis, arXiv preprint arXiv:2204.05999 (2022)

    Pith/arXiv arXiv 2022

  15. [15]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, C. Sutton, Program synthesis with large language models, arXiv preprint arXiv:2108.07732 (2021)

    Pith/arXiv arXiv 2021

  16. [16]

    X. Chen, M. Lin, N. Schärli, D. Zhou, Teaching large language models to self-debug, arXiv preprint arXiv:2304.05128 (2023)

    Pith/arXiv arXiv 2023

  17. [17]

    R. R. Bellibatlu, E. Raff, W. Zhang, JudgeSense: A benchmark for prompt sensitivity in LLM-as-a-Judge systems, arXiv preprint arXiv:2604.23478 (2026).arXiv:2604.23478,doi:10.48550/arXiv.2604.23478. URLhttps://arxiv.org/abs/2604.23478

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.23478 2026

  18. [18]

    Huang, J

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y. Qing, H. Cui, AgentCoder: Multi-agent-based code generation with iterative testing and optimisation, arXiv preprint arXiv:2312.13010 (2023)

    Pith/arXiv arXiv 2023

  19. [19]

    M. A. Islam, M. E. Ali, M. R. Parvez, MapCoder: Multi-agent code gen- eration for competitive problem solving, arXiv preprint arXiv:2405.11403 (2024)

    Pith/arXiv arXiv 2024

  20. [20]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, J. Schmidhuber, MetaGPT: Meta programming for a multi-agent collabo- rative framework, arXiv preprint arXiv:2308.00352 (2023)

    Pith/arXiv arXiv 2023

  21. [21]

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, C. Wang, 38 AutoGen: Enabling next-gen llm applications via multi-agent conversation, arXiv preprint arXiv:2308.08155 (2023)

    Pith/arXiv arXiv 2023

  22. [22]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, Y. Cao, ReAct: Synergizing reasoning and acting in language models, in: Proc. Int. Conf. Learn. Representations (ICLR), 2023

    2023

  23. [23]

    T. B. Richards, et al., Auto-GPT: An autonomous GPT-4 experiment, https://github.com/Significant-Gravitas/Auto-GPT(2023)

    2023

  24. [24]

    X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, J. Tang, AgentBench: Evaluating LLMs as agents, arXiv preprint arXiv:2308.03688 (2023)

    Pith/arXiv arXiv 2023

  25. [25]

    Schick, J

    T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettle- moyer, N. Cancedda, T. Scialom, Toolformer: Language models can teach themselves to use tools, arXiv preprint arXiv:2302.04761 (2023)

    Pith/arXiv arXiv 2023

  26. [26]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küt- tler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval- augmented generation for knowledge-intensive NLP tasks, in: Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2020

    2020

  27. [27]

    Chroma Authors, Chroma: The AI-native open-source embedding database,https://docs.trychroma.com/, open-source software (2024)

    2024

  28. [28]

    LangChain Authors, LangChain: Building applications with LLMs through composability,https://docs.langchain.com/, open-source soft- ware (2024)

    2024

  29. [29]

    LangGraph Authors, LangGraph: A library for building stateful, multi- actor applications with LLMs,https://github.com/langchain-ai/ langgraph, open-source software (2024). 39

    2024

  30. [30]

    Ollama Authors, Ollama: Run large language models locally,https:// ollama.com/, open-source software (2024)

    2024

  31. [31]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y. Miao, S. Quan, Y. Feng, X. Ren, X. Ren, J. Zhou, J. Lin, Qwen2.5-Coder technical report, arXiv preprint arXiv:2409.12186 (2024)

    Pith/arXiv arXiv 2024

  32. [32]

    DeepSeek-AI, DeepSeek-Coder-V2: Breaking the barrier of closed-source models in code intelligence, arXiv preprint arXiv:2406.11931 (2024)

    Pith/arXiv arXiv 2024

  33. [33]

    OpenAI, gpt-oss: An open-weight language model from OpenAI, Apache- 2.0 weights athttps://huggingface.co/openai/gpt-oss-20b(2025)

    2025

  34. [34]

    Qwen Team, Qwen3-Coder: Agentic code generation with mixture-of- experts, Technical report; weights athttps://huggingface.co/Qwen/ Qwen3-Coder-30B-A3B-Instruct(2025). 40

    2025