pith. sign in

arxiv: 2603.15676 · v2 · pith:QJYJUSFEnew · submitted 2026-03-13 · 💻 cs.SE · cs.AI

Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications

Alexandre Cristov\~ao Maiorano This is my paper

Pith reviewed 2026-05-22 10:26 UTC · model grok-4.3

classification 💻 cs.SE cs.AI
keywords LLM applicationsautomated testingquality gatesrelease managementmulti-agent systemsevidence coverageself-testing
0
0 comments X

The pith

An automated self-testing framework applies five quality dimensions to make promote, hold, or rollback decisions for LLM application releases.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that LLM applications require new testing approaches because their outputs are nondeterministic and models evolve over time. It introduces an automated self-testing framework that runs persona-grounded, multi-turn, and adversarial scenarios to produce evidence-based release decisions. In a four-week case study of an internal multi-agent system, the framework caught two rollback-worthy builds early and tracked stable quality improvement across dozens of runs. Statistical checks and a human calibration study with sixty cases show that evidence coverage best flags severe problems while the gate and human judges catch different failure types.

Core claim

The automated self-testing framework introduces quality gates with evidence-based release decisions (PROMOTE/HOLD/ROLLBACK) across five empirically grounded dimensions: task success rate, research context preservation, P95 latency, safety pass rate, and evidence coverage. Evaluation across 38 runs and more than twenty internal releases of a deployed multi-agent conversational system showed the gate identifying two rollback-grade builds and supporting stable quality evolution, with evidence coverage emerging as the main discriminator for severe regressions and runtime scaling predictably with suite size.

What carries the argument

The quality gate that scores outputs on the five dimensions to produce a PROMOTE, HOLD, or ROLLBACK release decision.

If this is right

  • Evidence coverage serves as the strongest indicator for detecting severe regressions.
  • Test runtime grows in a predictable way as the number of scenarios increases.
  • The automated gate and LLM-as-judge evaluations miss different failure modes and therefore complement each other.
  • Statistical trend tests can track whether quality improves or declines across successive releases.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could extend to other nondeterministic AI systems such as code generators or image models by adapting the five dimensions.
  • Emphasizing evidence coverage may push developers to add explicit logging of reasoning steps inside LLM applications.
  • Releasing the framework and artifacts supports independent teams testing the same gate logic on their own systems.

Load-bearing premise

The five chosen dimensions together with results from one internal multi-agent conversational system supply a representative basis for claims about release governance across LLM applications.

What would settle it

Running the same framework on additional LLM applications and observing that it misses known severe regressions or fails to show quality stabilization over repeated releases.

Figures

Figures reproduced from arXiv: 2603.15676 by Alexandre Cristov\~ao Maiorano.

Figure 1
Figure 1. Figure 1: shows the high-level interaction between the question bank and the orchestrator, while [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: CI/CD integration pipeline. A merge to the main branch triggers automated build checkout, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Decision flowchart: PROMOTE, HOLD, or ROLLBACK based on five quality dimensions [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Success rate across 38 evaluation runs. Green markers indicate PROMOTE decisions; red markers indicate ROLLBACK. The dashed line marks the 80% acceptance threshold [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of P95 latency by suite phase la [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Joint evolution of the five gate dimensions across 38 evaluation runs. Task success trend reflects [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
read the original abstract

LLM applications are AI systems whose nondeterministic outputs and evolving model behavior make traditional testing insufficient for release governance. We present an automated self-testing framework that introduces quality gates with evidence-based release decisions (PROMOTE/HOLD/ROLLBACK) across five empirically grounded dimensions: task success rate, research context preservation, P95 latency, safety pass rate, and evidence coverage. We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system with specific marketing capabilities in active development, covering 38 evaluation runs across 20+ internal releases. The gate identified two ROLLBACK-grade builds in early runs and supported stable quality evolution over a four-week staging lifecycle while exercising persona-grounded, multi-turn, adversarial, and evidence-required scenarios. Statistical analysis (Mann-Kendall trends, Spearman correlations, bootstrap confidence intervals), gate ablation, and overhead scaling indicate that evidence coverage is the primary severe-regression discriminator and that runtime scales predictably with suite size. A human calibration study (n=60 stratified cases, two independent evaluators, LLM-as-judge cross-validation) reveals complementary multi-modal coverage: LLM-judge disagreements with the system gate (kappa=0.13) are attributable to structural failure modes - latency violations and routing errors - invisible in response text alone, while the judge independently surfaces content quality failures missed by structural checks, consistent with a multi-dimensional gate design. The framework, supplementary pseudocode, and calibration artifacts are provided to support AI-system quality assurance and independent replication.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents an automated self-testing framework for LLM applications that deploys quality gates making PROMOTE/HOLD/ROLLBACK decisions across five empirically grounded dimensions: task success rate, context preservation, P95 latency, safety pass rate, and evidence coverage. It reports a longitudinal case study of 38 evaluation runs across 20+ releases of one internally deployed multi-agent conversational system, in which the gate flagged two ROLLBACK-grade builds early and tracked stable quality evolution over a four-week staging period. Supporting elements include Mann-Kendall trend tests, Spearman correlations, bootstrap intervals, gate ablation, overhead scaling, and a human calibration study (n=60 stratified cases, two evaluators, LLM-as-judge cross-validation) that reports low agreement (kappa=0.13) interpreted as complementary multi-modal coverage. Supplementary pseudocode and calibration artifacts are supplied for replication.

Significance. If the reported results hold under broader scrutiny, the work supplies a practical, evidence-based template for release governance of nondeterministic LLM systems, an area where traditional testing is acknowledged to be insufficient. The longitudinal design, statistical trend analysis, ablation experiments, and human calibration with independent judges provide concrete empirical grounding. The explicit release of pseudocode and calibration artifacts is a clear strength that directly supports reproducibility and independent validation.

major comments (2)
  1. [Longitudinal Case Study] The central claim that the automated gate reliably identifies ROLLBACK builds and supports stable quality evolution is supported only by 38 runs on a single internally deployed multi-agent marketing system. This single-system scope, without cross-application replication or external benchmark comparisons, limits the generalizability of the five-dimension weighting (evidence coverage as primary discriminator) and the observed trends to other LLM architectures or domains. This is load-bearing for the abstract's positioning of the framework as a general solution for LLM application release management.
  2. [Evaluation and Results] No control condition or direct comparison against traditional testing suites is described. Without such a baseline, it is difficult to quantify whether the observed rollback detections and quality trends represent an improvement over existing practices or are specific to the chosen system and scenario set (persona-grounded, multi-turn, adversarial, evidence-required).
minor comments (2)
  1. [Abstract] The abstract uses both 'research context preservation' and 'context preservation'; consistent terminology across the manuscript would reduce minor ambiguity.
  2. [Statistical Analysis] The bootstrap confidence interval procedure is referenced but the number of resamples and exact percentile method are not stated; adding these details would aid reproducibility of the statistical claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and positive review, which highlights both the practical value of the framework and areas where the manuscript can be strengthened. We address the two major comments point by point below, with honest acknowledgment of limitations and concrete plans for revision.

read point-by-point responses

  1. Referee: [Longitudinal Case Study] The central claim that the automated gate reliably identifies ROLLBACK builds and supports stable quality evolution is supported only by 38 runs on a single internally deployed multi-agent marketing system. This single-system scope, without cross-application replication or external benchmark comparisons, limits the generalizability of the five-dimension weighting (evidence coverage as primary discriminator) and the observed trends to other LLM architectures or domains. This is load-bearing for the abstract's positioning of the framework as a general solution for LLM application release management.

    Authors: We agree that the single-system longitudinal design limits strong claims of broad generalizability, and this is a substantive limitation for positioning the work as a general template. The study was intentionally scoped as an in-depth case study of one complex, actively developed multi-agent system to enable repeated evaluation across 20+ releases and four weeks of staging, which would be difficult to achieve across multiple independent applications in a single paper. The five dimensions and statistical methods (Mann-Kendall trends, ablation, bootstrap intervals) are intended to be portable, but we recognize that the observed weighting (evidence coverage as primary discriminator) may require domain-specific retuning. In revision we will (1) add an explicit Limitations subsection that qualifies the generalizability of both the weighting and the trend results, (2) rephrase the abstract and introduction to foreground the case-study nature while preserving the framework's design rationale, and (3) outline concrete next steps for cross-application replication. These changes will be partial because a new multi-system study cannot be completed within the revision cycle. revision: partial

  2. Referee: [Evaluation and Results] No control condition or direct comparison against traditional testing suites is described. Without such a baseline, it is difficult to quantify whether the observed rollback detections and quality trends represent an improvement over existing practices or are specific to the chosen system and scenario set (persona-grounded, multi-turn, adversarial, evidence-required).

    Authors: We accept that the lack of an explicit control arm or head-to-head comparison with conventional testing suites makes it harder to isolate the incremental benefit of the automated gate. The manuscript's premise (stated in the introduction) is that traditional testing is acknowledged to be insufficient for nondeterministic LLM behavior; therefore the study design prioritized longitudinal monitoring of real releases over a controlled baseline experiment. Nevertheless, the two ROLLBACK detections and the human-calibration results provide indirect evidence of value (e.g., structural failures invisible to text-only judges). In the revised manuscript we will add a new subsection under Evaluation that qualitatively contrasts the gate's detections against what standard unit, integration, and scenario-based tests would likely have caught in the same builds, using the two rollback cases and the latency/routing failures as concrete examples. This addition will be made without new data collection. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical case study relies on independent human validation and statistical tests

full rationale

The paper reports an empirical longitudinal case study of an automated self-testing framework applied to one internal multi-agent LLM system across 38 runs and 20+ releases. Claims about ROLLBACK detection and stable quality evolution are grounded in observed data, Mann-Kendall trends, Spearman correlations, bootstrap intervals, gate ablation, and a separate human calibration study (n=60 cases with two independent evaluators plus LLM-as-judge cross-validation). No equations, fitted parameters, or self-citations are presented as load-bearing derivations; the five dimensions are described as empirically grounded from the study context itself, and the central results do not reduce by construction to inputs defined within the paper. This is a standard self-contained empirical report whose validity rests on replication potential rather than internal definitional loops.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the assumption that the chosen five dimensions are sufficient to capture release quality; no explicit free parameters or new physical entities are introduced in the abstract.

axioms (1)
  • domain assumption The five dimensions (task success rate, research context preservation, P95 latency, safety pass rate, evidence coverage) adequately represent quality for LLM applications
    The gate decisions and regression discrimination rely on these dimensions being comprehensive.

pith-pipeline@v0.9.0 · 5803 in / 1224 out tokens · 37421 ms · 2026-05-22T10:26:02.505044+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

31 extracted references · 31 canonical work pages · 1 internal anchor

  1. [1]

    Parashar et al

    Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachi- appan Nagappan, Besmira Nushi, and Thomas Zimmermann. Software engineering for ma- chine learning: A case study. InProceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 291–300. IEEE, 2019. do...

    work page doi:10.1109/icse-seip.2019.00042 2019

  2. [2]

    Constitutional AI: Harmlessness from AI feedback, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback, 2022

    work page 2022

  3. [3]

    Basili, Gianluigi Caldiera, and H

    Victor R. Basili, Gianluigi Caldiera, and H. Di- eter Rombach. The goal question metric ap- proach. InEncyclopedia of Software Engineer- ing. John Wiley & Sons, 1994. doi: 10.1002/ 0471028959.sof142

    work page 1994

  4. [4]

    Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. The ML test score: A rubric for ML production readiness and technical debt reduction. InProceedings of IEEE Big Data 2017, 2017. doi: 10.1109/BigData.2017.8258038

    work page doi:10.1109/bigdata.2017.8258038 2017

  5. [5]

    STELLAR: A search- based testing framework for large language model applications

    Duygu Cetinkaya et al. STELLAR: A search- based testing framework for large language model applications. InProceedings of the 33rd Inter- national Conference on Software Analysis, Evo- lution and Reengineering (SANER 2026), 2026. doi: 10.48550/arXiv.2601.00497. 16

    work page doi:10.48550/arxiv.2601.00497 2026

  6. [6]

    Challenges in testing large language model based soft- ware: A faceted taxonomy.arXiv preprint arXiv:2503.00481, 2025

    Felix Dobslaw and Robert Feldt. Challenges in testing large language model based soft- ware: A faceted taxonomy.arXiv preprint arXiv:2503.00481, 2025

    work page arXiv 2025

  7. [7]

    Feitelson, Eitan Frachtenberg, and Kent L

    Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck. Development and deployment at Facebook.IEEE Internet Computing, 17(4): 8–17, 2013. doi: 10.1109/MIC.2013.25

    work page doi:10.1109/mic.2013.25 2013

  8. [8]

    Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Ka- mal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  9. [9]

    InProceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec @ CCS 2023)

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compro- mising real-world LLM-integrated applications with indirect prompt injection. InProceedings of the 16th ACM Workshop on Artificial Intelli- gence and Security (AISec), pages 79–90, 2023. doi: 10.1145/3605764.3623985

    work page doi:10.1145/3605764.3623985 2023

  10. [10]

    Eating your own dog food

    Warren Harrison. Eating your own dog food. IEEE Software, 23(3):5–7, 2006. doi: 10.1109/ MS.2006.72

    work page 2006

  11. [11]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InProceedings of the International Conference on Learning Represen- tations (ICLR), 2021

    work page 2021

  12. [12]

    MetaGPT: Meta programming for a multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xi- awu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zi- juan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J¨ urgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. InProceedings of the International Conference on Learning Represe...

    work page 2024

  13. [13]

    Multi- agent LLM committees for autonomous software beta testing.arXiv preprint arXiv:2512.21352,

    Sai Bhargav Haswanth Karanam et al. Multi- agent LLM committees for autonomous software beta testing.arXiv preprint arXiv:2512.21352,

    work page arXiv

  14. [14]

    doi: 10.48550/arXiv.2512.21352

    work page doi:10.48550/arxiv.2512.21352

  15. [15]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. The mea- surement of observer agreement for categorical data.Biometrics, 33(1):159–174, 1977. doi: 10.2307/2529310

    work page doi:10.2307/2529310 1977

  16. [16]

    Holistic evaluation of lan- guage models.Transactions on Machine Learning Research, 2023

    Percy Liang, Rishi Bommasani, Tony Lee, Dim- itris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of lan- guage models.Transactions on Machine Learning Research, 2023

    work page 2023

  17. [17]

    On human intellect and ma- chine failures: Troubleshooting integrative ma- chine learning systems

    Besmira Nushi, Ece Kamar, Eric Horvitz, and Donald Kossmann. On human intellect and ma- chine failures: Troubleshooting integrative ma- chine learning systems. InThirty-First AAAI Conference on Artificial Intelligence, 2017. doi: 10.1609/aaai.v31i1.10633

    work page doi:10.1609/aaai.v31i1.10633 2017

  18. [18]

    Training language mod- els to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback. InAdvances in Neural Information Processing Systems 35 (NeurIPS 2022), pages 27730–27744, 2022

    work page 2022

  19. [19]

    Red team- ing language models with language models

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red team- ing language models with language models. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pages 3419–3448, 2022. doi: 10.18653/ v1/2022.emnlp-main.225

    work page 2022

  20. [20]

    Ellie Pavlick and Tom Kwiatkowski

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, 2020. doi: 10.18653/v1/2020. acl-main.442

    work page doi:10.18653/v1/2020 2020

  21. [21]

    Beck, Tony Savor, and Michael Stumm

    Chuck Rossi, Elisa Shibley, Shi Su, Kent L. Beck, Tony Savor, and Michael Stumm. Continuous de- ployment of mobile software at Facebook (show- case). InProceedings of the 2016 24th ACM SIGSOFT International Symposium on Founda- tions of Software Engineering (FSE). ACM, 2016. doi: 10.1145/2950290.2994157

    work page doi:10.1145/2950290.2994157 2016

  22. [22]

    Guidelines for conducting and reporting case study research in software engineering.Empirical Software En- gineering, 14(2):131–164, 2009

    Per Runeson and Martin H¨ ost. Guidelines for conducting and reporting case study research in software engineering.Empirical Software En- gineering, 14(2):131–164, 2009. doi: 10.1007/ s10664-008-9102-8

    work page 2009

  23. [23]

    Gerald Schermann, J¨ urgen Cito, Philipp Leitner, and Harald C. Gall. Towards quality gates in continuous delivery and deployment. In2016 17 IEEE 24th International Conference on Program Comprehension (ICPC), pages 1–4. IEEE, 2016. doi: 10.1109/ICPC.2016.7503737

    work page doi:10.1109/icpc.2016.7503737 2016

  24. [24]

    Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Fran¸ cois Cre- spo, and Dan Dennison

    D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Fran¸ cois Cre- spo, and Dan Dennison. Hidden technical debt in machine learning systems. InAdvances in Neural Information Processing Systems 28 (NIPS 2015), pages 2503–2511, 2015

    work page 2015

  25. [25]

    IEEE Trans

    Sergio Segura, Gordon Fraser, Ana B. Sanchez, and Antonio Ruiz-Cort´ es. A survey on meta- morphic testing.IEEE Transactions on Soft- ware Engineering, 42(9):805–824, 2016. doi: 10.1109/TSE.2016.2532875

    work page doi:10.1109/tse.2016.2532875 2016

  26. [26]

    IEEE Ac- cess PP(99), 1–1 (2017)

    Mojtaba Shahin, Muhammad Ali Babar, and Liming Zhu. Continuous integration, delivery and deployment: A systematic review on approaches, tools, challenges and practices.IEEE Access, 5: 3909–3943, 2017. doi: 10.1109/ACCESS.2017. 2685629

    work page doi:10.1109/access.2017 2017

  27. [27]

    Shreya Shankar, J. D. Zamfirescu-Pereira, Bj¨ orn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, 2024. doi: 10.1145/3654777.3676450

    work page doi:10.1145/3654777.3676450 2024

  28. [28]

    Brown, Adam Santoro, Aditya Gupta, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023. BIG-bench collabora- tion

    work page 2023

  29. [29]

    Ohlsson, Bj¨ orn Regnell, and Anders Wessl´ en.Experimentation in Software Engineer- ing

    Claes Wohlin, Per Runeson, Martin H¨ ost, Mag- nus C. Ohlsson, Bj¨ orn Regnell, and Anders Wessl´ en.Experimentation in Software Engineer- ing. Springer Berlin, Heidelberg, 2012. doi: 10.1007/978-3-642-29044-2

    work page doi:10.1007/978-3-642-29044-2 2012

  30. [30]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Has- san Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. InProceedings of the Conference on Language Modeling (COLM 2024), 2024

    work page 2024

  31. [31]

    ROLLBACK

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. InAdvances in Neural Infor- mation Processing Systems 36 (NeurIPS 2023), 2023. A Supplementary Material This appendix pro...

    work page 2023