Pith. sign in

REVIEW 4 major objections 4 minor 48 references

The paper's central claim is that LLM database agents should be judged by whether they safely restore a live, faulty database, and its 106-scenario benchmark shows the best agent achieves 17.9% Safe Pass versus 93.4% for a human DBA referen

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:33 UTC pith:MAAE26E7

load-bearing objection A well-built benchmark that closes a real gap in comparing LLM database agents, but the headline human-vs-agent gap rests on unvalidated success/safety contracts and single-run numbers. the 4 major comments →

arxiv 2607.22165 v1 pith:MAAE26E7 submitted 2026-07-24 cs.DB cs.AIcs.CLcs.LG

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

Junming Chen , Junyang Jiang , Xu Chen , Zibo Liang , Kai Zheng This is my paper
classification cs.DB cs.AIcs.CLcs.LG
keywords database operationsLLM agentsbenchmarkfault diagnosisPostgreSQLoutcome-first evaluationoperational safetySafe Pass
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing evaluations of LLM database agents are not comparable and not faithful to production: they often test diagnosis or recommendations rather than executed fixes on live systems. To close that gap, it constructs DBA-Bench, a benchmark of 106 PostgreSQL fault scenarios across seven operational domains, where success means the agent measurably recovers the database without violating safety rules while a workload is still running. Across 848 automated runs, agents diagnose 32.7%, achieve the required outcome 19.6%, and pass safely 12.4%; the best automated system reaches 17.9% Safe Pass, against 93.4% for a human DBA reference. The paper's point is that the field needs one shared, outcome-first evaluation contract, and by that contract current agents are far from production-ready.

Core claim

The central claim is that a database-operations agent is only as good as the state it leaves the database in, so evaluation must be outcome-first: an agent passes only when scenario-specific verifiers confirm the fault is eliminated and trace-based rules confirm the path was operationally safe. The benchmark instantiates this as a closed loop—restore a dirty PostgreSQL snapshot, re-run manifestation predicates, let the agent read metrics, logs, and plans and execute SQL while workloads stay active, then score the post-fix environment, structured report, and action trace. The measurements show a consistent bottleneck: diagnosis does not become repair (62.1% of diagnosis-passing runs fail the

What carries the argument

The load-bearing mechanism is the scenario evaluator J_s = (Q_s, S_s, Eff_s), with three separable judgments: an outcome verifier V_s, an F1-based diagnosis matcher, and trace-based safety rules with weighted penalties. To make these judgments reproducible, every run starts from a snapshot-restored dirty environment admitted only after fault-manifestation predicates return true, and the agent operates through a five-API tool boundary (metrics, SQL, instance management, logs, knowledge base) against a live workload. The difficulty annotations—reference-path diagnostic depth D_s and environmental complexity C_s—are separately recorded so failures can be attributed to causal depth or noise.

Load-bearing premise

Everything rests on the scenario-specific outcome verifiers and safety rules correctly encoding what 'recovered and safe' means in production; if those hand-written predicates are systematically too lenient or too strict, every reported Safe Pass rate, including the human reference, measures the contract rather than the task.

What would settle it

Take a held-out sample of scenarios and the internal states they produce; have independent production DBAs, blind to the verifier's verdict, classify each post-run environment as 'safe to resume service' or not. If DBA labels disagree with V_s and the safety rules on more than a small fraction of cases, the benchmark's scores do not measure production readiness. A second check: replay a scenario against a real production incident of the same class and see whether agents that pass DBA-Bench also resolve the real incident.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If outcome-first evaluation is right, agent rankings in prior work that score diagnoses or recommendations need re-examination: several published systems would likely score far lower when fixes must be executed and verified on a live database.
  • The measured 75.5-point gap gives a concrete target: safe end-to-end remediation, not raw diagnostic accuracy, is the bottleneck to production deployment.
  • Safety is a control-loop property: because 80% of unsafe recoveries are scope or safeguard violations rather than destructive actions, agents need repair contracts carried through the whole trajectory, not a final safety filter.
  • Difficulty decomposes: deeper causal chains and noisier evidence each independently lower Outcome Pass, so benchmark comparisons should report depth and complexity rather than a single easy/hard split.
  • Cost matters on the frontier: one system matches the top Safe Pass at 36.5% lower cost, and another trades 3.7 points of Safe Pass for a 69.1% cost reduction, so deployment choice should read Safe Pass and cost jointly.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the verifier contracts are the benchmark's own definition of 'production success'; an independent study asking DBAs to label the same post-fix states could calibrate whether 93.4% human Safe Pass reflects the contract being too easy or humans being genuinely proficient.
  • Editorial inference: the same harness could be pointed at other engines (MySQL, SQL Server, cloud-managed databases) and at multi-engine incidents; if the outcome-first protocol transfers, the benchmark's claim extends beyond PostgreSQL.
  • Editorial inference: the 62.1% diagnosis-to-outcome drop and 36.7% outcome-to-safe drop suggest an agent design target: verification of repair, not just hypothesis maintenance, is where most current systems lose points. A testable extension is to give agents an explicit post-repair verification step and measure the Safe Pass delta.
  • Editorial inference: using the difficulty attributes D_s and C_s as control variables, one could test whether retrieval augmentation or larger context actually helps deep-diagnosis scenarios, since the paper's trace analysis predicts it will not by itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DBA-Bench, a benchmark for LLM-based database operations agents, built around 106 PostgreSQL scenarios in seven operational task domains. It argues that existing agent evaluations miss four properties of production database operations: live read-write environments, observation-space scale and noise, open solution spaces, and compound/misleading fault scenarios. The benchmark offers a unified tool interface, snapshot-based scenario restoration, outcome verifiers, trace-based safety rules, and separately reported Diagnosis, Outcome, and Safe Pass metrics plus token cost. The empirical study evaluates eight automated baseline groups in 848 runs and one Human DBA reference, reporting a best automated Safe Pass of 17.9% versus 93.4% for the human, and aggregate automated Diagnosis/Outcome/Safe Pass rates of 32.7%, 19.6%, and 12.4%, respectively.

Significance. If the measurement contract is sound, this is a valuable contribution: it gives the database-agents community a shared, reproducible evaluation protocol and a field-level result that is sobering and actionable. The paper's strengths include the breadth of the scenario library, the separation of diagnosis, outcome, safety, and cost, the fixed-backbone comparison of ReAct versus tree-search and knowledge-graph architectures, and the design effort around restoring and revalidating dirty environments before each run. The main weakness is that the scoring contract—the scenario-specific verifiers and safety rules—is not independently validated, and the human reference is barely described. Because the headline numbers are produced entirely by that unvalidated contract, the paper's central comparative claim is not yet established at the claimed strength.

major comments (4)
  1. [§5.1–5.2, Safe Pass definition] The primary endpoint is defined by hand-authored outcome verifiers V_s and safety rules G_s with weights λ_g; Safe Pass is the exact-zero conjunction of full outcome and no rule violation. The paper reports no external validation of these contracts: no inter-rater reliability, no calibration of verifier verdicts against independent production-DBA judgment, and no audit for false positives/negatives. The DBA involvement described in §4.1 concerns scenario construction and reference diagnostic paths, not the strictness or completeness of the scoring predicates. Since both automated and human runs are judged by the same unvalidated contract, the headline 17.9% vs. 93.4% gap may reflect how the authors wrote the predicates rather than production readiness. This is load-bearing and should be addressed with, at minimum, independent DBA labeling of a stratified sample of runs and safety traces,
  2. [§6.1 Human DBA baseline] The human reference is described only as 'one valid Human DBA result for each of the 106 scenarios.' The paper does not state the DBA's experience or role, the number of participants, whether a single person handled all scenarios, time limits, tool-use restrictions, or whether scoring was blinded. The 93.4% figure is a point estimate with no uncertainty. Because the abstract's central comparative claim is the 75.5-point gap to this reference, the description is insufficient. Add a detailed protocol, and ideally multiple raters or at least a discussion of expected human variability.
  3. [§6.1 Primary run configuration] The headline ordering is based on a single-run pass@1 evaluation with no confidence intervals or significance tests. The text itself notes that provider nondeterminism is retained where the API does not expose decoding controls. Thus the differences among 17.9% (GPT-5.5, Claude Opus 4.8), 14.2–15.1% (GLM-5.1, Qwen3.7-Max, DBAIOps), and 10.4% (DeepSeek V4 Pro) may be within run-to-run noise. Before asserting a 'clear performance ordering' and a tie for first, the paper should report variance over repeated runs for at least the leading baselines and a scenario sample, or use paired bootstrap/significance testing.
  4. [§8 vs. §1 artifact availability] The artifact statement is internally inconsistent: §1 and the abstract provide a GitHub URL, while §8 says 'The complete scenarios and evaluation artifacts will be released publicly upon publication.' For a benchmark whose value is reproducibility, this is not merely a presentation issue. Readers cannot currently inspect verifier predicates, safety rules, scenario manifests, or snapshot tooling that determine every reported number. Please state clearly what is available now, include verifier and rule definitions in the artifact, and provide versioned release metadata.
minor comments (4)
  1. [§4.2, Figure 5(c)] The C_0 = 0.5 boundary yields only 7 Low-complexity scenarios versus 99 High-complexity scenarios, so the complexity comparison rests on 56 versus 792 runs. The aggregate 5.8-point drop should be interpreted cautiously, and a sensitivity analysis around C_0 and D_0 would strengthen the difficulty-label claims.
  2. [§6.3, Figure 6] The relationship between the 848 automated runs and the 700 'evaluator-readable non-clean runs' should be made explicit. What happens to the remaining 148 runs, and how do clean runs overlap with the failure-mode population? A short accounting would prevent confusion.
  3. [§6.1, §6.5 DBAIOps reimplementation] DBAIOps is an independent reimplementation because no public implementation existed. The conclusion that knowledge-graph-guided reasoning does not improve over ReAct would be stronger with a validation of the reimplementation against the original paper's reported behavior, or a clear statement of fidelity limits.
  4. [§3.2 wording] The phrase 'the resulting task is a live fault' reads as though the fault occurs naturally; consider 'the task is a live manifested fault' or similar, since the fault is injected and then admitted by manifestation predicates.

Circularity Check

0 steps flagged

No significant circularity: the evaluation contracts, difficulty labels, and safety rules are specified independently of the measured baselines; the 17.9% vs. 93.4% gap is an empirical measurement, not a result derived from its own inputs.

full rationale

DBA-Bench is an empirical benchmark rather than a derivation, and its central claims are measurements made under explicitly defined evaluation contracts. The outcome verifiers V_s, safety rules G_s, diagnosis matcher, and formulas for Diagnosis Pass, Outcome Pass, and Safe Pass (Section 5) are specified before any baseline is run; they are not fitted to or derived from the agents' outputs. Difficulty labels are computed from independently annotated reference-path diagnostic depth and environmental complexity (Section 4.2), and the paper explicitly states that the reference path is not unique or shortest and that alternative valid paths receive the same outcome credit, which prevents the difficulty label from being defined by the success contract. The Human DBA reference is scored by the same contracts, but those contracts are not constructed from the human's answers, so the 93.4% Safe Pass is a measured result rather than a tautology. Self-citations, most notably DBAgent [5], appear only as related-work context and are not load-bearing: no uniqueness theorem, fitted parameter, or ansatz is imported from the authors' prior work to force the benchmark's conclusions. The absence of external validation of the Section 5.1/5.2 success and safety predicates is a legitimate construct-validity and reproducibility concern, as is the Section 8 statement that artifacts will be released upon publication despite the Section 1 GitHub URL, but neither constitutes circularity by construction. No equation or fitted parameter reduces to its own input, so the benchmark's quantitative results are not circular.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The benchmark rests on a few hand-set thresholds (D_0, C_0, diagnosis F1 cutoff) and domain assumptions about PostgreSQL representativeness and the validity of DBA-validated reference paths. No new physical or mathematical entities are postulated.

free parameters (4)
  • Low-depth boundary D_0 = 2
    Hand-set threshold in Section 4.2 for classifying a scenario as deep-diagnosis; directly changes the Easy/Hard split and the reported Easy-to-Hard performance drops.
  • Complexity boundary C_0 = 0.5
    Hand-set threshold in Section 4.2 for low-versus-high environmental complexity; affects the aggregate complexity analysis in Section 6.3.
  • Diagnosis pass threshold = 0.8
    Set-based F1 threshold in Section 5.1 that a run must meet to receive a diagnosis pass; affects all reported Diagnosis Pass rates.
  • Safety severity weights lambda_g
    Positive weights in Section 5.2 set the risk profile but do not affect the exact-zero Safe Pass test. They are nonetheless free choices made by the authors.
axioms (4)
  • domain assumption PostgreSQL is a representative production database engine
    All 106 scenarios run only on PostgreSQL (Section 3.2, Section 6.1); results may not generalize to other DBMSs such as MySQL or Oracle.
  • domain assumption DBA-validated reference paths and root causes are correct
    Section 4.1 states that DBAs review causal faults, reference diagnostic paths, and verifiers, but the identity and independence of these DBAs are not described; they may be the benchmark authors.
  • ad hoc to paper Tool-output noise ratio approximates diagnostic difficulty
    Environmental complexity C_s is defined as 1 minus the signal ratio in tool outputs (Section 4.2). This is a proxy for diagnostic difficulty and is not independently validated.
  • standard math Standard set-theoretic F1 and arithmetic means
    Diagnosis accuracy and pass-rate aggregation use standard formulas (Sections 5.1 and 5.4), requiring no special justification.

pith-pipeline@v1.3.0-alltime-deepseek · 19700 in / 10927 out tokens · 110053 ms · 2026-08-01T05:33:55.629762+00:00 · methodology

0 comments
read the original abstract

LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation and production operations: live-environment fidelity (multi-turn read-write interaction with a running database); observation-space scale and complexity (causal diagnosis across thousands of time series, business logs, and concurrent activity); solution-space openness (multiple remediations with different operational trade-offs); and scenario complexity and coverage (faults cascading across internal mechanisms and operational domains). We present DBA-Bench, a benchmark addressing these gaps through production fidelity, outcome-first evaluation, and controlled scenario reproducibility. It uses instrumented PostgreSQL environments with active workloads, persistent state, and multi-source observations; defines success by measurable recovery or fault elimination under safety constraints; and restores snapshots with scenario-specific checks before each run. The benchmark contains 106 scenarios across seven task domains, with two public difficulty labels based on reference-path diagnostic depth and environmental complexity. We evaluate nine baseline groups, including six foundation-model systems, two GPT-5.5-backed database agents, and a Human DBA reference. Across 848 automated runs, Diagnosis, Outcome, and Safe Pass rates are 32.7%, 19.6%, and 12.4%; the best automated baseline reaches 17.9% Safe Pass versus 93.4% for the Human DBA reference. Automated Safe Pass falls from 19.6% on Easy scenarios to 7.6% on Hard scenarios, underscoring the difficulty of safe end-to-end remediation.

Figures

Figures reproduced from arXiv: 2607.22165 by Junming Chen, Junyang Jiang, Kai Zheng, Xu Chen, Zibo Liang.

Figure 1
Figure 1. Figure 1: DBA-Bench architecture for 106 PostgreSQL scenarios, from scenario construction through stateful agent– environment interaction to outcome-first, multi-dimensional evaluation [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: A representative PostgreSQL scenario with diagnos [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: PostgreSQL Safe Pass, Outcome Pass, and Diagnosis Pass overall and across seven scenario categories. The legend [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: PostgreSQL Safe Pass–mean Token Cost trade-off. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: PostgreSQL capability boundaries. Panel (a) reports Outcome Pass by Easy/Hard difficulty; panel (b) reports Outcome [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Failure-mode composition by scenario category across 700 evaluator-readable non-clean runs. Each [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 3 canonical work pages

  1. [1]

    Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023. Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1737–1749. doi:10.1109/ ICSE48619.2023.00149

    arXiv 2023

  2. [2]

    Haicheng Chen, Wensheng Dou, Dong Wang, and Feng Qin. 2020. CoFI: Consistency-Guided Fault Injection for Cloud Systems. InProceedings of the 12 35th IEEE/ACM International Conference on Automated Software Engineering. 536–

    2020

  3. [3]

    Sibei Chen, Ju Fan, Bin Wu, Nan Tang, Chao Deng, Pengyi Wang, Ye Li, Jian Tan, Feifei Li, Jingren Zhou, and Xiaoyong Du. 2025. Automatic Database Configura- tion Debugging using Retrieval-Augmented Language Models.Proceedings of the ACM on Management of Data3, 1 (2025), 1–27. doi:10.1145/3709663

    doi:10.1145/3709663 2025

  4. [4]

    Xu Chen, Haitian Chen, Zibo Liang, Shuncheng Liu, Jinghong Wang, Kai Zeng, Han Su, and Kai Zheng. 2023. LEON: A New Framework for ML-Aided Query Optimization.Proceedings of the VLDB Endowment16, 9 (2023), 2261–2273. doi:10.14778/3598581.3598597

    arXiv 2023

  5. [5]

    Xu Chen, Junmin Chen, Shuncheng Liu, Yuyang Xia, Zibo Liang, Xiaoming Qin, Kai Zeng, Han Su, and Kai Zheng. 2026. DBAgent: An RL-Based Agent for Autonomous Database Operations and Maintenance.Proceedings of the VLDB Endowment19, 12 (2026)

    2026

  6. [6]

    Xu Chen, Shuncheng Liu, Tong Yuan, Tao Ye, Kai Zeng, Han Su, and Kai Zheng

  7. [7]

    Xu Chen, Zhen Wang, Shuncheng Liu, Yaliang Li, Kai Zeng, Bolin Ding, Jingren Zhou, Han Su, and Kai Zheng. 2023. BASE: Bridging the Gap between Cost and Latency for Query Optimization.Proceedings of the VLDB Endowment16, 8 (2023), 1958–1966. doi:10.14778/3594512.3594525

    arXiv 2023

  8. [8]

    Xu Chen, Ximu Zeng, Yuze Wang, Zibo Liang, Kai Zeng, Han Su, and Kai Zheng

  9. [9]

    Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, and Saravan Rajmohan. 2025. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Au- tonomous Clouds. InProceedings of Machine Learning and Systems, Vol. 7. https://arxiv.org/abs/2501.06706

    Pith/arXiv arXiv 2025

  10. [10]

    Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2024. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. InProceedings of the Nineteenth European Conference on Computer Systems. 674–688. doi:10.1145/3627703.3629553

    arXiv 2024

  11. [11]

    Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao

  12. [12]

    Shiyue Huang, Ziwei Wang, Xinyi Zhang, Yaofeng Tu, Zhongliang Li, and Bin Cui

  13. [13]

    Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. 2024. R2E: Turning any GitHub Repository into a Programming Agent Environment. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). 21196–21224. https: //proceedings.mlr.press/v235/jain24c.html

    2024

  14. [14]

    Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang

  15. [15]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real- World GitHub Issues?. InInternational Conference on Learning Representations. https://arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  16. [16]

    Jiale Lao, Yibo Wang, Yufei Li, Jianping Wang, Yunjia Zhang, Zhiyuan Cheng, Wanghu Chen, Mingjie Tang, and Jianguo Wang. 2024. GPTuner: A Manual- Reading Database Tuning System via GPT-Guided Bayesian Optimization.Pro- ceedings of the VLDB Endowment17, 8 (2024), 1939–1952. doi:10.14778/3659437. 3659449

    doi:10.14778/3659437 2024

  17. [17]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural Information Process- ing Systems 33. 9459–9474. https://proceedings.neur...

    2020

  18. [18]

    Zibo Liang, Xu Chen, Yuyang Xia, Runfan Ye, Haitian Chen, Jiandong Xie, and Kai Zheng. 2024. DACE: A Database-Agnostic Cost Estimator. In2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 4925–4937. doi:10. 1109/ICDE60146.2024.00374

    arXiv 2024

  19. [19]

    Ping Liu, Shenglin Zhang, Yongqian Sun, Yuan Meng, Jiahai Yang, and Dan Pei. 2020. FluxInfer: Automatic Diagnosis of Performance Anomaly for Online Database System. In2020 IEEE 39th International Performance Computing and Communications Conference (IPCCC). IEEE, 1–8. doi:10.1109/IPCCC50635.2020. 9391550

    arXiv 2020

  20. [20]

    Xiaoze Liu, Zheng Yin, Chao Zhao, Congcong Ge, Lu Chen, Yunjun Gao, Dimeng Li, Ziting Wang, Gaozhong Liang, Jian Tan, and Feifei Li. 2022. PinSQL: Pinpoint Root Cause SQLs to Resolve Performance Issues in Cloud Databases. In2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2549–2561. doi:10.1109/ICDE53745.2022.00236

    arXiv 2022

  21. [21]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. AgentBench: Evaluating LLMs as Agents. InInternational Conference on Learning Represe...

    Pith/arXiv arXiv 2024

  22. [22]

    Yuhe Liu, Changhua Pei, Longlong Xu, Bohan Chen, Mingze Sun, Zhirui Zhang, Yongqian Sun, Shenglin Zhang, Kun Wang, Haiming Zhang, Jianhui Li, Gaogang Xie, Xidao Wen, Xiaohui Nie, Minghua Ma, and Dan Pei. 2023. OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models. arXiv preprint arXiv:2310.07637(2023). https://arxiv.org/abs/2310.07637

    Pith/arXiv arXiv 2023

  23. [23]

    Xianglin Lu, Zhe Xie, Zeyan Li, Mingjie Li, Xiaohui Nie, Nengwen Zhao, Qingyang Yu, Shenglin Zhang, Kaixin Sui, Lin Zhu, and Dan Pei. 2022. Generic and Robust Performance Diagnosis via Causal Inference for OLTP Database Systems. In2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE, 655–664. doi:10.1109/CCGrid545...

    arXiv 2022

  24. [24]

    Minghua Ma, Zheng Yin, Shenglin Zhang, Sheng Wang, Christopher Zheng, Xin- hao Jiang, Hanwen Hu, Cheng Luo, Yilin Li, Nengjun Qiu, Feifei Li, Changcheng Chen, and Dan Pei. 2020. Diagnosing Root Causes of Intermittent Slow Queries in Cloud Databases.Proceedings of the VLDB Endowment13, 8 (2020), 1176–1189. doi:10.14778/3389133.3389136

    arXiv 2020

  25. [25]

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. In International Conference on Learning Representations (ICLR). https://arxiv.org/ abs/2311.12983

    Pith/arXiv arXiv 2024

  26. [26]

    Biao Ouyang, Yingying Zhang, Hanyin Cheng, Yang Shu, Chenjuan Guo, Bin Yang, Qingsong Wen, Lunting Fan, and Christian S. Jensen. 2024. RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems. Proceedings of the VLDB Endowment18, 4 (2024), 1169–1182. doi:10.14778/3717755. 3717774

    doi:10.14778/3717755 2024

  27. [27]

    2026.PostgreSQL 16.14 Documentation

    PostgreSQL Global Development Group. 2026.PostgreSQL 16.14 Documentation. https://www.postgresql.org/docs/16/ Accessed 2026-07-20

    2026

  28. [28]

    Vikramank Singh, Kapil Eknath Vaidya, Vinayshekhar Bannihatti Kumar, Sopan Khosla, Balakrishnan Narayanaswamy, Rashmi Gangadharaiah, and Tim Kraska

  29. [29]

    Xudong Sun, Wenqing Luo, Jiawei Tyler Gu, Aishwarya Ganesan, Ramnatthan Alagappan, Michael Gasch, Lalith Suresh, and Tianyin Xu. 2022. Automatic Reliability Testing for Cluster Management Controllers. In16th USENIX Sym- posium on Operating Systems Design and Implementation (OSDI 22). 143–159. https://www.usenix.org/conference/osdi22/presentation/sun

    2022

  30. [30]

    Gordon, and Bohan Zhang

    Dana Van Aken, Andrew Pavlo, Geoffrey J. Gordon, and Bohan Zhang. 2017. Automatic Database Management System Tuning Through Large-scale Machine Learning. InProceedings of the 2017 ACM International Conference on Management of Data. 1009–1024. doi:10.1145/3035918.3064029

    arXiv 2017

  31. [31]

    Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Jihong Wang, Fengbin Yin, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2024. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 4966–4974. doi:10.1145/3627673.3680016

    arXiv 2024

  32. [32]

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback. InAdvances in Neural Information Processing Systems 36. 23826–23854. doi:10.52202/075280-1035

    work page doi:10.52202/075280-1035 2023

  33. [33]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InThe Eleventh International Conference on Learning Representations (ICLR). OpenReview.net. https://openreview.net/forum?id=WE_vluYUL-X

    2023

  34. [34]

    Dong Young Yoon, Ning Niu, and Barzan Mozafari. 2016. DBSherlock: A Performance Diagnostic Tool for Transactional Databases. InProceedings of the 2016 International Conference on Management of Data. ACM, 1599–1614. doi:10.1145/2882903.2915218

    arXiv 2016

  35. [35]

    In Conference on Innovative Data Systems Research (CIDR)

    Panda: Performance Debugging for Databases using LLM Agents. In Conference on Innovative Data Systems Research (CIDR). https://www.cidrdb.org/ cidr2024/papers/p6-singh.pdf

  36. [36]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. InInternational Conference on Learning Representations. https://arxiv. 13 org/abs/2307.13854

    Pith/arXiv arXiv 2024

  37. [37]

    Wei Zhou, Ji Sun, Xuanhe Zhou, Guoliang Li, Luyang Liu, Hao Wu, and Tianyuan Wang. 2025. GaussMaster: An LLM-based Database Copilot System.arXiv preprint arXiv:2506.23322(2025). https://arxiv.org/abs/2506.23322

    Pith/arXiv arXiv 2025

  38. [38]

    Wei Zhou, Peng Sun, Xuanhe Zhou, Qianglei Zang, Ji Xu, Tieying Zhang, Guo- liang Li, and Fan Wu. 2026. DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphs.Proceedings of the VLDB Endowment19, 6 (2026), 1319–1331. doi:10.14778/3797919.3797937

    arXiv 2026

  39. [39]

    Xuanhe Zhou, Lianyuan Jin, Ji Sun, Xinyang Zhao, Xiang Yu, Jianhua Feng, Shifu Li, Tianqing Wang, Kun Li, and Luyang Liu. 2021. DBMind: A Self-Driving Platform in openGauss.Proceedings of the VLDB Endowment14, 12 (2021), 2743–

    2021

  40. [40]

    Xuanhe Zhou, Guoliang Li, Zhaoyan Sun, Zhiyuan Liu, Weize Chen, Jianming Wu, Jiesi Liu, Ruohang Feng, and Guoyang Zeng. 2024. D-Bot: Database Diagnosis System using Large Language Models.Proceedings of the VLDB Endowment17, 10 (2024), 2514–2527. doi:10.14778/3675034.3675043 14

    arXiv 2024

  41. [42]

    Zhaoyang Yu, Minghua Ma, Chaoyun Zhang, Si Qin, Yu Kang, Chetan Bansal, Saravan Rajmohan, Yingnong Dang, Changhua Pei, Dan Pei, Qingwei Lin, and Dongmei Zhang. 2024. MonitorAssistant: Simplifying Cloud Service Monitor- ing via Large Language Models. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering...

    arXiv 2024

  42. [547]

    doi:10.1145/3324884.3416548

    arXiv

  43. [2022]

    InAdvances in Neural Information Processing Systems, Vol

    ADBench: Anomaly Detection Benchmark. InAdvances in Neural Information Processing Systems, Vol. 35. 32142–32159. https://papers.nips.cc/ paper_files/paper/2022/hash/cf93972b116ca5268827d575f2cc226b-Abstract- Datasets_and_Benchmarks.html

    2022

  44. [2023]

    Proceedings of the ACM on Management of Data1, 1 (2023), 1–26

    DBPA: A Benchmark for Transactional Database Performance Anomalies. Proceedings of the ACM on Management of Data1, 1 (2023), 1–26. doi:10.1145/ 3588926

    2023

  45. [2024]

    InProceedings of the IEEE/ACM 46th International Conference on Software Engineering

    Xpert: Empowering Incident Management with Query Recommendations via Large Language Models. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. doi:10.1145/3597503.3639081

    arXiv

  46. [2025]

    doi:10.1145/3709710

    Optimizing Block Skipping for High-Dimensional Data with Learned Adaptive Curve.Proceedings of the ACM on Management of Data3, 1 (2025), 1–26. doi:10.1145/3709710

    work page doi:10.1145/3709710 2025

  47. [2026]

    doi:10.1007/s00778-026-00973-9

    LEON+: Towards Robust ML-Aided Query Optimization.The VLDB Journal 35, 3 (2026), 21. doi:10.1007/s00778-026-00973-9

    work page doi:10.1007/s00778-026-00973-9 2026

  48. [2746]

    doi:10.14778/3476311.3476334

    arXiv