REVIEW 6 cited by
OpsEval: A Comprehensive IT Operations Benchmark Suite for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Information Technology (IT) Operations (Ops), particularly Artificial Intelligence for IT Operations (AIOps), is the guarantee for maintaining the orderly and stable operation of existing information systems. According to Gartner's prediction, the use of AI technology for automated IT operations has become a new trend. Large language models (LLMs) that have exhibited remarkable capabilities in NLP-related tasks, are showing great potential in the field of AIOps, such as in aspects of root cause analysis of failures, generation of operations and maintenance scripts, and summarizing of alert information. Nevertheless, the performance of current LLMs in Ops tasks is yet to be determined. In this paper, we present OpsEval, a comprehensive task-oriented Ops benchmark designed for LLMs. For the first time, OpsEval assesses LLMs' proficiency in various crucial scenarios at different ability levels. The benchmark includes 7184 multi-choice questions and 1736 question-answering (QA) formats in English and Chinese. By conducting a comprehensive performance evaluation of the current leading large language models, we show how various LLM techniques can affect the performance of Ops, and discussed findings related to various topics, including model quantification, QA evaluation, and hallucination issues. To ensure the credibility of our evaluation, we invite dozens of domain experts to manually review our questions. At the same time, we have open-sourced 20% of the test QA to assist current researchers in preliminary evaluations of their OpsLLM models. The remaining 80% of the data, which is not disclosed, is used to eliminate the issue of the test set leakage. Additionally, we have constructed an online leaderboard that is updated in real-time and will continue to be updated, ensuring that any newly emerging LLMs will be evaluated promptly. Both our dataset and leaderboard have been made public.
Forward citations
Cited by 6 Pith papers
-
SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios
SREGym supplies 90 high-fidelity SRE tasks in a live environment to measure how well frontier AI agents handle diverse faults, noises, and complex failure modes such as metastable and correlated failures.
-
DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents
A production-fidelity benchmark finds that LLM database-operation agents achieve at most 17.9% safe recovery versus 93.4% for human DBAs.
-
OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning
OpsLLM's pipeline (HITL data curation, SFT, GRPO RL with a domain process reward model) improves LLM accuracy on software-operations QA and RCA, especially on in-distribution root-cause-analysis tasks.
-
SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments
SetupBench, a 93-instance environment-bootstrap benchmark, finds coding agents succeed on only 34.4-62.4% of setup tasks, with database configuration and repo setup being the hardest.
-
AviationLLM: An LLM-based Knowledge System for Aviation Training
DPO fine-tuning plus RAG, named RALA-DPO, is reported to improve accuracy and timeliness of aviation theory answers over SFT, based on internal evaluations.
-
A Survey of AIOps in the Era of Large Language Models
A systematic survey that categorizes LLM-based AIOps research into four dimensions: data sources, tasks, methods, and evaluation, claiming to be the first comprehensive such overview.
Discussion (0). Sign in to comment.