Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Empowering AIOps: Leveraging Large Language Models for IT Operations Management

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tool-using LLM agents can automate routine Kubernetes IT operations, with GPT-4o solving every multi-step task it was given.

desk verdict Useful practitioner benchmark of LLM agents for OpenShift ops; send to review, but the accuracy ranking needs objective scoring and more advanced reasoning tasks before the GPT-4o 100% claim carries weight. read the letter →

arxiv 2501.12461 v2 pith:R3ZXKJW3 submitted 2025-01-21 cs.SE

Arthur Vitui , Tse-Hsun Chen This is my paper · ORCID
classification cs.SE
keywords AIOpslargelanguagemodelsLLMagentstooluseReActKubernetescapacityplanningevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an LLM equipped with a small set of typed tools can act as a practical assistant for IT operations management, and it measures how well ten models do on twenty-five realistic queries against a container-orchestration platform. The authors report that accuracy splits by task complexity: the Claude family leads on simple reasoning, while the GPT-4 family leads on advanced reasoning that chains two or more tools, with GPT-4o reaching 100% accuracy on the advanced multi-tool tasks. They also report that cost and latency vary widely—OpenAI models are the most token-efficient, Anthropic models the most verbose, and smaller models can still be viable for simple tasks. The paper further shows that a machine-learning capacity-planning tool can be folded into the agent, letting it move from answering questions toward making proactive parameter adjustments. If these results are borne out, organizations could automate many routine platform-management chores by choosing the model to match the task's complexity.

What carries the argument

The load-bearing mechanism is the ReAct loop—a prompting pattern in which the model alternates written reasoning with concrete tool calls and observes the results before deciding the next step—wrapped in an agent-orchestration layer and connected to nine Python tools with typed interfaces. The tools let the model go beyond its training data: time conversion, service and pod summaries, metric-name lookup, metric-range retrieval, plotting, documentation retrieval over a vector database (RAG), and a machine-learning capacity-planning tool. The evaluation machinery also includes a ten-fold repetition of every query and a human-expert accuracy judgment, which is what turns anecdotal demonstrations into reported percentages.

What would settle it

Re-run the 25 queries with automated scoring that checks objectively verifiable outputs—exact filenames, timestamps, CSV values, plotted metric ranges—and compare GPT-4o's and Claude 3.5 Sonnet's advanced-reasoning accuracy; if GPT-4o's 100% does not hold under that check, the claimed superiority on multi-tool tasks is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM agent built on a reasoning-and-acting loop (ReAct) and equipped with nine purpose-built tools—time conversion, namespace service and pod summaries, metric-name lookup, metric-range retrieval, plotting, documentation retrieval through a vector database (RAG), and a machine-learning capacity-planning tool—can resolve common IT-operations queries with high accuracy. In a ten-fold evaluation on 25 tasks, the top models reached 95–100% accuracy: Claude 3.5 Sonnet and Claude 3 Opus led simple-reasoning tasks, while GPT-4o solved all advanced multi-tool tasks and GPT-4 Turbo came close. The paper also claims that accuracy alone is not enough: response time and token cost vary by an order of magnitude across models, and failures take recognizable forms—hallucinated dates, truncated outputs, wrong tool orders, timeouts—that can be mitigated by prompt specificity and by disabling conversational memory for task-oriented queries.

Load-bearing premise

All reported accuracy figures rest on human expert grading that is described but not shown, with no published rubric or measure of agreement between graders, so if that grading is biased or inconsistent the reported model rankings could change.

Editorial extensions

If this is right

  • Routine operational queries—service discovery, metric retrieval, documentation lookups, and capacity planning—can be handed to a tool-using LLM agent on a container-orchestration platform.
  • Task complexity should drive model selection: cheaper and smaller models are viable for simple-reasoning tasks, while multi-step tool-chaining queries justify the cost of top-tier GPT-4 models.
  • Token consumption and latency are first-order operational costs, so the most accurate model is not automatically the best choice for continuous monitoring or high-volume queries.
  • Removing conversational memory improves correctness on timestamp-sensitive tasks, which argues for stateless task-oriented assistants unless contextual follow-up is explicitly needed.
  • Integrating a predictive machine-learning tool into the agent extends it from answering questions to recommending parameter configurations that meet a target key performance indicator (KPI).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because many of the 25 tasks have objectively checkable outputs—exact filenames, timestamps, CSV values, plotted metric ranges—the human-grading step could be replaced by automated verification, which would make the accuracy comparison reproducible without expert annotators.
  • Inference: the failure taxonomy (hallucination, deflection, truncation, timeouts) suggests a natural next benchmark: measure whether adding a verification step, such as forcing the agent to read back tool outputs, closes the accuracy gap between GPT-4o and the Claude models on multi-step tasks.
  • Inference: the paper's observation that a small 7B model did reasonably on simple tasks implies that local or on-premises deployments, where data cannot leave the environment, could use smaller open-weight models for low-complexity operations, at lower cost and with acceptable accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports an empirical evaluation of ReAct-based LLM agents equipped with nine custom tools (e.g., time lookup, Prometheus metric retrieval, RAG, and MLASP capacity planning) for IT operations tasks on Red Hat OpenShift. The authors define 25 queries, 21 of them classified as simple reasoning and 4 as advanced reasoning, run each query 10 times against 10 LLMs, and measure accuracy, response time, and token usage. The central claims are that Anthropic models are best for simple reasoning, OpenAI models are best for advanced reasoning, and GPT-4o reaches 100% accuracy on the advanced multi-tool tasks, while Mixtral 8x22B performs poorly and tends to hallucinate. The paper also discusses lessons learned about memory components, prompt specificity, and tool chaining, and it points to a public GitHub repository for the implementation and detailed results.

Significance. If the measurement issues are resolved, this is a useful experience report for practitioners choosing LLM-based agents for Kubernetes/OpenShift operations. The paper's strengths are its real OpenShift testbed, a custom workload, ten repetitions per query, joint reporting of accuracy, latency, and token cost, and a public code repository. The practical recommendations about model selection are potentially valuable. However, the accuracy measurements are not currently auditable, and the advanced-reasoning evidence base is very small, so the comparative claims in Table 2 go beyond what the reported data support. The paper is not proposing a new method; its contribution is an empirical comparison and lessons learned, which is appropriate for the venue if the measurement protocol is strengthened.

major comments (4)
  1. [Section 2.2, Section 3.3, Table 2] The accuracy metric is the load-bearing outcome of the paper, but Section 2.2 defines it only as "determined through human expert evaluation," and Section 3.3 does not specify the scoring protocol. No rubric, grader count, grader independence, or inter-rater reliability is reported. Because many outputs are objectively checkable (timestamps, service names, port numbers, file names, CSV contents), the authors should either provide a deterministic scoring procedure or a detailed rubric with inter-rater reliability before the model ranking in Table 2 can be accepted. Without this, even a small number of inconsistent grading decisions can reorder the closely clustered simple-reasoning results and alter the advanced-reasoning percentages.
  2. [Section 4.1, Table 1, Table 2] The advanced-reasoning comparison rests on four queries (Q-21, Q-22, Q-24, Q-25), ten repetitions each, and Q-24 and Q-25 are variants of the same Prometheus lookup-and-plot workflow. Consequently, one systematic failure mode can move a model's AR accuracy by 25 percentage points, and the gap between GPT-4o's 100% and Claude 3.5 Sonnet's or Claude 3 Opus's 95% is not statistically meaningful. The paper should report per-query accuracy, confidence intervals or significance tests, and ideally a larger and more diverse set of advanced queries before claiming that OpenAI models are "better on advanced reasoning."
  3. [Table 1 (Q-24) and Table 5] The definition of the evaluated query Q-24 differs between Table 1 and Table 5. Table 1 states the metric range begins "starting 40 days ago" and instructs "Return only the file name and nothing else," whereas Table 5's step 1 states "starting 48 hours ago" and "Return only the content string of the tool and nothing else." The authors need to reconcile this mismatch and state which wording produced the results in Table 2, since accuracy scoring depends on the exact expected output.
  4. [Section 5.1, Section 5.3] The experimental configuration was changed after observing model behavior: the memory component was disabled because it caused incorrect timestamp answers, and the Q-07 prompt was modified to include the product name, reportedly improving performance. These post hoc adjustments are disclosed, but the reported accuracy numbers are therefore generated under a tuned configuration, and the paper does not report the effect of the original configuration or control for the changes. An ablation or a clear statement of the exact configuration used for each result in Table 2 is needed to support the cross-model comparison.
minor comments (5)
  1. [Section 2.2] The metrics subsection defines "Latency and Throughput," but the paper reports only response times; no throughput results appear in Section 4.2 or the tables. The authors should either add throughput measurements or remove throughput from the metric definition.
  2. [Table 3] The table heading says "average response times" while the rows are labeled P-50 and P-90; the paper should clarify whether these are averages of per-query percentiles or percentile values computed over the pooled repetitions.
  3. [Section 4.2] The text says "OpenAI models generally respond the fastest on both SR and AR queries," but the immediately following sentence identifies Claude 3 Haiku as the fastest for SR queries based on P-50 values. This is confusing and should be reconciled.
  4. [Section 3.1] There is a typo: "Kubernete-based" should be "Kubernetes-based."
  5. [Section 7] The claim that this research is "the first to combine predictive machine learning models with LLMs in AIOps" is a strong novelty assertion based on a brief related-work discussion and the authors' own MLASP tool. The claim should be qualified or supported with a more systematic literature search.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical benchmark is self-contained; the model ranking does not reduce to any fitted parameter or self-cited definition, so no material circularity is found.

full rationale

The central results in Section 4 are accuracy, latency, and token measurements from a ten-fold execution of 25 queries against ten LLMs. The claimed ranking (Anthropic models best at simple reasoning, OpenAI/GPT-4o best at advanced reasoning) is a summary of Table 2, not a derivation from a fitted parameter, a definition, or a self-cited theorem. The only notable self-references are the authors' own tools and prior work: MLASP [30] is used as tool T1 for Q-23, the authors' GitHub [6] hosts full results, and related-work entries include LLMParser [23] and OpenLogParser [24]. Using a previously published, externally validated capacity-planning tool as a tool under test does not make the LLM accuracy measurement circular: the benchmark tests whether the agent invokes and formats the tool's output correctly, not whether MLASP's internal model is correct. The 'first to combine predictive ML models with LLMs in AIOps' claim in Section 7 is a novelty assertion supported by the authors' own knowledge, but it is not load-bearing for any measured result. The undocumented human expert grading in Section 2.2 and the internal mismatch between Table 1's Q-24 'starting 40 days ago' and Table 5's 'starting 48 hours ago' are validity and reproducibility concerns, not circularity: they do not show that any reported quantity is equivalent to the paper's inputs by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric free parameters are fit; the evaluation depends on design choices (25 tasks, 10 repetitions, SR/AR categories), on the reliability of human expert grading, and on the correct functioning of LangGraph/LangChain and provider endpoints. No new entities are postulated; tools T1-T9 are integrations of existing components such as the Kubernetes API, Prometheus, WireMock, and MLASP.

assumptions (3)
  • domain assumption Human expert evaluation of response correctness is a valid ground-truth source for the 25 ITOM tasks.
    Section 2.2 defines Accuracy as determined through human expert evaluation; Section 4.1 reports accuracy from that judgment. If grading is inconsistent, all accuracy comparisons are affected.
  • domain assumption The ReAct implementation in LangGraph/LangChain correctly mediates tool calls for all evaluated models.
    Section 3.2 relies on default ReAct prompts and the abstraction layer; Section 6 notes local vLLM could not replicate results due to a missing bind_tools feature, showing dependence on framework behavior.
  • domain assumption LLM provider endpoints (OpenAI, Anthropic, MistralAI) were stable and representative during testing.
    Section 6, construct validity, acknowledges dependence on provider serving capabilities; model versions and behavior can change over time.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Empowering AIOps: Leveraging Large Language Models for IT Operations Management." pith.science (2026). https://pith.science/paper/R3ZXKJW3

@misc{pith2026250112461,
  author       = {Pith},
  title        = {Pith review of: Empowering AIOps: Leveraging Large Language Models for IT Operations Management},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3ZXKJW3}},
  note         = {Machine review of arXiv:2501.12461}
}
read the original abstract

The integration of Artificial Intelligence (AI) into IT Operations Management (ITOM), commonly referred to as AIOps, offers substantial potential for automating workflows, enhancing efficiency, and supporting informed decision-making. However, implementing AI within IT operations is not without its challenges, including issues related to data quality, the complexity of IT environments, and skill gaps within teams. The advent of Large Language Models (LLMs) presents an opportunity to address some of these challenges, particularly through their advanced natural language understanding capabilities. These features enable organizations to process and analyze vast amounts of unstructured data, such as system logs, incident reports, and technical documentation. This ability aligns with the motivation behind our research, where we aim to integrate traditional predictive machine learning models with generative AI technologies like LLMs. By combining these approaches, we propose innovative methods to tackle persistent challenges in AIOps and enhance the capabilities of IT operations management.

Figures

Figures reproduced from arXiv: 2501.12461 by the authors.

Figure 1
Figure 1. An example workflow for LLM performance evalu [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 27 canonical work pages

  1. [1]

    Applying AIOps Platforms to Broader Datasets Will Create Unique Business Insights

    AIOps 2016. Applying AIOps Platforms to Broader Datasets Will Create Unique Business Insights. https://www.gartner.com/en/documents/3364418. Empowering AIOps: Leveraging Large Language Models for IT Operations Management FSE ’25, June 23-June 25, 2025, Trondheim, Norway

    work page arXiv 2016

  2. [2]

    Tse-Hsun Chen, Mark D Syer, Weiyi Shang, Zhen Ming Jiang, Ahmed E Hassan, Mohamed Nasser, and Parminder Flora. 2017. Analytics-driven load testing: An industrial experience report on load testing of large-scale systems. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engi- neering in Practice Track (ICSE-SEIP) . 243–252

    work page 2017

  3. [3]

    Qian Cheng, Doyen Sahoo, Amrita Saha, Wenzhuo Yang, Chenghao Liu, Gerald Woo, Manpreet Singh, Silvio Saverese, and Steven CH Hoi. 2023. Ai for it opera- tions (aiops) on cloud platforms: Reviews, opportunities and challenges. arXiv preprint arXiv:2304.04661 (2023)

    arXiv 2023

  4. [4]

    Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023. A survey of chain of thought reasoning: Advances, frontiers and future.arXiv preprint arXiv:2309.15402 (2023)

    arXiv 2023

  5. [5]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. ArXiv abs/2404.16130 (2024). https://api.semanticscholar.org/CorpusID:269363075

    arXiv 2024

  6. [6]

    Evaluat- ing LLM Agents for AIOps on Red Hat OpenShift - A full implementation example of agents with LangGraph and LangChain on Red Hat OpenShift

    Evaluating LLM Agents for AIOps on Red Hat OpenShift 2024. Evaluat- ing LLM Agents for AIOps on Red Hat OpenShift - A full implementation example of agents with LangGraph and LangChain on Red Hat OpenShift. https://github.com/eartvit/llm-agents-on-ocp

    work page 2024

  7. [7]

    Maloy Jyoti Goswami. [n.d.]. Challenges and Solutions in Integrating AI with Multi-Cloud Architectures. ([n. d.])

    work page

  8. [8]

    Hongcheng Guo, Jian Yang, Jiaheng Liu, Liqun Yang, Linzheng Chai, Jiaqi Bai, Junran Peng, Xiaorong Hu, Chao Chen, Dongfeng Zhang, Xu Shi, Tieqiao Zheng, Liangfan Zheng, Bo Zhang, Ke Xu, and Zhoujun Li. 2023. OWL: A Large Language Model for IT Operations. ArXiv abs/2309.09298 (2023). https://api.semanticscholar.org/CorpusID:262043747

    arXiv 2023

Show all 43 references
  1. [9]

    Fateme Hadadi, Qinghua Xu, Domenico Bianculli, and Lionel C. Briand. 2024. Anomaly Detection on Unstable Logs with GPT Models. ArXiv abs/2406.07467 (2024). https://api.semanticscholar.org/CorpusID:270379599

    2024

  2. [10]

    Anna Hedström, Leander Weber, Dilyara Bareeva, Daniel Krakowczyk, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M. C. Höhne

  3. [11]

    Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang

  4. [12]

    Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jia-Yuan Gu, Zhuangbin Chen, Jieming Zhu, and Michael R. Lyu. 2023. A Large-Scale Eval- uation for Log Parsing Techniques: How Far Are We? Proceedings of the 33rd ACM SIGSOFT International Symposium on Software T...

    2023

  5. [13]

    Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojciech Samek, and Sebastian Lapuschkin. 2020. Towards Best Practice in Explaining Neural Network Decisions with LRP. arXiv:1910.09840 [cs.LG] https://arxiv.org/abs/1910.09840

    2020 arXiv

  6. [14]

    Kubernetes - Production-Grade Container Orchestration

    Kubernetes 2022. Kubernetes - Production-Grade Container Orchestration. https://kubernetes.io/

    2022

  7. [15]

    Bharath Kumar. 2022. Challenges and Solutions for Integrating AI with Multi- Cloud Architectures. International Journal of Multidisciplinary Innovation and Research Methodology, ISSN: 2960-2068 1, 1 (2022), 71–77

    2022

  8. [16]

    LangChain - a framework for develop- ing applications powered by large language models (LLMs)

    LangChain 2024. LangChain - a framework for develop- ing applications powered by large language models (LLMs). https://python.langchain.com/docs/introduction/

    2024

  9. [17]

    LangGraph - Building Agents as Graphs

    LangGraph 2024. LangGraph - Building Agents as Graphs. https://langchain- ai.github.io/langgraph/

    2024

  10. [18]

    Kolodner, Dean H

    Anna Levin, Shelly Garion, Elliot K. Kolodner, Dean H. Lorenz, Katherine Barabash, Mike Kugler, and Niall McShane. 2019. AIOps for a Cloud Object Stor- age Service. In 2019 IEEE International Congress on Big Data (BigDataCongress) . IEEE, 165–169. https://doi.org/10.1109/bigda...

    2019

  11. [19]

    Yichen Li, Yintong Huo, Renyi Zhong, Zhihan Jiang, Jinyang Liu, Junjie Huang, Jiazhen Gu, Pinjia He, and Michael R. Lyu. 2024. Go Static: Contextualized Logging Statement Generation. ArXiv abs/2402.12958 (2024). https://api.semanticscholar. org/CorpusID:267760100

    2024 arXiv

  12. [20]

    Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yuhang Chen, Yanqing Zhao, Hao Yang, and Yanfei Jiang. 2023. Interpretable Online Log Analysis Using Large Language Models with Prompt Strategies. 2024 IEEE/ACM 32nd International Conference on Program Comprehension ...

    2023

  13. [21]

    LlamaIndex - a framework for building context-augmented generative AI applications with LLMs including agents and workflows

    LlamaIndex 2024. LlamaIndex - a framework for building context-augmented generative AI applications with LLMs including agents and workflows. https://docs.llamaindex.ai/en/stable/

    2024

  14. [22]

    Lucy Ellen Lwakatare, Aiswarya Raj, Ivica Crnkovic, Jan Bosch, and Helena Holm- ström Olsson. 2020. Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions. Information and Software Technol- ogy 127 (2020), 106368. https://d...

    2020

  15. [23]

    Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Husn Chen, and Shaowei Wang

  16. [24]

    Zeyang Ma, Dong Jae Kim, and Tse-Husn Chen. 2024. OpenLogParser: Unsuper- vised Parsing with Open-Source Large Language Models. ArXiv abs/2408.01585 (2024). https://api.semanticscholar.org/CorpusID:271709638

    2024 arXiv

  17. [25]

    Prometheus - Monitoring System and Timeseries Database

    Prometheus 2022. Prometheus - Monitoring System and Timeseries Database. https://prometheus.io/

    2022

  18. [26]

    2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2024), 1209–1221

    LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing. 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2024), 1209–1221. https://api.semanticscholar.org/CorpusID:269123285

    2024

  19. [27]

    Red Hat OpenShift - the industry’s leading hybrid cloud application platform powered by Kubernetes

    Red Hat OpenShift 2022. Red Hat OpenShift - the industry’s leading hybrid cloud application platform powered by Kubernetes. https://www.redhat.com/en/technologies/cloud-computing/openshift

    2022

  20. [28]

    Sai, Akash Kumar Mohankumar, and Mitesh M

    Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2020. A Survey of Evaluation Metrics Used for NLG Systems. arXiv:2008.12009 [cs.CL] https://arxiv.org/abs/2008.12009

    2020 arXiv

  21. [29]

    Pydantic - a widely used data validation library for Python

    Pydantic 2024. Pydantic - a widely used data validation library for Python. https://docs.pydantic.dev/latest/

    2024

  22. [30]

    Arthur Vitui and Tse-Hsun Peter Chen. 2021. MLASP: Machine learning assisted capacity planning. Empirical Software Engineering 26 (2021), 1–27. https://api. semanticscholar.org/CorpusID:236930377

    2021

  23. [31]

    Arthur Vitui and Tse-Hsun Peter Chen. 2024. MLOLET - Machine Learning Optimized Load and Endurance Testing: An industrial experience report. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE’24). IEEE Pres...

    2024

  24. [32]

    Komal Sarda, Zakeya Namrud, Marin Litoiu, Larisa Shwartz, and Ian Watts. 2024. Leveraging Large Language Models for the Auto-remediation of Microservice Applications: An Experimental Study. In SIGSOFT FSE Companion. https://api. semanticscholar.org/CorpusID:271099029

    2024

  25. [33]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    2023 arXiv

  26. [34]

    WireMock: Mock the APIs You Depend On

    WireMock 2022. WireMock: Mock the APIs You Depend On. https://wiremock.org/

    2022

  27. [35]

    vLLM - a fast and easy-to-use library for LLM inference and serving

    vLLM 2024. vLLM - a fast and easy-to-use library for LLM inference and serving. https://docs.vllm.ai/en/latest/

    2024

  28. [36]

    Rajmohan, and Dongmei Zhang

    Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, S. Rajmohan, and Dongmei Zhang

  29. [37]

    Junjielong Xu, Ruichun Yang, Yintong Huo, Chengyu Zhang, and Pinjia He. 2024. DivLog: Log Parsing with Prompt Enhanced In-Context Learning.2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2024), 2457–2468. https://api.semanticscholar.org/CorpusID:269123195

    2024

  30. [38]

    WireMock Metrics - extended WireMock with Prometheus metrics and global random string payload ResponseTransformer

    WireMock Metrics 2023. WireMock Metrics - extended WireMock with Prometheus metrics and global random string payload ResponseTransformer. https://github.com/eartvit/wiremock-metrics2

    2023

  31. [39]

    Rajmohan, Yingnong Dang, Changhua Pei, Dan Pei, Qingwei Lin, and Dongmei Zhang

    Zhaoyang Yu, Ming-Jie Ma, Chaoyun Zhang, Si Qin, Yu Kang, Chetan Bansal, S. Rajmohan, Yingnong Dang, Changhua Pei, Dan Pei, Qingwei Lin, and Dongmei Zhang. 2024. MonitorAssistant: Simplifying Cloud Service Monitoring via Large Language Models. In SIGSOFT FSE Companion. https:/...

    2024

  32. [40]

    2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2024), 1–12

    UniLog: Automatic Logging via LLM and In-Context Learning. 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) (2024), 1–12. https://api.semanticscholar.org/CorpusID:267523731

    2024

  33. [42]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629

    2023 arXiv

  34. [2023]

    arXiv:2202.06861 [cs.LG] https://arxiv.org/ abs/2202.06861

    Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. arXiv:2202.06861 [cs.LG] https://arxiv.org/ abs/2202.06861

    arXiv

  35. [2024]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24)

    Xpert: Empowering Incident Management with Query Recommendations via Large Language Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Association for Computing Machinery, New York, NY, USA, Article 92, 1...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.