{"id":"e814e7c3-d5a9-49a8-8562-1f6ac9af090b","arxiv_id":"2501.12461","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Tool-using LLM agents can resolve many Kubernetes IT operations tasks; GPT-4o led advanced multi-tool tasks while Anthropic models led simple ones, and Mixtral 8x22B failed with hallucinations.","lead":"An industry study evaluates 10 large language models as tool-using agents for IT operations tasks on a Kubernetes platform, measuring accuracy, response time, and token cost. A smart generalist might read it for practical guidance on choosing LLMs for AIOps assistants, though the results rest on a small, self-assessed benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy ranking rests on undocumented human grading and only four advanced tasks; objective scoring is needed before the GPT-4o 100% claim can be accepted.","rationale":"The paper is an experience report, and the public repository plus the detailed latency and token-count tables are genuine supporting evidence; the authors' good faith is not in question. The central ranking claim, however, is only as strong as the accuracy measurement, and Section 2.2 leaves that measurement underspecified. The simple-reasoning margin between Claude 3.5 Sonnet (95.23%) and Mistral Largest (94.76%) is one trial out of 210, and the advanced-reasoning comparison is built on only four tasks, two of which share nearly the same workflow. Under those conditions, undocumented human grading or a single workflow-specific failure mode can move the headline conclusions. The internal Q-24 discrepancy (40 days vs 48 hours) reinforces that the exact task text is not pinned down in the manuscript. None of this is fatal, because the repository may resolve it, so the reader's conditional disposition remains appropriate; the explicit conditions should include deterministic objective scoring and reconciliation of the Q-24 query text.","tokens_in":17499,"tokens_out":9940,"duration_ms":105783,"concrete_test":"Use the public repository [6] to reconstruct the executed workload: inspect the query definitions (especially Q-24: \"40 days ago\" vs \"48 hours ago\"), the tool traces, and the per-run final answers. Score each of the ten runs per model with a deterministic oracle for the objectively checkable outputs (operator names/versions, pod/service summaries, computed timestamps, Prometheus service/metric lists, CSV content, and existence of the plot file), then compare the per-model simple/advanced accuracy values to Table 2. Also report a grading rubric and have an independent second grader score a random 20% subset to compute inter-rater reliability. If objective scoring does not reproduce Table 2, the model ranking is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim (GPT-4o is 100% on advanced multi-tool tasks; Anthropic models are best on simple reasoning), Table 2 must be a reliable measurement. The load-bearing weakness is that the accuracy measure is not shown to be reliable. Section 2.2 says accuracy is \"determined through human expert evaluation,\" but gives no rubric, grader count, inter-rater reliability, or independence statement. Most outputs are objectively checkable (timestamps, operator names, service names, metric values, CSV content, file names), so a deterministic scoring protocol was feasible; its absence means a few inconsistent grading decisions can reorder the closely clustered simple-reasoning results and inflate the advanced-reasoning 100% figure. This is compounded by sample size: Table 1 contains only four advanced-reasoning queries (Q-21, Q-22, Q-24, Q-25), ten repetitions each, and Q-24/Q-25 are variants of the same Prometheus lookup-and-plot workflow. A single systematic failure mode can move advanced-reasoning accuracy by 25 percentage points, so the GPT-4o 100% figure is a thin basis for the broader claim that tool-using LLM agents automate many common Kubernetes operations tasks. Reproducibility is also undercut by an internal mismatch: Table 1's Q-24 says \"starting 40 days ago,\" while Table 5's trace of Q-24 says \"starting 48 hours ago.\" Without reconciling that mismatch and adding objective scoring, the accuracy table cannot be independently confirmed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical evaluation of ReAct-based LLM agents equipped with nine custom tools (e.g., time lookup, Prometheus metric retrieval, RAG, and MLASP capacity planning) for IT operations tasks on Red Hat OpenShift. The authors define 25 queries, 21 of them classified as simple reasoning and 4 as advanced reasoning, run each query 10 times against 10 LLMs, and measure accuracy, response time, and token usage. The central claims are that Anthropic models are best for simple reasoning, OpenAI models are best for advanced reasoning, and GPT-4o reaches 100% accuracy on the advanced multi-tool tasks, while Mixtral 8x22B performs poorly and tends to hallucinate. The paper also discusses lessons learned about memory components, prompt specificity, and tool chaining, and it points to a public GitHub repository for the implementation and detailed results.","tokens_in":17796,"tokens_out":5653,"duration_ms":53953,"significance":"If the measurement issues are resolved, this is a useful experience report for practitioners choosing LLM-based agents for Kubernetes/OpenShift operations. The paper's strengths are its real OpenShift testbed, a custom workload, ten repetitions per query, joint reporting of accuracy, latency, and token cost, and a public code repository. The practical recommendations about model selection are potentially valuable. However, the accuracy measurements are not currently auditable, and the advanced-reasoning evidence base is very small, so the comparative claims in Table 2 go beyond what the reported data support. The paper is not proposing a new method; its contribution is an empirical comparison and lessons learned, which is appropriate for the venue if the measurement protocol is strengthened.","major_comments":[{"comment":"The accuracy metric is the load-bearing outcome of the paper, but Section 2.2 defines it only as \"determined through human expert evaluation,\" and Section 3.3 does not specify the scoring protocol. No rubric, grader count, grader independence, or inter-rater reliability is reported. Because many outputs are objectively checkable (timestamps, service names, port numbers, file names, CSV contents), the authors should either provide a deterministic scoring procedure or a detailed rubric with inter-rater reliability before the model ranking in Table 2 can be accepted. Without this, even a small number of inconsistent grading decisions can reorder the closely clustered simple-reasoning results and alter the advanced-reasoning percentages.","section":"Section 2.2, Section 3.3, Table 2"},{"comment":"The advanced-reasoning comparison rests on four queries (Q-21, Q-22, Q-24, Q-25), ten repetitions each, and Q-24 and Q-25 are variants of the same Prometheus lookup-and-plot workflow. Consequently, one systematic failure mode can move a model's AR accuracy by 25 percentage points, and the gap between GPT-4o's 100% and Claude 3.5 Sonnet's or Claude 3 Opus's 95% is not statistically meaningful. The paper should report per-query accuracy, confidence intervals or significance tests, and ideally a larger and more diverse set of advanced queries before claiming that OpenAI models are \"better on advanced reasoning.\"","section":"Section 4.1, Table 1, Table 2"},{"comment":"The definition of the evaluated query Q-24 differs between Table 1 and Table 5. Table 1 states the metric range begins \"starting 40 days ago\" and instructs \"Return only the file name and nothing else,\" whereas Table 5's step 1 states \"starting 48 hours ago\" and \"Return only the content string of the tool and nothing else.\" The authors need to reconcile this mismatch and state which wording produced the results in Table 2, since accuracy scoring depends on the exact expected output.","section":"Table 1 (Q-24) and Table 5"},{"comment":"The experimental configuration was changed after observing model behavior: the memory component was disabled because it caused incorrect timestamp answers, and the Q-07 prompt was modified to include the product name, reportedly improving performance. These post hoc adjustments are disclosed, but the reported accuracy numbers are therefore generated under a tuned configuration, and the paper does not report the effect of the original configuration or control for the changes. An ablation or a clear statement of the exact configuration used for each result in Table 2 is needed to support the cross-model comparison.","section":"Section 5.1, Section 5.3"}],"minor_comments":[{"comment":"The metrics subsection defines \"Latency and Throughput,\" but the paper reports only response times; no throughput results appear in Section 4.2 or the tables. The authors should either add throughput measurements or remove throughput from the metric definition.","section":"Section 2.2"},{"comment":"The table heading says \"average response times\" while the rows are labeled P-50 and P-90; the paper should clarify whether these are averages of per-query percentiles or percentile values computed over the pooled repetitions.","section":"Table 3"},{"comment":"The text says \"OpenAI models generally respond the fastest on both SR and AR queries,\" but the immediately following sentence identifies Claude 3 Haiku as the fastest for SR queries based on P-50 values. This is confusing and should be reconciled.","section":"Section 4.2"},{"comment":"There is a typo: \"Kubernete-based\" should be \"Kubernetes-based.\"","section":"Section 3.1"},{"comment":"The claim that this research is \"the first to combine predictive machine learning models with LLMs in AIOps\" is a strong novelty assertion based on a brief related-work discussion and the authors' own MLASP tool. The claim should be qualified or supported with a more systematic literature search.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"This is a reasonable experience-report submission for the FSE Companion track, but the accuracy measurement is not yet auditable. The central fixes are to provide an objective scoring protocol or a detailed rubric with inter-rater reliability, reconcile the Q-24 definition mismatch, and either expand or carefully qualify the advanced-reasoning claims. The novelty claim in Section 7 should be softened. I see no evidence of misconduct, but the paper's reliance on self-authored tools and self-citations warrants editorial attention during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful experience report, not a breakthrough. It applies ReAct, RAG, and tool calling to a realistic OpenShift setup, adds an ML capacity-planning tool to the agent, and gives the first apples-to-apples accuracy/latency/token comparison of ten models on 25 ops tasks. The failure taxonomy (hallucination, truncation, query deflection) and the cost discussion are genuinely useful for practitioners picking a model. The repo is public, and the authors list threats to validity honestly. They get credit for that.\n\nThe soft spots are real, though none kills the paper as an experience report. The central accuracy numbers rely on human expert grading (Section 2.2) with no rubric, no grader count, and no inter-rater reliability. Most outputs are objective: timestamps, service names, file names, CSV content. A deterministic scoring script was feasible; without it, even small grading inconsistencies can reorder the close simple-reasoning results. That is a load-bearing problem for the claims that Anthropic is best on simple reasoning and GPT-4o is 100% on advanced reasoning.\n\nThe advanced-reasoning result is also built on four tasks (Q-21, Q-22, Q-24, Q-25), two of which are variants of the same Prometheus lookup-and-plot workflow. Ten repetitions per task means one systematic failure mode moves accuracy by 25 points, so 100% is a thin basis for the general statement about automating Kubernetes operations.\n\nThere is also an internal mismatch in the text: Table 1 asks Q-24 and Q-25 to start \"40 days ago,\" while the Table 5 trace for Q-24 says \"48 hours ago.\" Until that is reconciled, independent confirmation of the pipeline is hard.\n\nThe design changes—disabling memory and modifying Q-07 after initial runs—are disclosed, and I don't fault the authors for reporting them, but they are post hoc adjustments and should be stated as such in the protocol. The \"first to combine predictive ML with LLMs in AIOps\" claim deserves a broader literature check; it is self-referential and probably too strong.\n\nWho this is for: practitioners choosing an LLM for operations automation, and SE researchers interested in agent evaluation. It deserves serious review, but as a major-revision candidate: I would send it to referees only after the authors add an objective scoring protocol, confidence intervals or per-task counts, and fix the Q-24 discrepancy. If the repo verifies the tables, this could be a solid industry-facing benchmark. As is, it is a useful data point, not a demonstrated fact.","headline":"Useful practitioner benchmark of LLM agents for OpenShift ops; send to review, but the accuracy ranking needs objective scoring and more advanced reasoning tasks before the GPT-4o 100% claim carries weight.","tokens_in":18307,"tokens_out":3199,"would_cite":true,"duration_ms":33580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool-using LLM agents can automate routine Kubernetes IT operations, with GPT-4o solving every multi-step task it was given.","keywords":["AIOps","large language models","LLM agents","tool use","ReAct","Kubernetes","capacity planning","LLM evaluation"],"falsifier":"Re-run the 25 queries with automated scoring that checks objectively verifiable outputs—exact filenames, timestamps, CSV values, plotted metric ranges—and compare GPT-4o's and Claude 3.5 Sonnet's advanced-reasoning accuracy; if GPT-4o's 100% does not hold under that check, the claimed superiority on multi-tool tasks is not established.","tokens_in":17322,"feed_emoji":"🤖","tokens_out":9388,"duration_ms":81268,"temperature":0.7,"pith_summary":"This paper claims that an LLM equipped with a small set of typed tools can act as a practical assistant for IT operations management, and it measures how well ten models do on twenty-five realistic queries against a container-orchestration platform. The authors report that accuracy splits by task complexity: the Claude family leads on simple reasoning, while the GPT-4 family leads on advanced reasoning that chains two or more tools, with GPT-4o reaching 100% accuracy on the advanced multi-tool tasks. They also report that cost and latency vary widely—OpenAI models are the most token-efficient, Anthropic models the most verbose, and smaller models can still be viable for simple tasks. The paper further shows that a machine-learning capacity-planning tool can be folded into the agent, letting it move from answering questions toward making proactive parameter adjustments. If these results are borne out, organizations could automate many routine platform-management chores by choosing the model to match the task's complexity.","feed_headline":"LLM agents solve Kubernetes ops; GPT-4o perfect on hard tasks","feed_subtitle":"OpenAI leads multi-tool workflows, Anthropic wins simple queries, and Mixtral hallucinates responses.","key_machinery":"The load-bearing mechanism is the ReAct loop—a prompting pattern in which the model alternates written reasoning with concrete tool calls and observes the results before deciding the next step—wrapped in an agent-orchestration layer and connected to nine Python tools with typed interfaces. The tools let the model go beyond its training data: time conversion, service and pod summaries, metric-name lookup, metric-range retrieval, plotting, documentation retrieval over a vector database (RAG), and a machine-learning capacity-planning tool. The evaluation machinery also includes a ten-fold repetition of every query and a human-expert accuracy judgment, which is what turns anecdotal demonstrations into reported percentages.","core_discovery":"The paper's central claim is that an LLM agent built on a reasoning-and-acting loop (ReAct) and equipped with nine purpose-built tools—time conversion, namespace service and pod summaries, metric-name lookup, metric-range retrieval, plotting, documentation retrieval through a vector database (RAG), and a machine-learning capacity-planning tool—can resolve common IT-operations queries with high accuracy. In a ten-fold evaluation on 25 tasks, the top models reached 95–100% accuracy: Claude 3.5 Sonnet and Claude 3 Opus led simple-reasoning tasks, while GPT-4o solved all advanced multi-tool tasks and GPT-4 Turbo came close. The paper also claims that accuracy alone is not enough: response time and token cost vary by an order of magnitude across models, and failures take recognizable forms—hallucinated dates, truncated outputs, wrong tool orders, timeouts—that can be mitigated by prompt specificity and by disabling conversational memory for task-oriented queries.","pith_inferences":["Inference: because many of the 25 tasks have objectively checkable outputs—exact filenames, timestamps, CSV values, plotted metric ranges—the human-grading step could be replaced by automated verification, which would make the accuracy comparison reproducible without expert annotators.","Inference: the failure taxonomy (hallucination, deflection, truncation, timeouts) suggests a natural next benchmark: measure whether adding a verification step, such as forcing the agent to read back tool outputs, closes the accuracy gap between GPT-4o and the Claude models on multi-step tasks.","Inference: the paper's observation that a small 7B model did reasonably on simple tasks implies that local or on-premises deployments, where data cannot leave the environment, could use smaller open-weight models for low-complexity operations, at lower cost and with acceptable accuracy."],"forward_implications":["Routine operational queries—service discovery, metric retrieval, documentation lookups, and capacity planning—can be handed to a tool-using LLM agent on a container-orchestration platform.","Task complexity should drive model selection: cheaper and smaller models are viable for simple-reasoning tasks, while multi-step tool-chaining queries justify the cost of top-tier GPT-4 models.","Token consumption and latency are first-order operational costs, so the most accurate model is not automatically the best choice for continuous monitoring or high-volume queries.","Removing conversational memory improves correctness on timestamp-sensitive tasks, which argues for stateless task-oriented assistants unless contextual follow-up is explicitly needed.","Integrating a predictive machine-learning tool into the agent extends it from answering questions to recommending parameter configurations that meet a target key performance indicator (KPI)."],"supporting_citations":[{"why":"Supplies the ReAct reasoning-and-acting framework that defines how the agents choose tools and observe results.","marker":"[38]"},{"why":"Supplies the rationale for using human expert evaluation to judge open-ended LLM outputs.","marker":"[28]"},{"why":"Provides the integration layer that connects the LLMs to the tool definitions and orchestrates calls.","marker":"[16]"},{"why":"Provides the graph-based agent execution model used to run the ReAct loop.","marker":"[17]"},{"why":"The machine-learning capacity-planning tool whose integration with the LLM is one of the paper's contributions.","marker":"[30]"},{"why":"Supplies the retrieval-augmented generation approach used to answer documentation and procedure queries.","marker":"[5]"},{"why":"The enterprise container-platform distribution whose namespaces, services, and metrics are the subject of the operation tasks.","marker":"[27]"},{"why":"The companion repository containing the full per-query accuracy, latency, and token results summarized in the paper.","marker":"[6]"}],"fun_headline_variants":["LLM agents solve IT ops tasks with 9 custom tools","GPT-4o perfect on hard ops; Claude tops simple tasks","Tool-armed LLMs hit high accuracy on ops queries","IT ops via LLM agents: speed, cost, and accuracy","Kubernetes ops tamed by LLM tool-use agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All reported accuracy figures rest on human expert grading that is described but not shown, with no published rubric or measure of agreement between graders, so if that grading is biased or inconsistent the reported model rankings could change.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents solve IT ops tasks with 9 custom tools","GPT-4o perfect on hard ops; Claude tops simple tasks","Tool-armed LLMs hit high accuracy on ops queries","IT ops via LLM agents: speed, cost, and accuracy","Kubernetes ops tamed by LLM tool-use agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1353,"prompt_tokens":897,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":513,"tokens_out":456,"duration_ms":4891,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:09:38.864840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 25 queries with automated scoring that checks objectively verifiable outputs—exact filenames, timestamps, CSV values, plotted metric ranges—and compare GPT-4o's and Claude 3.5 Sonnet's advanced-reasoning accuracy; if GPT-4o's 100% does not hold under that check, the claimed superiority on multi-tool tasks is not established.","supporting_citations":[{"cited_title":"WireMock Metrics - extended WireMock with Prometheus metrics and global random string payload ResponseTransformer","cited_arxiv_id":null,"evidence_quote":"Supplies the ReAct reasoning-and-acting framework that defines how the agents choose tools and observe results."},{"cited_title":"LangChain - a framework for develop- ing applications powered by large language models (LLMs)","cited_arxiv_id":null,"evidence_quote":"Provides the integration layer that connects the LLMs to the tool definitions and orchestrates calls."},{"cited_title":"LangGraph - Building Agents as Graphs","cited_arxiv_id":null,"evidence_quote":"Provides the graph-based agent execution model used to run the ReAct loop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The machine-learning capacity-planning tool whose integration with the LLM is one of the paper's contributions."},{"cited_title":"Red Hat OpenShift - the industry’s leading hybrid cloud application platform powered by Kubernetes","cited_arxiv_id":null,"evidence_quote":"The enterprise container-platform distribution whose namespaces, services, and metrics are the subject of the operation tasks."},{"cited_title":"Evaluat- ing LLM Agents for AIOps on Red Hat OpenShift - A full implementation example of agents with LangGraph and LangChain on Red Hat OpenShift","cited_arxiv_id":null,"evidence_quote":"The companion repository containing the full per-query accuracy, latency, and token results summarized in the paper."}],"review_version":1}