CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

Jinkun Zhao; Junle Wang; Kui Zhang; Lei Huang; Ruibo Chen; Wenjun Wu; Xingchuang Liao; Xingjian Zhang; Yuanshuai Wang

REVIEW 5 major objections 5 minor 52 references

CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CoE-Ops claims that a two-stage router—a zero-shot LLM classifier with retrieved context and a benchmark-selected task–expert mapping—beats single AIOps specialists and larger Mixture-of-Experts models on DevOps-Eval question-answering.

desk verdict Incremental CoE extension with a useful idea, but the reported gains are uninterpretable because the RAG context and task-expert mapping are taken from the same benchmark used for evaluation. read the letter →

arxiv 2507.22937 v1 pith:RKELWF2J submitted 2025-07-25 cs.CL cs.AI

Jinkun Zhao , Yuanshuai Wang , Xingjian Zhang , Ruibo Chen , Xingchuang Liao , Junle Wang , Lei Huang , Kui Zhang

show 1 more author

Wenjun Wu

This is my paper · ORCID

classification cs.CLcs.AI

keywords CollaborationofExpertsAIOpsDevOpsretrieval-augmentedgenerationexpertroutingLLMensembleOps-Evalmixture

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoE-Ops addresses a practical bottleneck in AIOps: no single DevOps model covers all lifecycle phases, and deciding which specialist should answer a question is itself hard, especially for abstract tasks like Build, Code, and Release. The paper proposes a two-stage expert router in which a general-purpose LLM classifies the incoming question using a prompt-supplied task list plus retrieved similar examples, and then a fixed task–expert mapping forwards the question to the specialist model with the best per-task accuracy. On the DevOps-Eval benchmark, the paper reports routing-accuracy gains of up to 72.28% on high-level tasks over the fine-tuned Bench-CoE baseline, answer-accuracy gains of up to 8% over the best single expert, and gains of up to 14% over the 56B-parameter mixtral-8x7b-instruct Mixture-of-Experts model. If these numbers hold, dynamic composition of small, untuned specialists becomes a credible alternative to one large model in the DevOps domain.

What carries the argument

The central mechanism is the two-stage expert router. Stage 1 is a general-purpose LLM classifier running zero-shot on a prompt that contains the task list and, when retrieval-augmented generation is active, a few retrieved question–task examples; Stage 2 is a static task–expert mapping $M^*_i = \arg\max_M (1/N_i)\sum_j I(M(X_{ij})=A_{ij})$ that sends each task to the expert with the highest measured accuracy. The RAG module encodes a context corpus, retrieves the nearest examples by dot-product similarity $\mathrm{sim}(q,c)=q\cdot c$, and inserts them into the classifier prompt, with retrieval probabilities given by a softmax over similarity scores. Because the LLM classifier may return an "unknown" label, a fallback expert with the highest average accuracy handles those inputs. Expert models themselves are not fine-tuned; they are prompted zero-shot with chain of thought and asked to answer in a fixed multiple-choice format.

What would settle it

Rebuild the router using only the training split of DevOps-Eval for both the retrieval corpus and the task–expert mapping, with the test split held out, and compare per-task accuracy against the reported numbers; if accuracy drops toward the single-expert baseline, the reported gains are leakage. A quicker check is to inspect the retrieved context for a sampled test question: if it contains the identical question with its correct answer label, the retrieval step is directly supplying the answer.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in AIOps question-answering is not the individual expert models but the routing between them, and that routing can be made scalable without any training. To show this, CoE-Ops replaces the fine-tuned discriminative classifier of earlier two-stage routers with a zero-shot general-purpose LLM, adds retrieval-augmented generation so high-level task labels like Build or Test can be inferred from retrieved examples, and derives the task-expert mapping from per-task answer accuracy on the benchmark. Across four expert sets and two task sets, the paper reports consistent accuracy gains: up to 8% over the best single expert, up to 14% over the larger MoE model, and routing-accuracy improvements of 37.54% on low-level tasks and 72.28% on high-level tasks over the Bench-CoE baseline. The framework claims dual scalability: model scalability, because swapping experts only changes the task-expert mapping, and task scalability, because changing the task taxonomy only changes the prompt.

Load-bearing premise

The headline improvements assume the test questions are not used to build the retrieval context or to choose which specialist handles each task, yet the paper says the retrieval context comes from the evaluation set and the task–expert mapping is chosen from accuracy on the same benchmark.

Editorial extensions

If this is right

On the low-level Task Set A, CoE-Ops with Classifier 2 reaches 100% routing accuracy, so the remaining question-answering accuracy is bounded by the specialist experts rather than by the router.
Switching among four expert sets required only changing the prompt and the task-expert mapping, so newly added or replaced expert models can be absorbed without retraining or fine-tuning.
Retrieval-augmented generation lifts high-level routing accuracy for Classifier 1 from 13.91% to 43.84%, showing that retrieval context is the main driver of high-level task routing in this design.
An ensemble of small specialists (7B–27B parameters) surpasses the 56B-parameter mixtral-8x7b-instruct by up to 14% on the tested sets, supporting the paper's efficiency claim that small-model ensembles can beat larger MoE models.
Because the classifier runs zero-shot with a prompt-supplied task list, the framework can be pointed at a new task taxonomy by editing the prompt, which the paper frames as task scalability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

We infer that the same two-stage router would transfer to other multi-specialist question-answering domains, because the only domain-specific pieces—the task list, the expert pool, and the retrieval corpus—are all swappable without retraining.
A testable extension the paper does not run: restrict the retrieval corpus to examples from the same task label or exclude the exact query from retrieval, to separate genuine routing generalization from nearest-neighbor answer leakage.
We infer that the reported 100% low-level routing accuracy means the router is no longer the bottleneck on Task Set A; future gains there would have to come from better specialist models or from instance-level routing within a task.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper extends the authors' earlier Bench-CoE framework by replacing the fine-tuned classifier with an LLM and adding RAG, applied to DevOps-Eval QA. The writing is clear and the experimental sweep is broad. But the central accuracy claims are not supported as stated, because the RAG context is the eval split of the same benchmark on which accuracy is measured, and the task-expert mapping is chosen by accuracy on that same benchmark. That makes the routing numbers look like a lookup rather than generalization.

What's genuinely useful: applying two-stage routing to AIOps QA with a zero-shot LLM classifier plus RAG is a reasonable engineering contribution, even if the routing idea itself comes from Composition of Experts and Bench-CoE. The paper is honest about borrowing the framework, and it addresses some practical concerns like API failures and hallucination. The experiments cover multiple expert sets and both low- and high-level tasks, with precision/recall/F1 reported, not just accuracy. The "unknown" class fallback is a nice touch.

The soft spots are load-bearing. Section V-B says the RAG context is "the eval split from the DevOps-EVAL dataset," while accuracy is measured on the same benchmark. If that split is the test split, the retriever can return the query itself (or a near-duplicate) together with its label, and the LLM classifier can read the answer off the retrieved context. The 100% accuracy for Classifier 2 on Task Set A is exactly the signature of that leak. Separately, Eq. 5 selects the best expert per task by accuracy on the benchmark, so the router is optimized on the same data used for evaluation. That's parameter fitting on the test set. Also, the headline "72% improvement" is actually a 72-percentage-point gap between Bench-CoE (4.94%) and Classifier 2 (77.22%) on Task Set B, not a relative improvement. No error bars, no repeated runs, and API-based models can vary.

These flaws don't kill the approach. A proper train/validation split and a fixed mapping could salvage it. But as reported, the numbers can't be taken at face value. I'd send this to peer review because the engineering setup is plausible and the fix is straightforward, but it would need a re-run on a clean split before the claims can be trusted. A serious referee can help the authors produce something useful for AIOps tooling.

Referee Report

5 major / 5 minor

Summary. The paper proposes CoE-Ops, a collaboration-of-experts framework for AIOps multiple-choice question answering. It combines a two-stage router: a general-purpose LLM classifier (optionally augmented with retrieval over a context library) assigns each query to a DevOps task category, and a precomputed task-expert mapping then routes the query to a specialized LLM. The authors evaluate on the DevOps-Eval benchmark, using an English low-level task set (Task Set A) and a Chinese high-level task set (Task Set B), with four expert sets and several baselines including Bench-CoE, random routing, and Mixtral-8x7b-instruct. The central claims are that CoE-Ops improves routing accuracy by up to 72%, improves final answer accuracy over single specialists by up to about 8%, and outperforms larger MoE models by up to about 14%.

Significance. If the empirical claims were valid, the paper would make a useful contribution: a training-free, modular way to compose specialized LLMs with a scalable task classifier. The framework idea is clear, the paper covers multiple expert configurations, and the authors correctly identify model and task scalability as important design goals. However, the evaluation as reported is not trustworthy: the RAG context and the task-expert mapping appear to be derived from the same benchmark on which accuracy is measured, which can inflate or entirely manufacture the headline gains. The lack of error bars and the small margins in several tables further weaken the conclusions. As written, the paper does not establish its central empirical contribution.

major comments (5)

[Section V-B, Tables VII-VIII] The RAG context is stated to be 'the eval split from the DEVOPS-EVAL dataset' (Section V-B), while the evaluation is performed on the same benchmark. If the eval split is the same set of questions used as test queries, the retriever in Eq. (12) can return the very question being classified together with its task label, and Prompt 3 lets the LLM classifier read the answer directly from the retrieved context. This invalidates the classification-accuracy results, including the claimed 72.28-point improvement on Task Set B for Classifier 2. The authors must either use a genuinely disjoint context set or state clearly and convincingly that the eval split used for retrieval is not the same split used for evaluation.
[Section IV-A, Eqs. (4)-(5), Table II] The task-expert mapping is constructed by selecting, for each task, the expert with the highest per-task accuracy on the benchmark (Eqs. (4)-(5)). If this benchmark is DevOps-Eval, the same benchmark on which final accuracy is reported, then the router is given an oracle mapping on the test data. This makes the comparison against Bench-CoE and single models unfair: CoE-Ops is effectively evaluated with test-set labels baked into the routing policy. The mapping must be estimated on a separate training or validation split that is not used for evaluation, and the split must be explicitly disclosed.
[Section V-D, RQ3, Tables IX-X] The efficiency claim is not supported by the reported experiments. The comparison against Mixtral-8x7b-instruct is not an efficiency comparison in the usual sense: no inference latency, cost, throughput, or parameter-normalized metric is reported. Moreover, Mixtral is a general-purpose model, whereas the CoE-Ops expert pool includes DevOps-tuned models such as CodeFuse-DevOps-Model; outperforming a general-purpose model on a domain benchmark is unsurprising and does not demonstrate that 'integration of smaller models via LLM ensembling' is efficient. RQ3 needs a cost-aware evaluation to be meaningful.
[Section V, Tables III-VIII] All results are single numbers from a single pass, with no error bars, confidence intervals, or significance tests. This matters because several headline differences are small: in Tables IV, V, and VI the CoE-Ops gain over the best single expert is about 2.1, 1.0, and 1.4 percentage points, respectively, which could easily fall within run-to-run variance of stochastic LLM API calls. The suspicious 100% accuracy of Classifier 2 on Task Set A (Table VII) needs replication and a detailed explanation. Without repeated runs or variance reporting, the claimed improvements are not statistically established.
[Abstract and Section V-C] The headline '72% improvement in routing accuracy' is computed as an absolute percentage-point difference (77.22 - 4.94 = 72.28), not a relative improvement of 72%. The same applies to the 37.54-point gain on Task Set A. The abstract and the answer to RQ2 should be restated as percentage-point improvements, and the relative improvement should be reported separately if desired. This is not a purely cosmetic issue because it materially changes the magnitude of the claimed contribution.

minor comments (5)

[Section V-B] The retrieval setup is underspecified: the paper does not state the number of retrieved contexts (top-k), the choice of similarity threshold, or whether the query itself was excluded from the context library. These details are necessary for reproducibility.
[Section VI] The threats-to-validity section discusses API failures and hallucination but does not mention the most serious threat, namely the potential contamination of the RAG context and task-expert mapping with the evaluation data. This should be acknowledged and addressed directly.
[Section V-A, Table II] The table has formatting issues: the superscript footnote markers are hard to parse, and the mapping derivation is not explained in the caption. The caption should state which data were used to construct the mapping.
[Throughout] There are numerous typos and spacing inconsistencies (e.g., 'DEVOPS-EV AL', 'AIOps' variants) that should be cleaned up in a revision.
[Section VII] The paper does not provide a reproducibility artifact such as code, prompts, or the exact data split used. Given the centrality of the experimental protocol, releasing these would greatly strengthen the paper.

Circularity Check

2 steps flagged · score 7.0 of 10

Reported routing and accuracy gains are not independent: the RAG context is built from the eval split and the task-expert mapping is fitted to benchmark accuracy, so the headline improvements partly reduce to test-set lookup and argmax selection.

self definitional [Section V-B (RAG setup), Section IV-C Eqs. (12)-(13), Prompt 3, and Tab. VII]
"For the RAG component, we employed the eval split from the DEVOPS-EVAL dataset as the context. ... The formula for the Retriever's similarity calculation is shown in (13). sim(q, c) = q· c. ... You can refer to the following examples of questions and their corresponding tasks to decide the current question's task:{context}"

The context database is the eval split, and the similarity in Eq. (13) is q·c. A query taken from that split maximizes similarity with itself, so Eq. (12) retrieves that very example (or a near-duplicate) into the classifier prompt. Prompt 3 then supplies the example with its 'corresponding task' label, so the label the classifier is asked to predict is already present in its input. The 100% classification accuracy of Classifier 2 on Task Set A (Tab. VII) is the signature of this self-retrieval. The claimed routing improvements (37.54% and 72.28%) are therefore a label-lookup on the evaluation set, not an independent generalization result.
fitted input called prediction [Section IV-A Eqs. (4)-(5), Fig. 2 caption, and Section V-B evaluation tables (Tables III-VI)]
"To evaluate the performance of numerous expert models across diverse task domains, we established a comprehensive benchmark and constructed the “Task-Expert” mapping presented in Tab. II. ... M∗i = arg max M∈M (1/Ni Σ I(M(Xij)=Aij)) ... It is then routed to a designated expert model based on a pre-established ”Task-Expert” mapping derived from existing benchmark (Step-2)."

Eq. (5) selects, for each task Ti, the expert with the highest accuracy computed on the benchmark. The same benchmark, with the same task definitions, is then used in Section V-B to report CoE-Ops accuracy. Thus CoE-Ops is evaluated with a task-expert map that was fitted to the evaluation labels; the reported accuracy is close to a per-task argmax over test accuracy rather than an independent routing prediction. Comparisons against single models and MoE baselines, which were not given this test-label information, therefore reflect an oracle-like selection advantage rather than demonstrated generalization.

full rationale

The paper's central routing claim is circular under its own setup: the RAG context is explicitly the eval split, and the retriever's dot-product similarity makes the query itself the top retrieved context, so the LLM classifier reads the task label from the prompt instead of predicting it. The task-expert mapping is similarly fitted by Eq. (5) to maximize accuracy on the same benchmark used for final evaluation. Because single-model and MoE baselines do not have access to test labels, the reported 8% and 14% accuracy advantages are not independent evidence of generalization. The final answer text is still generated by the routed expert, so the end-to-end answer accuracy is not entirely constructed by the lookup; this keeps the score below 8 or 10. I did not count the self-citation to Bench-CoE [49] as load-bearing circularity by itself, since the framework's own equations and experiments carry the argument; the circularity comes from the eval-split RAG context and the test-fitted mapping.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of fitted design choices and domain assumptions. The task-expert mapping is an argmax over per-task accuracy (Eq. 5) and the unknown-class expert is chosen by average accuracy (Eq. 8), both fitted to the same benchmark used for scoring. The RAG library is the eval split, and retrieval hyperparameters are not reported. The domain assumptions are that DevOps-Eval accuracy is a valid proxy for AIOps capability and that the embedding model retrieves useful examples.

free parameters (3)

Task-expert mapping (Table II) = Per-task argmax expert chosen by accuracy on DevOps-Eval
Eq. (5) defines M*_i as the model with highest accuracy per task; the paper does not state that this mapping is estimated on a separate training split from the final evaluation, so it is a fitted quantity.
RAG context library and retrieval hyperparameters = Eval split used as context; top-k, threshold, and chunking not reported
Section V-B uses the eval split as retrieval context. The number of retrieved examples and similarity threshold are unspecified, and these choices directly affect classifier accuracy.
Unknown-class fallback expert = Model with highest average accuracy across all tasks
Eq. (8) selects M*_unk by argmax over total benchmark accuracy, another fit to the evaluation data.

assumptions (3)

domain assumption DevOps-Eval multiple-choice accuracy is a valid proxy for AIOps question-answering capability
All conclusions are drawn from one benchmark; the paper does not validate on real DevOps logs or tasks.
domain assumption The two-stage CoE routing architecture from Composition of Experts [48] and Bench-CoE [49] is a sound foundation
The paper adopts this architecture without re-deriving or testing its assumptions about classifier and router behavior.
domain assumption All-MiniLM-L6-v2 embeddings provide suitable similarity for retrieving relevant task examples
The retriever is not ablated; the paper assumes dot-product similarity returns examples that help the LLM classify high-level tasks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering." pith.science (2026). https://pith.science/paper/RKELWF2J

@misc{pith2026250722937,
  author       = {Pith},
  title        = {Pith review of: CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKELWF2J}},
  note         = {Machine review of arXiv:2507.22937}
}

read the original abstract

With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework's capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy.

Figures

Figures reproduced from arXiv: 2507.22937 by the authors.

**Figure 2.** Framework of the CoE Based on Two-Stage Expert Routing. The input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** AIOps Scenario Illustrating Model and Task Scalability. The end [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 5.** Figure 5: Capability Radar Chart of CoE-Ops with Expert Set 1 on DevOps [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Capability Radar Chart of CoE-Ops with Expert Set 3 on DevOps [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 6.** Figure 6: Capability Radar Chart of CoE-Ops with Expert Set 2 on DevOps [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 8.** Figure 8: Capability Radar Chart of CoE-Ops with Expert Set 4 on DevOps [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Heatmap Visualization of Classifier 1’s Confusion Matrix on DevOps [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 12.** Figure 12: Heatmap Visualization of Classifier 2’s Confusion Matrix on DevOps [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]

**Figure 11.** Figure 11: Heatmap Visualization of Classifier 1’s Confusion Matrix on DevOps [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

**Figure 13.** Figure 13: Capability Radar Chart of Comparative Experiments on DevOps [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]

**Figure 14.** Figure 14: Capability Radar Chart of Comparative Experiments on DevOps [PITH_FULL_IMAGE:figures/full_fig_p010_14.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 27 canonical work pages

[1]

& Serrano, N

Ebert, C., Gallardo, G., Hernantes, J. & Serrano, N. DevOps. IEEE Software. 33, 94-100 (2016)

work page 2016
[2]

& Tanveer, B

Jabbari, R., Ali, N., Petersen, K. & Tanveer, B. What is DevOps? A systematic mapping study on definitions and practices. Proceedings Of The Scientific Workshop Proceedings Of XP2016 . pp. 1-11 (2016)

work page 2016
[3]

& Meirelles, P

Leite, L., Rocha, C., Kon, F., Milojicic, D. & Meirelles, P. A survey of DevOps concepts and challenges. ACM Computing Surveys (CSUR) . 52, 1-35 (2019)

work page 2019
[4]

Towards MLOps: A DevOps Tools Recommender System for Machine Learning System

Shah, P., Ahmad, N. & Beg, M. Towards MLOps: A DevOps Tools Recommender System for Machine Learning System. ArXiv Preprint ArXiv:2402.12867. (2024)

work page Pith review arXiv 2024
[5]

& Hirschl, S

Kreuzberger, D., K ¨uhl, N. & Hirschl, S. Machine learning operations (mlops): Overview, definition, and architecture. IEEE Access . 11 pp. 31866-31879 (2023)

work page 2023
[6]

& Alsarayrah, K

Zarour, M., Alzabut, H. & Alsarayrah, K. MLOps best practices, chal- lenges and maturity models: A systematic literature review. Information And Software Technology. pp. 107733 (2025)

work page 2025
[7]

& Shan, T

Shan, R. & Shan, T. Enterprise LLMOps: Advancing Large Language Models Operations Practice. 2024 IEEE Cloud Summit . pp. 143-148 (2024)

work page 2024
[8]

& Torre-Bastida, A

Diaz-De-Arcaya, J., L ´opez-De-Armentia, J., Mi ˜n´on, R., Ojanguren, I. & Torre-Bastida, A. Large Language Model Operations (LLMOps): Defi- nition, Challenges, and Lifecycle Management. 2024 9th International Conference On Smart And Sustainable Technologies (SpliTech) . pp. 1-4 (2024)

work page 2024

Show all 52 references

[9]

& Chua, J

Tantithamthavorn, C., Palomba, F., Khomh, F. & Chua, J. MLOps, LLMOps, FMOps, and Beyond. IEEE Software. 42, 26-32 (2025)

2025
[10]

& Akhtar, Z

Pahune, S. & Akhtar, Z. Transitioning from MLOps to LLMOps: Navi- gating the Unique Challenges of Large Language Models. Information. 16, 87 (2025)

2025
[11]

& Huang, P

Dang, Y ., Lin, Q. & Huang, P. Aiops: real-world challenges and research innovations. 2019 IEEE/ACM 41st International Conference On Software Engineering: Companion Proceedings (ICSE-Companion) . pp. 4-5 (2019)

2019
[12]

& Gerndt, M

Notaro, P., Cardoso, J. & Gerndt, M. A survey of aiops methods for failure management. ACM Transactions On Intelligent Systems And Technology (TIST). 12, 1-45 (2021)

2021
[13]

A systems approach to effective aiops implementation

Hua, Y . A systems approach to effective aiops implementation. (Mas- sachusetts Institute of Technology,2021)

2021
[14]

& Almeida, A

Diaz-De-Arcaya, J., Torre-Bastida, A., Z ´arate, G., Mi˜n´on, R. & Almeida, A. A joint study of the challenges, opportunities, and roadmap of mlops and aiops: A systematic survey. ACM Computing Surveys . 56, 1-30 (2023)

2023
[15]

& Anabathula, T

Mondru, A., Shreyas, R. & Anabathula, T. A Roadmap to Success: Strategies and Challenges in Adopting Aiops for it Operations. Interna- tional Journal Of Interpreting Enigma Engineers (IJIEE) . 1 (2024)

2024
[16]

Beyond DevOps: The Evolution Toward Intelligent IT Operations with AIOps and MLOps

Brahmandam, B. Beyond DevOps: The Evolution Toward Intelligent IT Operations with AIOps and MLOps. (2025)

2025
[17]

& Anwar, A

Faraz Khan, A., Khan, A., Mohamed, A., Ali, H., Moolinti, S., Haroon, S., Tahir, U., Fazzini, M., Butt, A. & Anwar, A. LADs: Leveraging LLMs for AI-Driven DevOps. ArXiv E-prints. pp. arXiv-2502 (2025)

2025
[18]

& Neelanath, V

Krishnamurthy, D. & Neelanath, V . Establishing a Robust LLMOps Framework for Intelligent Automation: Strategies and Best Practices. 2025 Emerging Technologies For Intelligent Systems (ETIS) . pp. 1-5 (2025)

2025
[19]

Key Performance Indicators of Artificial Intelligence For IT Operations (AIOPS)

Mulongo, N. Key Performance Indicators of Artificial Intelligence For IT Operations (AIOPS). 2024 International Symposium On Networks, Computers And Communications (ISNCC) . pp. 1-8 (2024)

2024
[20]

Chen, Z., Li, J., Chen, P., Li, Z., Sun, K., Luo, Y ., Mao, Q., Yang, D., Sun, H. & Yu, P. Harnessing Multiple Large Language Models: A Survey on LLM Ensemble. (2025), https://arxiv.org/abs/2502.18036

2025 arXiv
[21]

& Jacquenet, F

Varangot-Reille, C., Bouvard, C., Gourru, A., Ciancone, M., Schaeffer, M. & Jacquenet, F. Doing More with Less – Implementing Routing Strategies in Large Language Model-Based Systems: An Extended Survey. (2025), https://arxiv.org/abs/2502.00409

2025 arXiv
[22]

& Zou, J

Chen, L., Zaharia, M. & Zou, J. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. (2023), https://arxiv.org/abs/2305.05176

2023 arXiv
[23]

& Lin, B

Jiang, D., Ren, X. & Lin, B. LLM-Blender: Ensembling Large Lan- guage Models with Pairwise Ranking and Generative Fusion. (2023), https://arxiv.org/abs/2306.02561

2023 arXiv
[24]

Sukhbaatar, S., Golovneva, O., Sharma, V ., Xu, H., Lin, X., Rozi `ere, B., Kahn, J., Li, D., Wen-Yih, Weston, J. & Li, X. Branch-Train- MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM. (2024), https://arxiv.org/abs/2403.07816

2024 arXiv
[25]

& Boyd-Graber, J

Si, C., Shi, W., Zhao, C., Zettlemoyer, L. & Boyd-Graber, J. Getting more out of mixture of language model reasoning experts. (2023), ArXiv Preprint ArXiv:2305.14628

2023 arXiv
[26]

Li, J., Zhang, Q., Yu, Y ., Fu, Q. & Ye, D. More agents is all you need. (2024), ArXiv Preprint ArXiv:2402.05120

2024 arXiv
[27]

& Zhong, Z

Zhang, Y ., Chen, Z. & Zhong, Z. Collaboration of experts: Achieving 80% top-1 accuracy on imagenet with 100m flops. (2021), ArXiv Preprint ArXiv:2107.03815

2021 arXiv
[28]

& West, R

ˇSakota, M., Peyrard, M. & West, R. Fly-swat or cannon? cost-effective language model choice via meta-modeling. Proceedings Of The 17th ACM International Conference On Web Search And Data Mining . pp. 606-615 (2024)

2024
[29]

& Yurochkin, M

Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y ., Solomon, J., Thompson, N. & Yurochkin, M. Large language model routing with benchmark datasets. (2023), ArXiv Preprint ArXiv:2309.15789

2023 arXiv
[30]

& Stoica, I

Ong, I., Almahairi, A., Wu, V ., Chiang, W., Wu, T., Gonzalez, J., Kadous, M. & Stoica, I. Routellm: Learning to route llms with pref- erence data, 2024. URL Https://arxiv. Org/abs/2406.18665

2024 arXiv
[31]

& Zheng, H

Huang, S., Pan, J. & Zheng, H. CCoE: A Compact LLM with Collab- oration of Experts. (2024), ArXiv Preprint ArXiv:2407.11686

2024 arXiv
[32]

& Kochmar, E

Maurya, K., Srivatsa, K. & Kochmar, E. SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models. (2024), ArXiv Preprint ArXiv:2408.08545

2024 arXiv
[33]

Stripelis, D., Hu, Z., Zhang, J., Xu, Z., Shah, A., Jin, H., Yao, Y ., Avestimehr, S. & He, C. Polyrouter: A multi-llm querying system. ArXiv E-prints. pp. arXiv-2408(2024)

2024
[34]

Stripelis, D., Hu, Z., Zhang, J., Xu, Z., Shah, A., Jin, H., Yao, Y ., Avestimehr, S. & He, C. TensorOpera Router: A Multi-Model Router for Efficient LLM Inference. (2024), ArXiv Preprint ArXiv:2408.12320

2024 arXiv
[35]

& Awadallah, A

Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V ., Lakshmanan, L. & Awadallah, A. Hybrid llm: Cost-efficient and quality- aware query routing. (2024), ArXiv Preprint ArXiv:2404.14618

2024 arXiv
[36]

Guha, N., Chen, M., Chow, T., Khare, I. & Re, C. Smoothie: Label free language model routing. Advances In Neural Information Processing Systems. 37 pp. 127645-127672 (2024)

2024
[37]

& Upadhyay, S

Hu, Q., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K. & Upadhyay, S. Routerbench: A benchmark for multi-llm routing system. (2024), ArXiv Preprint ArXiv:2403.12031

2024 arXiv
[38]

& Avestimehr, S

Zhang, T., Mehradfar, A., Dimitriadis, D. & Avestimehr, S. Leveraging uncertainty estimation for efficient llm routing. (2025), ArXiv Preprint ArXiv:2502.11021

2025 arXiv
[39]

& You, J

Feng, T., Shen, Y . & You, J. Graphrouter: A graph-based router for llm selections. (2024), ArXiv Preprint ArXiv:2410.03834

2024 arXiv
[40]

Yue, Y ., Zhang, G., Liu, B., Wan, G., Wang, K., Cheng, D. & Qi, Y . Masrouter: Learning to route llms for multi-agent systems. (2025), ArXiv Preprint ArXiv:2502.11133

2025 arXiv
[41]

& Kumar, S

Jitkrittum, W., Narasimhan, H., Rawat, A., Juneja, J., Wang, Z., Lee, C., Shenoy, P., Panigrahy, R., Menon, A. & Kumar, S. Universal Model Routing for Efficient LLM Inference. (2025), ArXiv Preprint ArXiv:2502.08773

2025 arXiv
[42]

& Zhou, J

Lu, K., Yuan, H., Lin, R., Lin, J., Yuan, Z., Zhou, C. & Zhou, J. Routing to the expert: Efficient reward-guided ensemble of large language models. (2023), ArXiv Preprint ArXiv:2311.08692

2023 arXiv
[43]

& Doan, K

Nguyen, Q., Hoang, D., Decugis, J., Manchanda, S., Chawla, N. & Doan, K. MetaLLM: A High-performant and Cost-efficient Dynamic Frame- work for Wrapping LLMs. (2024), ArXiv Preprint ArXiv:2407.10834

2024 arXiv
[44]

& Mao, Z

Zhao, Z., Jin, S. & Mao, Z. Eagle: Efficient training-free router for multi-llm inference. (2024), ArXiv Preprint ArXiv:2409.15518

2024 arXiv
[45]

& Chen, H

Wang, X., Liu, Y ., Cheng, W., Zhao, X., Chen, Z., Yu, W., Fu, Y . & Chen, H. Mixllm: Dynamic routing in mixed large language models. (2025), ArXiv Preprint ArXiv:2502.18482

2025 arXiv
[46]

& Others Sambanova sn40l: Scaling the ai memory wall with dataflow and composition of experts

Prabhakar, R., Sivaramakrishnan, R., Gandhi, D., Du, Y ., Wang, M., Song, X., Zhang, K., Gao, T., Wang, A., Li, X. & Others Sambanova sn40l: Scaling the ai memory wall with dataflow and composition of experts. 2024 57th IEEE/ACM International Symposium On Microar- chitecture (...

2024
[47]

& Huo, Z

Suo, J., Liao, X., Xiao, L., Ruan, L., Wang, J., Su, X. & Huo, Z. CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory. Proceedings Of The 30th ACM International Conference On Architectural Support For Programming Languages And Operating System...

2025
[48]

& Others Composition of Experts: A Modular Compound AI System Leveraging Large Language Models

Jain, S., Raju, R., Li, B., Csaki, Z., Li, J., Liang, K., Feng, G., Thakkar, U., Sampat, A., Prabhakar, R. & Others Composition of Experts: A Modular Compound AI System Leveraging Large Language Models. (2024), ArXiv Preprint ArXiv:2412.01868

2024 arXiv
[49]

Wang, Y ., Zhang, X., Zhao, J., Wen, S., Feng, P., Liao, S., Huang, L. & Wu, W. Bench-CoE: a Framework for Collaboration of Experts from Benchmark. ArXiv Preprint (2024), ArXiv:2412.04167

2024 arXiv
[50]

& Others Deepseek-v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C. & Others Deepseek-v3 technical report. (2024), ArXiv Preprint ArXiv:2412.19437

2024 arXiv
[51]

& Others Qwen3 technical report

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C. & Others Qwen3 technical report. (2025), ArXiv Preprint ArXiv:2505.09388

2025 arXiv
[52]

& Kallinteris, N

Khan, U. & Kallinteris, N. Autonomous Multi-Agent LLMs in Agile Development: A Framework for AI-Driven Collaboration. (2025)

2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

& Serrano, N

Ebert, C., Gallardo, G., Hernantes, J. & Serrano, N. DevOps. IEEE Software. 33, 94-100 (2016)

work page 2016

[2] [2]

& Tanveer, B

Jabbari, R., Ali, N., Petersen, K. & Tanveer, B. What is DevOps? A systematic mapping study on definitions and practices. Proceedings Of The Scientific Workshop Proceedings Of XP2016 . pp. 1-11 (2016)

work page 2016

[3] [3]

& Meirelles, P

Leite, L., Rocha, C., Kon, F., Milojicic, D. & Meirelles, P. A survey of DevOps concepts and challenges. ACM Computing Surveys (CSUR) . 52, 1-35 (2019)

work page 2019

[4] [4]

Towards MLOps: A DevOps Tools Recommender System for Machine Learning System

Shah, P., Ahmad, N. & Beg, M. Towards MLOps: A DevOps Tools Recommender System for Machine Learning System. ArXiv Preprint ArXiv:2402.12867. (2024)

work page Pith review arXiv 2024

[5] [5]

& Hirschl, S

Kreuzberger, D., K ¨uhl, N. & Hirschl, S. Machine learning operations (mlops): Overview, definition, and architecture. IEEE Access . 11 pp. 31866-31879 (2023)

work page 2023

[6] [6]

& Alsarayrah, K

Zarour, M., Alzabut, H. & Alsarayrah, K. MLOps best practices, chal- lenges and maturity models: A systematic literature review. Information And Software Technology. pp. 107733 (2025)

work page 2025

[7] [7]

& Shan, T

Shan, R. & Shan, T. Enterprise LLMOps: Advancing Large Language Models Operations Practice. 2024 IEEE Cloud Summit . pp. 143-148 (2024)

work page 2024

[8] [8]

& Torre-Bastida, A

Diaz-De-Arcaya, J., L ´opez-De-Armentia, J., Mi ˜n´on, R., Ojanguren, I. & Torre-Bastida, A. Large Language Model Operations (LLMOps): Defi- nition, Challenges, and Lifecycle Management. 2024 9th International Conference On Smart And Sustainable Technologies (SpliTech) . pp. 1-4 (2024)

work page 2024