{"id":"92b60058-0270-48a1-baaf-909ed607860d","arxiv_id":"2507.22937","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"CoE-Ops routes DevOps questions to specialized LLM experts using an LLM classifier plus retrieval, reporting gains on DevOps-Eval that are compromised by possible test-set leakage.","lead":"CoE-Ops combines several existing large language models into a question-answering system for IT operations, using a separate LLM and retrieved examples to decide which specialist model should answer each question. It reports accuracy gains on a DevOps benchmark, but the evaluation setup may leak the test answers into the router, so the headline numbers are not yet trustworthy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are not yet supported because both RAG context and task-expert mapping appear to use the same benchmark on which accuracy is measured.","rationale":"The reader's weakest assumption is essentially the same concern I would raise: no held-out evaluation is described, so both the RAG corpus and the task-expert mapping can leak test information. The strongest claim in the paper is quantitative, and the experimental section is the only support for it; if these numbers are contaminated, the contribution reduces to a plausible but unvalidated architecture. I considered whether \"eval split\" might mean a separate validation split distinct from the test split, but the paper never describes a train/test partition and reports accuracy on DevOps-Eval without such a caveat. The 100% classification result strongly indicates self-retrieval, and the task-expert mapping fitted on benchmark accuracy would give CoE-Ops an oracle-like selection advantage over single models. The concern lands, and the reader's REJECT verdict stands. The approach may still be salvageable with a proper split and released code, but the current claims are unsupported.","tokens_in":14554,"tokens_out":4259,"duration_ms":43822,"concrete_test":"Rerun the Section V-B/V-C experiments with an explicit split: construct the RAG corpus from the training split only and exclude the query itself from the top-k retrieval results; fit the task-expert mapping (Eq. 5) on training/validation accuracy only; then report accuracy on a held-out test split that is not in the RAG index and not used in mapping selection. If Classifier 2's Task Set A accuracy drops below 100% or the CoE-Ops margins over single experts shrink materially, the reported gains are leakage artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claims depend on a clean separation between information used to build the system (RAG context, task-expert mapping) and data used to measure it, and that separation is absent. Section V-B states that the RAG context is \"the eval split from the DEVOPS-EVAL dataset,\" while accuracy is measured on the same benchmark. If the eval split is the test set, the retriever can return the query itself (or a near-duplicate) together with its task label; Prompt 3 then lets the LLM classifier read the answer off the retrieved context. The 100% classification accuracy of Classifier 2 on Task Set A (Tab. VII) is exactly the signature of such self-retrieval. Separately, Section IV-A (Eqs. 4–5) selects the expert for each task by accuracy computed on the benchmark; fitting that mapping to the same data that is later used for evaluation means routing is optimized on the test set, so CoE-Ops is compared against single models with an oracle-like advantage. Neither the 8% improvement over single models nor the 14% MoE gain can be interpreted as generalization until evaluation is performed on a split that was not used for retrieval or mapping.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoE-Ops, a collaboration-of-experts framework for AIOps multiple-choice question answering. It combines a two-stage router: a general-purpose LLM classifier (optionally augmented with retrieval over a context library) assigns each query to a DevOps task category, and a precomputed task-expert mapping then routes the query to a specialized LLM. The authors evaluate on the DevOps-Eval benchmark, using an English low-level task set (Task Set A) and a Chinese high-level task set (Task Set B), with four expert sets and several baselines including Bench-CoE, random routing, and Mixtral-8x7b-instruct. The central claims are that CoE-Ops improves routing accuracy by up to 72%, improves final answer accuracy over single specialists by up to about 8%, and outperforms larger MoE models by up to about 14%.","tokens_in":14743,"tokens_out":7194,"duration_ms":75074,"significance":"If the empirical claims were valid, the paper would make a useful contribution: a training-free, modular way to compose specialized LLMs with a scalable task classifier. The framework idea is clear, the paper covers multiple expert configurations, and the authors correctly identify model and task scalability as important design goals. However, the evaluation as reported is not trustworthy: the RAG context and the task-expert mapping appear to be derived from the same benchmark on which accuracy is measured, which can inflate or entirely manufacture the headline gains. The lack of error bars and the small margins in several tables further weaken the conclusions. As written, the paper does not establish its central empirical contribution.","major_comments":[{"comment":"The RAG context is stated to be 'the eval split from the DEVOPS-EVAL dataset' (Section V-B), while the evaluation is performed on the same benchmark. If the eval split is the same set of questions used as test queries, the retriever in Eq. (12) can return the very question being classified together with its task label, and Prompt 3 lets the LLM classifier read the answer directly from the retrieved context. This invalidates the classification-accuracy results, including the claimed 72.28-point improvement on Task Set B for Classifier 2. The authors must either use a genuinely disjoint context set or state clearly and convincingly that the eval split used for retrieval is not the same split used for evaluation.","section":"Section V-B, Tables VII-VIII"},{"comment":"The task-expert mapping is constructed by selecting, for each task, the expert with the highest per-task accuracy on the benchmark (Eqs. (4)-(5)). If this benchmark is DevOps-Eval, the same benchmark on which final accuracy is reported, then the router is given an oracle mapping on the test data. This makes the comparison against Bench-CoE and single models unfair: CoE-Ops is effectively evaluated with test-set labels baked into the routing policy. The mapping must be estimated on a separate training or validation split that is not used for evaluation, and the split must be explicitly disclosed.","section":"Section IV-A, Eqs. (4)-(5), Table II"},{"comment":"The efficiency claim is not supported by the reported experiments. The comparison against Mixtral-8x7b-instruct is not an efficiency comparison in the usual sense: no inference latency, cost, throughput, or parameter-normalized metric is reported. Moreover, Mixtral is a general-purpose model, whereas the CoE-Ops expert pool includes DevOps-tuned models such as CodeFuse-DevOps-Model; outperforming a general-purpose model on a domain benchmark is unsurprising and does not demonstrate that 'integration of smaller models via LLM ensembling' is efficient. RQ3 needs a cost-aware evaluation to be meaningful.","section":"Section V-D, RQ3, Tables IX-X"},{"comment":"All results are single numbers from a single pass, with no error bars, confidence intervals, or significance tests. This matters because several headline differences are small: in Tables IV, V, and VI the CoE-Ops gain over the best single expert is about 2.1, 1.0, and 1.4 percentage points, respectively, which could easily fall within run-to-run variance of stochastic LLM API calls. The suspicious 100% accuracy of Classifier 2 on Task Set A (Table VII) needs replication and a detailed explanation. Without repeated runs or variance reporting, the claimed improvements are not statistically established.","section":"Section V, Tables III-VIII"},{"comment":"The headline '72% improvement in routing accuracy' is computed as an absolute percentage-point difference (77.22 - 4.94 = 72.28), not a relative improvement of 72%. The same applies to the 37.54-point gain on Task Set A. The abstract and the answer to RQ2 should be restated as percentage-point improvements, and the relative improvement should be reported separately if desired. This is not a purely cosmetic issue because it materially changes the magnitude of the claimed contribution.","section":"Abstract and Section V-C"}],"minor_comments":[{"comment":"The retrieval setup is underspecified: the paper does not state the number of retrieved contexts (top-k), the choice of similarity threshold, or whether the query itself was excluded from the context library. These details are necessary for reproducibility.","section":"Section V-B"},{"comment":"The threats-to-validity section discusses API failures and hallucination but does not mention the most serious threat, namely the potential contamination of the RAG context and task-expert mapping with the evaluation data. This should be acknowledged and addressed directly.","section":"Section VI"},{"comment":"The table has formatting issues: the superscript footnote markers are hard to parse, and the mapping derivation is not explained in the caption. The caption should state which data were used to construct the mapping.","section":"Section V-A, Table II"},{"comment":"There are numerous typos and spacing inconsistencies (e.g., 'DEVOPS-EV AL', 'AIOps' variants) that should be cleaned up in a revision.","section":"Throughout"},{"comment":"The paper does not provide a reproducibility artifact such as code, prompts, or the exact data split used. Given the centrality of the experimental protocol, releasing these would greatly strengthen the paper.","section":"Section VII"}],"recommendation":"reject","confidential_remarks":"The core evaluation appears to be contaminated: the RAG context and the task-expert mapping are derived from the same benchmark used for evaluation, so the headline routing and accuracy gains may be artifacts of test-set leakage. Even if the framework idea is plausible, the current manuscript does not support its central claims. I would recommend rejection, though a substantially revised version with a clean held-out split, a properly estimated task-expert mapping, and repeated-run statistics could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper extends the authors' earlier Bench-CoE framework by replacing the fine-tuned classifier with an LLM and adding RAG, applied to DevOps-Eval QA. The writing is clear and the experimental sweep is broad. But the central accuracy claims are not supported as stated, because the RAG context is the eval split of the same benchmark on which accuracy is measured, and the task-expert mapping is chosen by accuracy on that same benchmark. That makes the routing numbers look like a lookup rather than generalization.\n\nWhat's genuinely useful: applying two-stage routing to AIOps QA with a zero-shot LLM classifier plus RAG is a reasonable engineering contribution, even if the routing idea itself comes from Composition of Experts and Bench-CoE. The paper is honest about borrowing the framework, and it addresses some practical concerns like API failures and hallucination. The experiments cover multiple expert sets and both low- and high-level tasks, with precision/recall/F1 reported, not just accuracy. The \"unknown\" class fallback is a nice touch.\n\nThe soft spots are load-bearing. Section V-B says the RAG context is \"the eval split from the DevOps-EVAL dataset,\" while accuracy is measured on the same benchmark. If that split is the test split, the retriever can return the query itself (or a near-duplicate) together with its label, and the LLM classifier can read the answer off the retrieved context. The 100% accuracy for Classifier 2 on Task Set A is exactly the signature of that leak. Separately, Eq. 5 selects the best expert per task by accuracy on the benchmark, so the router is optimized on the same data used for evaluation. That's parameter fitting on the test set. Also, the headline \"72% improvement\" is actually a 72-percentage-point gap between Bench-CoE (4.94%) and Classifier 2 (77.22%) on Task Set B, not a relative improvement. No error bars, no repeated runs, and API-based models can vary.\n\nThese flaws don't kill the approach. A proper train/validation split and a fixed mapping could salvage it. But as reported, the numbers can't be taken at face value. I'd send this to peer review because the engineering setup is plausible and the fix is straightforward, but it would need a re-run on a clean split before the claims can be trusted. A serious referee can help the authors produce something useful for AIOps tooling.","headline":"Incremental CoE extension with a useful idea, but the reported gains are uninterpretable because the RAG context and task-expert mapping are taken from the same benchmark used for evaluation.","tokens_in":15348,"tokens_out":3226,"would_cite":false,"duration_ms":30213,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoE-Ops claims that a two-stage router—a zero-shot LLM classifier with retrieved context and a benchmark-selected task–expert mapping—beats single AIOps specialists and larger Mixture-of-Experts models on DevOps-Eval question-answering.","keywords":["Collaboration of Experts","AIOps","DevOps","retrieval-augmented generation","expert routing","LLM ensemble","DevOps-Eval","mixture of experts"],"falsifier":"Rebuild the router using only the training split of DevOps-Eval for both the retrieval corpus and the task–expert mapping, with the test split held out, and compare per-task accuracy against the reported numbers; if accuracy drops toward the single-expert baseline, the reported gains are leakage. A quicker check is to inspect the retrieved context for a sampled test question: if it contains the identical question with its correct answer label, the retrieval step is directly supplying the answer.","tokens_in":1773,"feed_emoji":"🔀","tokens_out":2904,"duration_ms":120696,"temperature":0.7,"pith_summary":"CoE-Ops addresses a practical bottleneck in AIOps: no single DevOps model covers all lifecycle phases, and deciding which specialist should answer a question is itself hard, especially for abstract tasks like Build, Code, and Release. The paper proposes a two-stage expert router in which a general-purpose LLM classifies the incoming question using a prompt-supplied task list plus retrieved similar examples, and then a fixed task–expert mapping forwards the question to the specialist model with the best per-task accuracy. On the DevOps-Eval benchmark, the paper reports routing-accuracy gains of up to 72.28% on high-level tasks over the fine-tuned Bench-CoE baseline, answer-accuracy gains of up to 8% over the best single expert, and gains of up to 14% over the 56B-parameter mixtral-8x7b-instruct Mixture-of-Experts model. If these numbers hold, dynamic composition of small, untuned specialists becomes a credible alternative to one large model in the DevOps domain.","feed_headline":"LLM router with retrieval lifts DevOps QA accuracy up to 14%","feed_subtitle":"Routing each query to the best specialist beats single AIOps experts and larger Mixture-of-Experts models.","key_machinery":"The central mechanism is the two-stage expert router. Stage 1 is a general-purpose LLM classifier running zero-shot on a prompt that contains the task list and, when retrieval-augmented generation is active, a few retrieved question–task examples; Stage 2 is a static task–expert mapping $M^*_i = \\arg\\max_M (1/N_i)\\sum_j I(M(X_{ij})=A_{ij})$ that sends each task to the expert with the highest measured accuracy. The RAG module encodes a context corpus, retrieves the nearest examples by dot-product similarity $\\mathrm{sim}(q,c)=q\\cdot c$, and inserts them into the classifier prompt, with retrieval probabilities given by a softmax over similarity scores. Because the LLM classifier may return an \"unknown\" label, a fallback expert with the highest average accuracy handles those inputs. Expert models themselves are not fine-tuned; they are prompted zero-shot with chain of thought and asked to answer in a fixed multiple-choice format.","core_discovery":"The paper's central claim is that the bottleneck in AIOps question-answering is not the individual expert models but the routing between them, and that routing can be made scalable without any training. To show this, CoE-Ops replaces the fine-tuned discriminative classifier of earlier two-stage routers with a zero-shot general-purpose LLM, adds retrieval-augmented generation so high-level task labels like Build or Test can be inferred from retrieved examples, and derives the task-expert mapping from per-task answer accuracy on the benchmark. Across four expert sets and two task sets, the paper reports consistent accuracy gains: up to 8% over the best single expert, up to 14% over the larger MoE model, and routing-accuracy improvements of 37.54% on low-level tasks and 72.28% on high-level tasks over the Bench-CoE baseline. The framework claims dual scalability: model scalability, because swapping experts only changes the task-expert mapping, and task scalability, because changing the task taxonomy only changes the prompt.","pith_inferences":["We infer that the same two-stage router would transfer to other multi-specialist question-answering domains, because the only domain-specific pieces—the task list, the expert pool, and the retrieval corpus—are all swappable without retraining.","A testable extension the paper does not run: restrict the retrieval corpus to examples from the same task label or exclude the exact query from retrieval, to separate genuine routing generalization from nearest-neighbor answer leakage.","We infer that the reported 100% low-level routing accuracy means the router is no longer the bottleneck on Task Set A; future gains there would have to come from better specialist models or from instance-level routing within a task."],"forward_implications":["On the low-level Task Set A, CoE-Ops with Classifier 2 reaches 100% routing accuracy, so the remaining question-answering accuracy is bounded by the specialist experts rather than by the router.","Switching among four expert sets required only changing the prompt and the task-expert mapping, so newly added or replaced expert models can be absorbed without retraining or fine-tuning.","Retrieval-augmented generation lifts high-level routing accuracy for Classifier 1 from 13.91% to 43.84%, showing that retrieval context is the main driver of high-level task routing in this design.","An ensemble of small specialists (7B–27B parameters) surpasses the 56B-parameter mixtral-8x7b-instruct by up to 14% on the tested sets, supporting the paper's efficiency claim that small-model ensembles can beat larger MoE models.","Because the classifier runs zero-shot with a prompt-supplied task list, the framework can be pointed at a new task taxonomy by editing the prompt, which the paper frames as task scalability."],"supporting_citations":[{"why":"Supplies the two-stage expert-routing architecture that CoE-Ops modifies and evaluates against.","marker":"[48]"},{"why":"The fine-tuned-classifier CoE baseline whose routing accuracy CoE-Ops exceeds, and the source of the benchmark-derived task-expert mapping idea.","marker":"[49]"},{"why":"Establishes routing large language models with benchmark datasets, the basis for choosing each task's best expert by per-task accuracy.","marker":"[29]"},{"why":"Defines the DeepSeek-V3 model family used as Classifier 2 and represents the MoE foundation-model paradigm compared in RQ3.","marker":"[50]"}],"fun_headline_variants":["Zero-shot LLM router boosts DevOps QA by up to 14%","Training-free routing lifts AIOps QA accuracy by 14%","Retrieval-augmented LLM router beats bigger MoE by 14%","CoE-Ops: specialist LLMs collaborate to answer DevOps queries","Task routing with LLM beats single AIOps experts by 8%"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The headline improvements assume the test questions are not used to build the retrieval context or to choose which specialist handles each task, yet the paper says the retrieval context comes from the evaluation set and the task–expert mapping is chosen from accuracy on the same benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM router boosts DevOps QA by up to 14%","Training-free routing lifts AIOps QA accuracy by 14%","Retrieval-augmented LLM router beats bigger MoE by 14%","CoE-Ops: specialist LLMs collaborate to answer DevOps queries","Task routing with LLM beats single AIOps experts by 8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2518,"prompt_tokens":1009,"completion_tokens":1509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":1413}},"tokens_in":625,"tokens_out":1509,"duration_ms":12115,"temperature":1.0,"reasoning_tokens":1413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:03:49.818874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the router using only the training split of DevOps-Eval for both the retrieval corpus and the task–expert mapping, with the test split held out, and compare per-task accuracy against the reported numbers; if accuracy drops toward the single-expert baseline, the reported gains are leakage. A quicker check is to inspect the retrieved context for a sampled test question: if it contains the identical question with its correct answer label, the retrieval step is directly supplying the answer.","supporting_citations":[{"cited_title":"Composition of Experts: A Modular Compound AI System Leveraging Large Language Models","cited_arxiv_id":"2412.01868","evidence_quote":"Supplies the two-stage expert-routing architecture that CoE-Ops modifies and evaluates against."}],"review_version":2}