{"id":"6b608d67-cec9-49e1-b152-1bfbd580b71d","arxiv_id":"2505.12594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multi-agent LLM framework converts natural language anomaly detection requests into runnable code across three AD libraries, with reported success rates above 90 percent.","lead":"AD-AGENT is a multi-agent system that turns plain English requests into runnable anomaly detection programs using PyOD, PyGOD, and TSLib. It reports high success rates in generating executable pipelines and competitive LLM model recommendations at low API cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reliability claim for generated scripts rests on a synthetic dry run (Section 2.1) that checks execution, not semantic correctness; the paper's own GAAN failure (Section 3.1) shows data-specific constraints can slip through, so Table 1's high success rate may overstate reliability on real user…","rationale":"The reader's weakest assumption is exactly that a successful synthetic dry run generalizes to the user's actual data, and the paper's own Failure Discussion in Section 3.1 provides a concrete counterexample (GAAN and binary targets). This is the most load-bearing concern because it directly targets the abstract's claim of producing 'reliable scripts': if a script executes on synthetic data but fails on real data, the headline success rate is not evidence of reliability for the intended non-expert user. The concern is not that the system is useless; it is that the evaluation metric is too weak to support the claim as stated. The system is open-sourced and the agent design is clearly described, so conditional acceptance with a request for semantic validation is appropriate. I do not see a reason to reject the paper outright, and the reader's conditional verdict already captures the needed revision. Hence I leave the verdict unchanged, while emphasizing that the concrete test above should be run before the reliability claim is taken at face value.","tokens_in":9382,"tokens_out":4118,"duration_ms":48607,"concrete_test":"Re-run AD-AGENT on PyGOD's GAAN for the five BOND datasets using the original (non-binary) label files, and record whether the Reviewer's synthetic-sample dry run passes and whether the generated script then runs and produces sane AUROC on the real data; also inspect the generated script for explicit binary-label handling. If dry-run success does not imply real-data success for GAAN, Table 1's reliability metric is not a valid proxy for 'reliable scripts.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AD-AGENT 'produces reliable scripts' is supported only by a Reviewer dry run on LLM-generated synthetic samples (Section 2.1). Table 1's success rate measures whether code executes without error, not whether the resulting pipeline is semantically correct on the user's actual data. The paper's own Failure Discussion (Section 3.1) confirms the gap: GAAN in PyGOD expects binary targets, but the pipeline sometimes passes values outside the valid range, and Pyraformer fails on input-size mismatches. These are precisely the classes of constraints a synthetic-sample dry run cannot detect, since synthetic samples can be constructed to satisfy the constraint while the real dataset does not. Moreover, the evaluation never measures detection quality of the generated pipelines on the benchmark datasets; only execution success and model-selection AUROC (where recommendations are evaluated separately from generated code) are reported. Therefore the headline reliability metric can be high even when scripts silently produce wrong or meaningless results, which is the scenario most damaging for non-expert users. Since the paper's own limitations section concedes that 'not all model or data-specific constraints can be automatically detected,' the abstract's unqualified 'reliable scripts' overstates what is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AD-AGENT, an LLM-driven multi-agent framework that generates end-to-end anomaly detection pipelines from natural-language instructions. The system coordinates specialized agents for intent parsing, library and model selection, documentation mining, code generation and review, and optional evaluation and tuning, using short-term shared memory and a long-term cache. It targets three AD libraries: PyOD for multivariate data, PyGOD for graph data, and TSLib for time series. Experiments report 90–100% execution success for generated pipelines, model-selection AUROC/F1 values above the average baseline, and latency/cost savings from long-term memory. The system is released open source at GitHub.","tokens_in":9726,"tokens_out":6188,"duration_ms":64289,"significance":"If the reliability claim were fully supported, AD-AGENT would be a valuable contribution to applied anomaly detection, lowering the programming barrier for non-experts and unifying three major AD libraries into a single workflow. The paper has concrete strengths: the architecture is clearly described, the implementation is open-sourced, and the cost/latency measurements in Table 1 and Table 2 are useful operational data. The model-selection experiments and the Optimizer results in Appendix B.3 provide some evidence that LLM reasoning can aid model choice. However, the central claim that AD-AGENT produces reliable scripts currently rests on execution-only validation, and the multi-agent design is not compared with simpler baselines. These gaps need to be addressed before the results can be regarded as conclusive.","major_comments":[{"comment":"The success rate in Table 1 is defined in Section 3.1 as whether the generated code runs without error, but the abstract's claim of producing reliable scripts requires semantic correctness on the user's data. The paper never reports detection quality, such as AUROC or F1, of the scripts generated by AD-AGENT on the benchmark datasets; the AUROC results in Figure 3 come from model recommendations, not from running the generated pipelines. A script that executes but silently misconfigures the model would still count as a success, so the headline reliability metric does not support the abstract's claim. Please either add an end-to-end evaluation of the generated scripts on benchmark data or qualify the claim to executable scripts.","section":"§2.1, Table 1"},{"comment":"The Reviewer validates code through a dry run on LLM-generated synthetic samples, and the Failure Discussion in Section 3.1 confirms that data-specific constraints such as GAAN requiring binary targets and Pyraformer input-size mismatches are missed by this validation. Since synthetic samples can be constructed to satisfy constraints that the user's real dataset violates, the dry run cannot guarantee correctness on arbitrary inputs. The paper's own Limitations section concedes that not all model or data-specific constraints can be automatically detected, so the unqualified reliable-scripts claim should be revised and the evaluation should report how often these failure modes occur on actual benchmark inputs.","section":"§2.1, §3.1"},{"comment":"The model-selection experiments evaluate the LLM's recommendations by averaging the AUROC or F1 of the recommended models, but they do not measure how well the full AD-AGENT pipeline performs end-to-end. There is also no baseline showing a single LLM prompt or a non-agent script-generation method, so the value added by multi-agent coordination is not isolated. Please add such baselines or an ablation to support the claim that the multi-agent architecture improves reliability or selection quality.","section":"§3.2, Appendix B.4"}],"minor_comments":[{"comment":"The text has numerous spacing and typographical issues, such as multigent in Section 1 and AD-AGENTdemonstrates in Section 3.1; please run a thorough spell-check and fix the LaTeX rendering.","section":"§1, §3.1"},{"comment":"The model name V AE is rendered with a spurious space in Table 3 and Table 4; fix the macro or escaping so it appears as VAE.","section":"Table 3, Table 4"},{"comment":"Section 3.1 says GPT-4o is used to build all agents, while Section 3.2 says o4-mini is used for recommendations; clarify which model is used for which agent and why.","section":"§3.1, §3.2"},{"comment":"The Optimizer experiment uses only the cardio dataset and reports no error bars or significance testing; in particular, DevNet shows no change (0.0323 to 0.0323), which suggests the tuning loop may not have engaged for that model. Please discuss this case.","section":"Appendix B.3"},{"comment":"The dry-run procedure for the Reviewer is underspecified: the paper should state how synthetic samples are generated, how many samples are used, what random seed is used, and how the validation loop terminates.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The evaluation gap in Section 3.1 is the main barrier to acceptance. The paper is a systems contribution with an open-source release, and the architecture is interesting, so the result is potentially salvageable through additional experiments or more careful claims. The stress-test concern about the synthetic dry run is valid and should be pressed in the revision. The self-citations are appropriate given the authors' prior work on these libraries; I see no citation-pattern problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: AD-AGENT is a solid systems paper, not a scientific breakthrough, and its headline claim needs one important qualifier: \"reliable\" currently means \"executes without error on a synthetic dry run,\" not \"produces semantically correct detection on your data.\" The paper is worth engaging, but the evaluation needs tightening before I'd trust the abstract.\n\nWhat's actually new: it's the first, as far as I know, multi-agent LLM framework that spans three AD libraries—PyOD, PyGOD, TSLib. Prior agents like Audit-LLM and Argos are domain-specific, and prior libraries are modality-specific. The integration is real, and the architecture is clearly described: intent parsing, selection, doc mining, code generation with iterative debugging, short-term/long-term memory. That's a legitimate contribution, especially with the open-source release.\n\nThe paper does several things well. It uses established benchmarks (ADBench, BOND, TSLib benchmarks). It reports honest costs, latency, and token usage. The failure discussion in Section 3.1 is refreshing—GAAN's binary-target constraint and Pyraformer input-size mismatches are exactly the kinds of data-specific constraints that a synthetic dry run misses. The long-term memory caching result is simple but useful.\n\nWhere the soft spots are: the reliability metric in Table 1 is explicitly \"code runs without error,\" and the Reviewer dry-runs on LLM-generated synthetic samples. That catches type errors and import problems but not silently wrong outputs. The paper's own GAAN example proves the gap, so the abstract's \"reliable scripts\" overstates what is demonstrated. Second, there is no baseline against a single LLM agent or a non-agent pipeline, so the multi-agent architecture's value is not isolated. Third, model selection (Section 3.2) is evaluated separately from the generated code, so we don't see end-to-end detection quality. Fourth, no error bars on success rates or AUROC. These are real but not fatal; the limitations section admits the first and third.\n\nThis paper is for people building LLM agents for automated machine learning or turning AD libraries into natural-language interfaces, and for AD practitioners who want a usable tool. It is a systems contribution, not a new method. It deserves a serious referee—I'd send it to review with a request for semantic validation of pipelines and at least a single-LLM baseline.","headline":"A workmanlike multi-agent LLM system that genuinely spans three AD libraries, but the reliability claim is about scripts executing, not about detection quality—send it to review, ask for a semantic check and a baseline.","tokens_in":10177,"tokens_out":2348,"would_cite":true,"duration_ms":25168,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AD-AGENT turns plain-language requests into executable anomaly-detection pipelines across data types.","keywords":["anomaly detection","multi-agent LLM framework","natural language to code","pipeline generation","model selection","PyOD","PyGOD","TSLib"],"falsifier":"Take the generated, dry-run-verified GAAN pipeline and run it on a graph dataset whose target attribute contains continuous values; if it crashes or produces invalid loss values, the paper's claimed validation reliability overstates what the Reviewer detects. Equivalently, measure the pipeline success rate on a held-out set of realistic user-uploaded datasets rather than the benchmark pairs in Table 1.","tokens_in":9179,"feed_emoji":"🤖","tokens_out":8364,"duration_ms":69605,"temperature":0.7,"pith_summary":"This paper presents AD-AGENT, a multi-agent LLM system that converts plain-language instructions, such as \"Detect anomalies in cardio.mat,\" into fully executable anomaly-detection scripts. It coordinates specialized agents for intent parsing, data preparation, library and model selection, documentation mining, and iterative code generation and debugging, integrating the PyOD, PyGOD, and TSLib libraries. The paper reports that the system produces runnable pipelines with 90–100 percent success across those libraries, and that its recommended models outperform the average baseline for most benchmark datasets. If this holds, domain experts who do not program would be able to run detection pipelines on multivariate, graph, and time-series data without learning library APIs.","feed_headline":"Natural language becomes a runnable anomaly-detection pipeline","feed_subtitle":"Specialized LLM agents write and debug code across PyOD, PyGOD, and TSLib with success rates of 90–100 percent.","key_machinery":"The machinery is a multi-agent architecture in which distinct LLM-based agents—Processor, Selector, Info Miner, Code Generator, Code Reviewer, and optional Evaluator and Optimizer—share a short-term session workspace and a long-term documentation cache. The load-bearing mechanism is the Generator–Reviewer loop: the Reviewer tests generated scripts on LLM-created synthetic samples, catches execution errors such as missing parameters or wrong import names, and triggers regeneration until the script runs, while the Selector uses the LLM's own knowledge of model families to choose a model when the user does not specify one.","core_discovery":"The paper's central claim is that AD-AGENT, by decomposing the anomaly-detection workflow into cooperating LLM-powered agents, can take a user's natural-language description of an anomaly-detection task and deliver a ready-to-run Python script that uses the appropriate library—PyOD for tabular and multivariate data, PyGOD for graph data, and TSLib for time series. The system does not just recommend code; it validates the generated script through a dry run on synthetic samples in a feedback loop between the Code Generator and Reviewer, and it caches model documentation in long-term memory to avoid repeated web searches. The paper reports per-library pipeline success rates of 100 percent (PyOD), 91.1 percent (PyGOD), and 90.0 percent (TSLib), and shows that LLM-recommended models achieve AUROC or F1 scores well above the average of all available models and close to the best model, while the long-term memory removes roughly ten seconds per lookup. The authors frame this as a step toward a general-purpose AD platform that makes specialized anomaly-detection libraries accessible to non-expert users.","pith_inferences":["The same division of labor—parse intent, mine docs, generate, dry-run-validate—should transfer to other library-heavy machine-learning tasks, such as automated feature engineering or forecasting, where the reviewer-agent role can be reused almost unchanged.","The reported success rates are computed on benchmark dataset-and-model pairs; a user's real data may deviate from those formats, so an end-to-end field rollout would likely show lower reliability than Table 1 suggests.","A concrete stress test inspired by the paper's failure discussion: hold out datasets that violate a model's documented constraints (e.g., continuous targets for GAAN, missing timestamps for TSLib) and measure the fraction of dry-run-verified pipelines that fail at execution on those datasets.","The long-term memory cache trades currency for speed: if a library changes a function signature between refreshes, cached documentation can steer the generator toward calls that no longer exist, so version-aware cache invalidation would be a natural next test."],"forward_implications":["A user who can describe an anomaly-detection task in words can obtain a runnable script without writing code, for the three supported data modalities and libraries.","The Generator–Reviewer loop automatically repairs common execution errors, such as missing constructor arguments like n_features for DeepSVDD, without user intervention.","The Selector's LLM-based model recommendation beats the average model baseline and approaches the best model's AUROC or F1 on most datasets across PyOD, PyGOD, and TSLib.","The long-term memory removes roughly ten seconds of web-search latency per lookup and eliminates its per-call cost after the first session.","Because the agents are loosely coupled, adding a new AD library or data modality requires only a new documentation source and model set, not a redesign of the workflow."],"supporting_citations":[{"why":"Supplies the PyOD library with LLM-powered model selection and the benchmark datasets and models used for the PyOD reliability tests.","marker":"Chen et al. (2024)"},{"why":"Provides the BOND benchmark datasets and the nine graph models used for the PyGOD evaluation.","marker":"Liu et al. (2022)"},{"why":"Provides the TSLib benchmark protocol and five time-series datasets (MSL, PSM, SMAP, SMD, SWaT) used for evaluation.","marker":"Wu et al. (2023)"},{"why":"Source of the 17 PyOD datasets adapted from ADBench for the reliability tests.","marker":"Han et al. (2022)"},{"why":"Supplies GPT-4o, the LLM that powers all agents in the main experiments.","marker":"OpenAI (2024)"},{"why":"Provides the Web Search tool used by the Info Miner to query model documentation.","marker":"OpenAI (2025b)"},{"why":"The PyGOD library itself, used as the graph-detection library integrated into the framework.","marker":"Liu et al. (2024b)"},{"why":"The TSLib library used for time-series detection in the framework.","marker":"Wang et al. (2024)"}],"fun_headline_variants":["AD-AGENT: LLM agents turn text into runnable AD pipelines","Multi-agent framework auto-builds anomaly detection code","Natural language to debugged anomaly detection scripts","LLM-driven agents handle end-to-end anomaly detection","From plain English to PyOD, PyGOD, TSLib pipelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a script that runs without error on the synthetic samples generated by the LLM will also run correctly on the user's actual data, including data-specific constraints like binary labels or fixed input sizes.","fun_headline_variants_meta":{"raw":{"variants":["AD-AGENT: LLM agents turn text into runnable AD pipelines","Multi-agent framework auto-builds anomaly detection code","Natural language to debugged anomaly detection scripts","LLM-driven agents handle end-to-end anomaly detection","From plain English to PyOD, PyGOD, TSLib pipelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1618,"prompt_tokens":938,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":596}},"tokens_in":554,"tokens_out":680,"duration_ms":6606,"temperature":1.0,"reasoning_tokens":596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:13.628330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the generated, dry-run-verified GAAN pipeline and run it on a graph dataset whose target attribute contains continuous values; if it crashes or produces invalid loss values, the paper's claimed validation reliability overstates what the Reviewer detects. Equivalently, measure the pipeline success rate on a held-out set of realistic user-uploaded datasets rather than the benchmark pairs in Table 1.","supporting_citations":[],"review_version":1}