{"id":"3904252f-5cbe-439b-ab24-653aea88dc83","arxiv_id":"2412.12154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PyOD 2 is a library update that unifies deep outlier detectors on PyTorch and uses GPT-4o to automatically select a model, reporting the best mean AUROC rank on 17 ADBench datasets.","lead":"PyOD 2 adds 12 deep learning outlier detectors under a unified PyTorch framework and introduces an automated model selector that uses a large language model to choose an algorithm from dataset statistics. The paper reports the selector achieves the best average AUROC ranking on 17 benchmark datasets, but the result is presented without error bars or significance tests.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.24-point mean-rank advantage of AutoModelSelector over AutoEncoder is reported without per-dataset variance or significance testing, so the headline performance claim is not statistically established.","rationale":"The reader and I converge on the statistical fragility of the central empirical claim. I do not see a deeper internal inconsistency: the equations are illustrative but not wrong, and the library contribution is plausible. The main issue is evidential: a 0.24 mean-rank gap over 17 datasets is within the noise floor for ranks, and without per-dataset results or a paired test the claim of best mean AUROC ranking cannot be distinguished from random fluctuation. I also note that the Average Performance baseline is presented as simulating random user choices; while the average AUROC is a reasonable proxy for the expected AUROC of a uniformly random model, it is not the expected rank, so the ranking comparison is partly uninterpretable as a random-choice baseline. Finally, the LLM-based selection is not reproducible because prompts and parameters are not given. These concerns do not overturn the paper's value as a software and demo contribution, so the reader's CONDITIONAL verdict is unchanged, with conditions: release per-dataset results, significance tests, and prompt and parameter details.","tokens_in":6775,"tokens_out":8460,"duration_ms":91256,"concrete_test":"Run the released PyOD 2 code on the 17 ADBench datasets and save, for each dataset, the AUROC of AutoModelSelector without notes and AutoEncoder, along with the rank used in Figure 2. Then perform a paired permutation test (or Wilcoxon signed-rank test) on the 17 paired rank differences and compute a bootstrap 95% confidence interval for the mean-rank difference. Also log the exact prompts and LLM responses for the without-notes runs and confirm whether dataset names or ADBench results appear in the prompt. If the 95% CI includes zero or the prompt contains dataset identifiers, the claimed superiority is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central result is a 0.2353-point mean-rank advantage for AutoModelSelector over AutoEncoder (2.4118 vs 2.6471) computed from 17 datasets, yet the paper reports only the aggregate rank means in Figure 2 and omits per-dataset AUROCs, variance, and any significance test. With 17 paired observations on a 1-5 rank scale, a shift of this size can be produced by one or two datasets flipping, so the headline claim of robust performance is not established. The evaluation also does not validate the LLM-generated symbolic tags (Eqs. 1-3) or disclose the scoring functions in Eqs. 4-5, the threshold delta, the exact prompt, or the sampling parameters of GPT-4o. Because the pipeline is a black box, the reported 2.4118 cannot be independently reproduced or attributed to the described symbolic-neural reasoning rather than to GPT-4o's prior knowledge of ADBench datasets. The load-bearing assumption is that the aggregate mean AUROC ranking is a stable, meaningful measure of selection quality; that assumption is currently unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PyOD 2 is presented as a major update of the PyOD outlier-detection library: it refactors deep models into a unified PyTorch framework, adds automated model selection driven by GPT-4o, and claims best mean AUROC ranking (2.4118) over 17 ADBench datasets compared with the average of all models (4.0588), LUNAR (3.4118), and AutoEncoder (2.6471). The selection pipeline encodes model strengths/weaknesses and dataset statistics as symbolic tags (Eqs. 1-3), computes a score via sim/penalty (Eq. 4), thresholds candidates (Eq. 5), and asks an LLM to choose among them (Eq. 6). The paper includes two usage demonstrations and reports only aggregate rank means.","tokens_in":6986,"tokens_out":4357,"duration_ms":43214,"significance":"Integrating 10 deep OD models into one PyTorch codebase and providing a documented base_dl class is practically valuable, and the library has a large existing user base. However, the novel scientific claim—that LLM-based symbolic-neural model selection outperforms baselines—rests on a small aggregate mean-rank difference with no variance or significance measures; the pipeline's core functions are unspecified; and the model-count claims are inconsistent. The contribution is best judged as a systems/demo paper whose effectiveness claims need further evidence.","major_comments":[{"comment":"The headline result, AutoModelSelector without notes achieving mean AUROC ranking 2.4118 versus AutoEncoder's 2.6471, is an aggregate of 17 paired observations on a 1-to-5 rank scale. The paper reports no per-dataset AUROC ranks, no error bars, no confidence intervals, and no paired test (e.g., Wilcoxon signed-rank). With 17 datasets, a 0.2353 mean-rank shift can be driven by one or two datasets flipping, so the claim that the selector 'delivers robust performance' is not statistically established. Please provide the per-dataset table, distribution of rank differences, and a paired significance test or bootstrap interval, and interpret the result accordingly.","section":"§4, Figure 2"},{"comment":"The core scoring functions sim(·) and penalty(·), the threshold δ, and the GPT-4o configuration (snapshot version, temperature, prompt templates) are not specified. Because GPT-4o both generates the model/dataset tags and performs the final selection, the 'symbolic reasoning' component cannot be separately validated or attributed. Please disclose these details and add an ablation that isolates the symbolic matching step (e.g., a deterministic tag-matching variant) and a validation of the generated tags against human annotations.","section":"§2.2, Eqs. (3)-(5)"},{"comment":"The experimental protocol is incomplete: the 17-dataset ADBench subset is listed, but the paper does not state the train/test split, contamination ratio, model hyperparameters, or how AUROC is computed for each dataset. Without these details the evaluation is not reproducible. Please specify the protocol or provide a config file in the repository.","section":"§4, Datasets and Models"},{"comment":"The abstract and §1 claim 12 state-of-the-art deep learning models are integrated, while Table 1 lists exactly 10 models (AE, VAE, AnoGAN, DeepSVDD, ALAD, MO-GAAL, SO-GAAL, AE1SVM, DevNet, LUNAR), and §4 evaluates 10 models. This is a factual inconsistency in the central coverage claim. Please reconcile the count, either by adding the missing two models to the table/evaluation or by correcting the abstract and intro.","section":"Abstract/§1 vs Table 1"}],"minor_comments":[{"comment":"The code listing contains spurious spacing in 'decis ion_sco res_' and 'de ci si on _f unc ti on', which would not compile; ensure code listings are reproduced without line-breaking artifacts.","section":"§3, Case 1 code snippet"},{"comment":"The caption should state n=17, the rank scale (1-5), and that lower ranks are better, so the reader can interpret the figure without hunting through the text.","section":"§4, Figure 2 caption"},{"comment":"The sentence 'domain insights can be helpful' is not supported by the reported means, since the without-notes variant has a better ranking (2.4118) than the with-notes variant (2.4706); please rephrase to avoid implying a benefit the data do not show.","section":"§4, first paragraph after Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like a systems/demo submission; the strongest concern is that the only evidence for the LLM selector is an aggregate mean rank without variance or significance testing. I also note the model-count inconsistency. With the requested per-dataset statistical analysis and pipeline disclosure, the contribution could be acceptable for the venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real news in this paper is the library, not the headline result. PyOD 2 is a genuine improvement to a widely used tool, and the LLM-driven selector is a sensible idea. But the empirical claim that the selector outperforms standard baselines is not backed by the reported statistics.\n\nWhat's new: a unified PyTorch framework for 12 deep outlier detection models, a clean base_dl API, and an automated selection pipeline that turns model papers and code into symbolic tags, profiles the dataset, and uses GPT-4o to match them. That's a reasonable application of LLMs to model selection, and the with/without notes comparison is a nice touch. The library engineering is real, and the paper explains the workflow clearly.\n\nWhere it is soft: the evaluation is too thin to support the central conclusion. The whole case rests on mean AUROC ranks across 17 datasets, with no per-dataset table, no variance, no significance test. The margin over AutoEncoder is 0.24 rank points; with 17 paired observations on a 1-5 scale, that's within noise. The symbolic tags are generated by GPT-4o with no validation, and the sim(·) and penalty(·) functions, the threshold δ, and the exact prompt and sampling parameters are undisclosed. So the reported ranking cannot be independently reproduced or credited to the described symbolic-neural reasoning. Also missing is any comparison to AutoML or meta-learning baselines, which would be the natural controls.\n\nThis is a system/demo paper, not a rigorous empirical study. It will be useful to practitioners who want a maintained library with deep models and an automated selector. For a research claim, the evaluation needs revision: per-dataset results, error bars, statistical tests, and reproducible details. The 'first' claim also needs tempering absent those comparisons.\n\nBottom line: deserves a serious referee if the venue values systems and code contributions, with the expectation of heavy empirical revision. I'd send it to review but push hard on the missing statistics and reproducibility.","headline":"PyOD 2 is a solid engineering contribution whose headline performance claim lacks statistical support.","tokens_in":7549,"tokens_out":2135,"would_cite":false,"duration_ms":21321,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PyOD 2 adds a dozen deep models and an LLM that picks the best outlier detector for your data.","keywords":["outlier detection","anomaly detection","deep learning","large language models","automated model selection","PyTorch","machine learning library","AutoML"],"falsifier":"Recompute the per-dataset AUROC for AutoModelSelector without notes and for AutoEncoder, then run a paired Wilcoxon signed-rank test on the ranks; if the p-value is above 0.05, the claimed edge is within noise and the central claim is unsupported.","tokens_in":6576,"feed_emoji":"🤖","tokens_out":5167,"duration_ms":44796,"temperature":0.7,"pith_summary":"The paper presents PyOD 2, a refresh of the widely used Python outlier-detection library. It folds 12 modern deep models into a single PyTorch framework, lifting the algorithm count to 45, and adds a three-step pipeline that uses a large language model to recommend a detector for a given dataset. The paper's central demonstration is that this automated selector, running without any user-provided notes, achieves the best mean AUROC ranking of 2.4118 across 17 datasets, ahead of fixed baselines including an autoencoder (2.6471) and LUNAR (3.4118). The implied payoff is that non-experts can get competitive outlier-detection results without manually trying dozens of models.","feed_headline":"LLM-driven tool picks outlier detector better than fixed models","feed_subtitle":"PyOD 2 unifies 12 deep models and adds an LLM that chooses the right detector automatically.","key_machinery":"The load-bearing mechanism is the three-step AutoModelSelector pipeline. Step 1 extracts model metadata as a set of strengths and weaknesses; Step 2 turns dataset statistics into tags such as 'imbalanced data' and 'noisy features'; Step 3 scores each candidate with $S(m_i, T_D) = \\mathrm{sim}(\\mathrm{strengths}(m_i), T_D) - \\mathrm{penalty}(\\mathrm{weaknesses}(m_i), T_D)$, then lets the LLM refine among candidates above a threshold $\\delta$. The similarity-minus-penalty identity is what connects dataset tags to model strengths and carries the selection logic.","core_discovery":"The central claim is that a pipeline which first converts each model's paper and code into symbolic strength and weakness tags, then profiles a dataset into symbolic tags, and finally scores model-dataset compatibility with an explicit similarity-minus-penalty formula refined by LLM reasoning, can outperform standard fixed model choices on outlier detection. On 17 datasets, the AutoModelSelector without additional notes attains a mean AUROC ranking of 2.4118, slightly edging out the version with notes (2.4706) and clearly beating AutoEncoder (2.6471), LUNAR (3.4118), and average performance across all models (4.0588). The paper treats this ranking as evidence that data-driven, LLM-guided selection is more effective than naive strategies.","pith_inferences":["The same symbolic-tag-plus-LLM pattern could be tested for other AutoML tasks, such as classifier or clustering selection, but the paper does not claim that extension.","The reported margin over AutoEncoder (2.4118 vs 2.6471) is small; running a paired significance test across the 17 datasets would indicate whether the difference is real or noise.","The accuracy of the LLM-generated tags is unmeasured; ablating them, for instance by randomizing tags, would reveal how much of the selector's performance depends on tag quality."],"forward_implications":["Users can train and evaluate a deep outlier detector like LUNAR in five lines of code.","The AutoModelSelector works without user notes, lowering the barrier for non-experts.","Adding new models only requires tagging their strengths and weaknesses, after which the selector can include them.","The pipeline returns an explanation alongside the recommended model, making the choice inspectable.","All deep models share a unified PyTorch base class, simplifying maintenance and future additions."],"supporting_citations":[{"why":"Supplies the 17 benchmark datasets used in the evaluation of model selection.","marker":"[5]"},{"why":"The original PyOD library that PyOD 2 extends and whose API it preserves.","marker":"[22]"},{"why":"The AutoEncoder baseline that the AutoModelSelector is compared against and whose ranking it must beat.","marker":"[1]"},{"why":"The LUNAR baseline and one of the integrated deep models.","marker":"[4]"},{"why":"PyTorch, the unified framework all deep models in PyOD 2 are refactored onto.","marker":"[16]"}],"fun_headline_variants":["PyOD 2's LLM picks outlier detectors that beat fixed models","LLM-guided detector selection in PyOD 2 outperforms fixed baselines","PyOD 2 adds LLM to auto-select outlier detection models","LLM chooses outlier detector in PyOD 2 better than fixed choices","PyOD 2's LLM outperforms fixed models on outlier detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the automated selector beats an autoencoder rests on a mean ranking difference of about 0.24 across 17 datasets, reported without error bars, per-dataset variance, or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["PyOD 2's LLM picks outlier detectors that beat fixed models","LLM-guided detector selection in PyOD 2 outperforms fixed baselines","PyOD 2 adds LLM to auto-select outlier detection models","LLM chooses outlier detector in PyOD 2 better than fixed choices","PyOD 2's LLM outperforms fixed models on outlier detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3661,"prompt_tokens":979,"completion_tokens":2682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2596}},"tokens_in":595,"tokens_out":2682,"duration_ms":20372,"temperature":1.0,"reasoning_tokens":2596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:07:44.698139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the per-dataset AUROC for AutoModelSelector without notes and for AutoEncoder, then run a paired Wilcoxon signed-rank test on the ranks; if the p-value is above 0.05, the claimed edge is within noise and the central claim is unsupported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 17 benchmark datasets used in the evaluation of model selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The AutoEncoder baseline that the AutoModelSelector is compared against and whose ranking it must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LUNAR baseline and one of the integrated deep models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PyTorch, the unified framework all deep models in PyOD 2 are refactored onto."}],"review_version":1}