{"id":"235a29d8-cf30-47c8-8e75-a46bf0ab53a7","arxiv_id":"2510.02060","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 20-dataset benchmark with textual metadata reports that zero-shot LLMs detect tabular anomalies better when given semantic context, but label descriptions embedded in the metadata may explain much of the gain.","lead":"This paper introduces ReTabAD, a benchmark of 20 tabular datasets paired with textual descriptions of columns, domains, and target labels, and a zero-shot LLM pipeline that uses this metadata to score anomalies. The authors report that adding such context improves average AUROC by 7.6 points and approaches trained detectors, but the prompts also include the exact anomaly definition, which likely inflates the gain.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported zero-shot gains conflate semantic context with the ground-truth anomaly rule: Label-Level Descriptions in the prompt tell the LLM which class is anomalous.","rationale":"The paper's central empirical claim is that semantic metadata improves zero-shot LLM AD and approaches trained SOTA. For that to hold, the performance gain should come from feature/domain semantics rather than from revealing the target definition. I checked the prompt construction and metadata schema. §3.1 and Figure 8 show that the 'Full Desc.' prompt includes a Target Label section that explicitly states which class is normal and which is anomalous, and the guidelines instruct the LLM to 'Leverage target label definition.' This is not an incidental detail: it is a designed component of C_domain. Consequently, the +7.6 AUROC improvement in Table 4 and the competitive results in Table 3 are confounded. The benchmark's anomaly definitions amplify the problem: several are arbitrary class-as-anomaly choices (vertebral, census, wine, glass, yeast, quasar), so the label description effectively provides the classification rule. I would not call this fraud or trivial leakage of per-row labels; it is a methodological confound in the zero-shot evaluation. The resource itself (datasets, metadata format, algorithm implementations) has standalone value and the paper is transparent about its metadata schema, which is why a conditional accept with a required re-run is appropriate. The concrete test—ablating the label description—cleanly separates rule leakage from genuine semantic context.","tokens_in":35009,"tokens_out":7340,"duration_ms":65616,"concrete_test":"Re-run the zero-shot LLM evaluation (all five models; Type C and Type D) after removing only the 'Target Label' / label-description component from the prompt, keeping dataset description, feature descriptions, and normal statistics. Compare per-dataset AUROC on the 20 datasets, with particular attention to census, quasar, vertebral, glass, wine, and yeast. If average AUROC drops by more than ~3 points, or if the zero-shot LLM no longer matches MCM/NeuTraL on those datasets, the reported semantic-context gains are largely rule leakage; if performance is unchanged, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"To support the claim that semantic context improves detection, the prompt must not contain the ground-truth rule. §3.1 defines Label-Level Descriptions as specifying 'which classes are considered normal and which are treated as anomalies,' and the Type D prompt (Figure 8) includes 'Target Label: {target_description}' with the instruction 'Leverage target label definition.' Thus the zero-shot LLM is told the anomaly class before scoring—e.g., 'Quasars (QSO) are considered anomalies,' 'High-income individuals (>$50K) are anomalies,' and in vertebral the original Normal class is the anomaly. This is rule-level supervision, not context-aware anomaly detection, and it makes the comparison with training-based unsupervised methods inequitable: those methods never see the target definition. Table 4's +7.6 AUROC (Type A vs. Type D) and Tables 3/5 therefore measure the LLM's ability to apply a told rule, not the contribution of feature/domain semantics. The problem is sharpest on class-imbalance datasets (census, glass, wine, yeast, quasar, vertebral) where 'anomaly' is a semantic class label; several definitions are arbitrary inversions (vertebral) or economic rarity (census), undercutting the paper's 'faithful anomaly definitions' claim. A missing control—Full Desc. with vs. without label description—is needed before the central empirical claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ReTabAD introduces a benchmark of 20 tabular datasets enriched with structured textual metadata (dataset-level, column-level, and label-level descriptions), together with implementations of classical, deep, and LLM-based detectors and a zero-shot LLM baseline. The paper reports that adding semantic context to LLM prompts improves average AUROC by +7.6 points, that the zero-shot LLM achieves performance comparable to state-of-the-art training-based methods, and that metadata improves feature-attribution alignment and reasoning quality. The benchmark and code are publicly released.","tokens_in":35359,"tokens_out":5958,"duration_ms":74081,"significance":"If the central empirical claim were cleanly established, ReTabAD would be a useful community resource: it ships a metadata schema, baseline implementations, and a reproducible evaluation pipeline, and the idea of using feature/domain semantics in tabular AD is timely. However, the headline result is currently confounded because the 'Full Desc.' prompt contains the ground-truth anomaly rule through Label-Level Descriptions, so the zero-shot LLM is effectively told which class is anomalous. This moves the comparison from unsupervised context-aware AD toward rule application, and the interpretability analysis has an additional name-matching confound. With a proper control and appropriately narrowed claims, the benchmark could still be valuable; in its current form the main empirical claims are not yet supported.","major_comments":[{"comment":"The central empirical claim is confounded by label leakage. §3.1 defines Label-Level Descriptions as specifying 'which classes are considered normal and which are treated as anomalies,' and the Type D prompt (Figure 8) includes 'Target Label: {target_description}' plus the instruction 'Leverage target label definition.' Thus the +7.6 AUROC gain (Table 4) and Type D win rates (Table 5) may measure the LLM's ability to apply a told rule, not the contribution of feature/domain semantics. Training-based unsupervised baselines in Table 3 never see this rule, making the comparison inequitable. Please add a control that compares Full Description with versus without the label description / target instruction, and report whether the gains persist without the label rule.","section":"§3.1, Figure 8, Tables 4–5"},{"comment":"The reasoning-alignment comparison is invalid as presented. Type A anonymizes column names (AA, AB, ...) while the SHAP reference set uses original feature names, and the paper does not describe any mapping between them. The near-zero F1@1/F1@3 values for Type A are therefore largely an artifact of name mismatch rather than missing semantics. Please either map predicted anonymized names back to original columns or evaluate Type A against anonymized references. Additionally, since Type D includes the target label definition, the reported F1 gains may again reflect rule following rather than improved semantic reasoning.","section":"§4.3, Appendix G, Table 6"},{"comment":"The claim of 'faithful anomaly definitions' is undercut by several curation choices. Vertebral treats the original Normal class as anomalous and pathological classes as normal; census defines high income (>$50K) as anomalous; wine/glass select rare or less common classes. These are class-imbalance/rarity selections rather than domain-faithful anomalies. The criteria in §3.1 — discarding 'too easy' datasets and capping anomaly ratio at 1/3 — further distort representativeness. Please relabel these as 'class-defined evaluation labels' and provide robustness analyses, such as label-inversion sensitivity or anomaly-ratio sensitivity, to show that the benchmark's conclusions are not artifacts of these choices.","section":"§3.1, Table 8, Appendix A.1"}],"minor_comments":[{"comment":"The manuscript inconsistently states 16 or 17 algorithms. Clarify whether the zero-shot LLM baseline is counted among the implemented algorithms or presented separately.","section":"Abstract, §3.2, Table 1"},{"comment":"'No Desc.' is a misnomer: Type A still provides statistical context (normal ranges/values). Rename it 'Statistical only' or 'No Metadata' to avoid implying that no textual information is used.","section":"Table 4"},{"comment":"The prompt structure explicitly shows 'Label 0 means Normal, 1 means Suspect/Pathologic.' If this label mapping is intentionally part of the semantic context, it should be disclosed and discussed as a source of supervision; if not, it should be removed from the baseline prompt.","section":"Figure 2"},{"comment":"No statistical significance tests or confidence intervals are reported for average AUROC differences or win rates, despite multiple seeds and multiple LLMs. Add paired tests or bootstrap intervals for the headline +7.6 comparison.","section":"Tables 4–5, Appendix E"},{"comment":"Type C includes the target label description, so the 'domain knowledge alone' condition is not clean. For the ablation to isolate domain knowledge, a variant without the label/target instruction is required; otherwise the Type A→D decomposition is uninterpretable.","section":"§4.2, Table 5"},{"comment":"Minor typos: 'Forrestoring' in the title, and '5th-9th percentile' in prompt templates appears to be a typo for '5th-95th percentile.' Also check consistency of hyphenation and spacing in 'Normal Ranges'.","section":"Title, Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact is potentially salvageable and useful, and the authors' release of code and metadata is a genuine strength. However, the main empirical claim is not currently supported because the prompt leaks the ground-truth anomaly rule via Label-Level Descriptions, and the F1 attribution analysis has a separate name-matching confound. I would require the control experiment (full metadata with vs. without label/target instruction) and a re-framed contribution; if the gains vanish without the label rule, the paper's central claim should be withdrawn or substantially narrowed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, ReTabAD is a real resource: twenty tabular datasets with structured JSON metadata, source links, preprocessing scripts, and full appendix tables for sixteen baselines plus five LLMs under four prompt conditions. Anyone doing LLM-based tabular AD can now test whether textual metadata helps, and the documentation work is done properly. Second, the headline result — semantic context improves zero-shot detection by +7.6 AUROC and approaches trained SOTA — is overstated, because the 'domain knowledge' condition includes the Label-Level Description, which names the anomalous class outright.\n\nThe leakage is not subtle. In the paper's own words (§3.1), label descriptions specify 'which classes are considered normal and which are treated as anomalies.' The Type D prompt template (Figure 8) contains 'Target Label: {target_description}' and instructs the model to 'Leverage target label definition.' So the main ablation contrasts Type A (anonymized columns, statistics only) with Type D (statistics plus feature descriptions plus dataset background plus the explicit rule that this class is the anomaly). The trained baselines never get that rule. The +7.6 points, and the parity with MCM, therefore largely measure how well the LLM applies a told rule. To be fair: providing the label definition is defensible as a benchmark design choice, since practitioners usually know what they are hunting for. But then the claim needs reframing — this is not 'semantic context improves detection,' it is 'telling the model the target rule helps,' and the comparison should be against methods that also receive the target definition. The missing experiment is straightforward: full metadata with and without the label description. The paper does not run it, and it is the one control that would settle the question.\n\nSecondary issues, in proportion. Some anomaly definitions are arbitrary rather than 'faithful': vertebral flips the original classes so the normal class is the anomaly, and census treats high income as anomalous. That weakens the curation claim, though the metadata and source links make the choices transparent and debatable. The F1@K reasoning-alignment metric uses SHAP from a label-trained XGBoost as ground truth; a reasonable proxy, and the paper says as much. The appendix is also honest about where alignment fails (quasar, covertype) and where sample sizes are too small (automobile). The Gemini Type C score (0.673, below Type A's 0.691) is a useful data point: domain text without statistics does not help, which the paper discusses.\n\nWho this is for: anyone building or evaluating LLM-based tabular AD, and anyone who cares about benchmark hygiene in LLM evaluation. It deserves a serious referee — the flaw is fixable, the resource is valuable, the writing is transparent. I would send it to review with the label-leakage control required, and treat the headline number as unverified in the meantime.","headline":"Real benchmark resource, but the headline gain conflates semantic context with telling the LLM which class is anomalous — the label-leakage control is missing.","tokens_in":35827,"tokens_out":8542,"would_cite":true,"duration_ms":72222,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that restoring textual metadata to tabular anomaly detection lets a zero-shot LLM reach accuracy comparable to trained state-of-the-art detectors, without any task-specific training.","keywords":["tabular anomaly detection","semantic context","textual metadata","zero-shot LLM","benchmark","one-class classification","interpretability","feature attribution"],"falsifier":"Run the zero-shot LLM in the full-description prompt with the label definition removed—keeping only dataset background, column semantics, and normal statistics—and compare AUROC. If removing the label definition on datasets like census or vertebral erases most of the reported gain, the result is driven by the told rule rather than by context-aware reasoning. A complementary test is to create a synthetic tabular dataset whose anomaly class is a rare pattern the metadata does not mention; the framework would predict only a small gain.","tokens_in":34909,"feed_emoji":"📊","tokens_out":4206,"duration_ms":80962,"temperature":0.7,"pith_summary":"Tabular anomaly detection benchmarks have traditionally stripped away the text that tells experts what a column means, what is normal, and what counts as an anomaly. This paper builds a 20-dataset benchmark that restores that textual metadata, pairing each dataset with structured JSON descriptions covering dataset background, per-column semantics, and label definitions. It then shows that a zero-shot LLM prompted with this metadata achieves an average AUROC of 0.847, matching or beating training-based detectors, while adding metadata raises AUROC by 7.6 points on average across five LLMs. The authors argue that semantic context improves both detection and interpretability, since the model can name and explain the features that drive a record's anomaly.","feed_headline":"Zero-shot LLMs match trained anomaly detectors with text metadata","feed_subtitle":"Adding dataset, column, and label descriptions gives +7.6 AUROC and pinpoints why each record is anomalous.","key_machinery":"The carrying object is the ReTabAD metadata file: a structured JSON per dataset containing a dataset-level description, per-column descriptions with logical types and units, and a label-level description stating which classes are normal and which are anomalous. This metadata is injected into a prompt template alongside per-column normal ranges (5th-95th percentiles) and analysis guidelines, and the LLM outputs anomaly scores, key features, and reasoning. The label-level description is the piece that lets the model map raw values directly onto anomaly semantics, and the paper also preserves raw numerical scales and restores categorical features to their original text values rather than arbitr","core_discovery":"The central discovery is that the definition of an anomaly is often carried by text, not just numbers, and a language model can use that text directly. The paper's zero-shot framework serializes each row with feature names and supplies three kinds of context—domain knowledge, feature descriptions, and normal statistics derived from the training data—then asks the LLM for an anomaly score, key features, and reasoning. With full metadata, the best-performing evaluator LLM reaches an average AUROC of 0.847 and an average rank of 4.08 among the 17 compared models, while the same model without descriptions scores 0.691. Ablations show that combining all three context types gives the highest win r","pith_inferences":["The label-level description in the prompt states the anomaly rule outright (e.g., high income is anomalous, or the original 'normal' class is anomalous). A fair reading is that the +7.6 AUROC gain measures how well LLMs apply a told rule; whether they can discover an anomaly rule from feature semantics alone is a testable open question.","If metadata becomes the bottleneck, automatic generation of column descriptions or retrieval from documentation could extend this style of context-aware detection to larger, uncurated datasets.","The framework points toward hybrid systems where statistical detectors flag candidates and LLMs re-rank or explain them using semantic context, rather than replacing trained detectors entirely.","The reasoning-alignment metric uses supervised attributions as ground truth; because those attributions come from a model trained on the same labels, high alignment could partly reflect agreement with the label rule rather than independent domain reasoning."],"forward_implications":["Semantic context can be treated as a first-class input in tabular anomaly detection, opening a benchmark standard for context-aware evaluation rather than purely numeric comparison.","A zero-shot LLM baseline becomes a strong reference point without task-specific training, lowering the cost of obtaining competitive detection baselines on new datasets.","Feature descriptions deliver large gains on categorical-heavy domains where raw integer codes are meaningless, so restoring textual categorical values can unlock performance that numeric-only pipelines miss.","Reasoning quality can be measured, and high-quality reasoning texts can be re-injected as few-shot examples to improve detection, giving an evaluation pathway for interpretability.","Anomaly definitions in benchmarks should be verified against original source documentation; several datasets in this benchmark differ from earlier processed versions after such verification."],"fun_headline_variants":["Zero-shot LLM uses text to top tabular anomaly detection","Text metadata gives LLM 7.6 AUROC boost in anomaly detection","Semantic context turns LLM into top anomaly detector","LLM explains anomalies with text, not just numbers"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the label-level descriptions in the metadata—which explicitly say which classes are normal and which are anomalous—are legitimate semantic context rather than leakage of the ground-truth anomaly rule; if that premise fails, the measured gains reflect rule-following, not context-aware detection.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM uses text to top tabular anomaly detection","Text metadata gives LLM 7.6 AUROC boost in anomaly detection","Semantic context turns LLM into top anomaly detector","LLM explains anomalies with text, not just numbers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000598,"raw_usage":{"total_tokens":2623,"prompt_tokens":727,"completion_tokens":1896,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1826}},"tokens_in":471,"tokens_out":1896,"duration_ms":10813,"temperature":1.0,"reasoning_tokens":1826,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T12:44:32.455863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the zero-shot LLM in the full-description prompt with the label definition removed—keeping only dataset background, column semantics, and normal statistics—and compare AUROC. If removing the label definition on datasets like census or vertebral erases most of the reported gain, the result is driven by the told rule rather than by context-aware reasoning. A complementary test is to create a synthetic tabular dataset whose anomaly class is a rare pattern the metadata does not mention; the framework would predict only a small gain.","supporting_citations":[],"review_version":1}