{"id":"ba8387db-a9a9-4eaf-8d5b-ef05ee46e491","arxiv_id":"2504.12250","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A four-phase pipeline combines static call-graph analysis with LLM chain-of-thought reasoning to synthesize log sequences with anomaly labels for Hadoop and HDFS, expanding event coverage and modestly improving downstream anomaly detection F1.","lead":"AnomalyGen is a pipeline that reads Java source code and uses a large language model to invent realistic log files for training anomaly detectors. It claims to cover far more log event types than existing public datasets and to slightly improve three detection models when its synthetic logs are added to training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The coverage claim rests on a self-defined, version-mismatched denominator; the 38-95x improvement may reflect source-version growth rather than genuine event coverage.","rationale":"Coverage is the headline claim in the abstract and the focus of RQ1. The formula in Section 5.1 is only meaningful if Total_Log_Event is an anchored, version-appropriate ground truth. The paper never specifies how the 9,662 and 2,889 totals were computed; Sections 3.2-3.3 describe a static-analysis pipeline that prunes a call graph to log-related nodes and then enriches CFGs with LLM inference, so the denominator is produced by the same framework that later generates the 'covered' sequences. This creates a risk of tautology: if the enumeration includes control-flow-expanded variants of the same logger call site, the denominator can be thousands even though the distinct log event vocabulary an anomaly detector actually sees is only hundreds. The additional version mismatch, R-Hadoop/R-HDFS were collected years before Hadoop 3.3.6 and from different workloads, means that events absent from the baselines are not necessarily missing coverage; they may simply not exist in the baseline version. The partial validation against LogHub event types (93/107 and 14/15) is useful, but it checks only the generated set's recall of existing event types, not the denominator's validity. If the recomputation test shows the denominator is inflated, the 38-95x claims, the 97.48% coverage, and the motivation that existing datasets are incomplete all weaken substantially, though the F1 augmentation result would remain a separate, weaker claim. Because the concern is concrete and checkable, a conditional verdict, with the independent enumeration as a condition, is appropriate.","tokens_in":15375,"tokens_out":8868,"duration_ms":94147,"concrete_test":"Independently enumerate the log-event vocabulary of the exact source versions AnomalyGen analyzed (Hadoop 3.3.6 and the HDFS version used): scan all Java source for logger call sites (Logger.info/warn/error/debug/trace) and count distinct static message templates at file:line granularity, without expanding control-flow paths into separate events. Then recompute Table 2's coverage using that denominator, and, for the baseline rows, use a denominator matched to the source version from which R-Hadoop/R-HDFS and LHub were collected. If the independent denominator is materially below 9,662/2,889, or if version-matched coverage ratios no longer show 38-95x gaps, the coverage claim should be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's primary quantitative claim--38x-95x more log event coverage, Table 2--depends on the denominator Total_Log_Event in the coverage metric defined in Section 5.1. The denominator (9,662 for Hadoop, 2,889 for HDFS) is not independently defined or externally validated; it is obtained from AnomalyGen's own static analysis of Hadoop 3.3.6, while the R-Hadoop/R-HDFS and LogHub baselines were collected from different, older source versions and workloads. The paper itself acknowledges version differences in Section 5.2. If 'total log events' counts log-statement sites reachable via control-flow expansion, many of which are in code paths no workload in the baselines exercises, the denominator is inflated relative to the observable event vocabulary, making the 2.50% and 1.04% coverage of existing datasets artificially low. The 38-95x ratio would then partly measure source-code version growth and enumeration choices rather than a genuine deficiency of existing datasets or a capability of AnomalyGen. The reported R-coverage validation (93/107 and 14/15 against LogHub events) checks recall of existing LogHub event types, but it does not validate the denominator or the version-matched comparison. Without an independent, version-matched enumeration of log event types, the headline coverage contribution is unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AnomalyGen, a four-phase automated log synthesis framework that combines static call-graph analysis with LLM chain-of-thought reasoning to generate log sequences and anomaly labels for Hadoop and HDFS without executing the system. The claimed contributions are (i) a 38x-95x increase in log event coverage over existing datasets, (ii) more realistic log sequences than static-analysis-only tools, (iii) retention of execution-context semantics, and (iv) F1-score improvements of up to 3.7 percentage points (average 1.8) when augmenting LSTM, Transformer, and CNN anomaly detectors. The evaluation is based on two systems and three detectors, with the quantitative claims appearing in Tables 2 and 3.","tokens_in":15620,"tokens_out":3840,"duration_ms":35783,"significance":"The problem is relevant and timely: public log datasets are scarce, and LLM-assisted generation is an active research direction. If the coverage and realism claims were rigorously supported, AnomalyGen would be a useful contribution, and the authors' decision to release artifacts is commendable. However, the headline coverage claim depends on a self-defined denominator, and the detection gain rests on what appears to be a single unseeded run; these issues must be addressed before the quantitative claims can be accepted. The paper combines established program-analysis techniques with LLM reasoning in a plausible architecture, but the current evidence does not yet substantiate the magnitude of the reported improvements.","major_comments":[{"comment":"The coverage metric uses Total_Log_Event = 9,662 (Hadoop) and 2,889 (HDFS), obtained from AnomalyGen's own static analysis of Hadoop 3.3.6, while the comparison datasets (R-Hadoop, R-HDFS, LHub-Hadoop, LHub-HDFS) originate from different source versions and workloads; the paper itself admits version-related filtering in Section 5.2. Because the denominator is self-constructed and version-mismatched, the reported 38x-95x increment may reflect enumeration choices rather than a genuine deficiency of existing datasets, and the R-coverage validation against LogHub events checks recall on the small observed event set but does not validate the denominator. Please provide an independent, version-matched enumeration of the expected log event vocabulary, or explicitly restrict all coverage claims to the ratio between generated and existing event counts and rephrase the headline accordingly.","section":"Section 5.1 / Table 2"},{"comment":"The anomaly-detection gain is reported from what appears to be a single run on HDFS with no error bars, no random seeds, and no statistical significance tests; the observed improvements (average 1.8 percentage points, maximum 3.7 points) are small relative to typical run-to-run variance of LSTM, Transformer, and CNN models on this benchmark. The claim that AnomalyGen benefits anomaly detection requires multiple independent runs with statistical testing, and the authors should either report results on the Hadoop dataset as well or justify the single-system scope.","section":"Table 3 / Section 5.5"},{"comment":"The realism claim is supported primarily by qualitative code examples (Figures 8-10) and an informal statement in Section 5.4 that practitioners \"save more than 5 times\" the time, with no measurement procedure, sample size, or quantitative metric described. Without an operational definition of realism (e.g., a controlled user study, comparison of generated parameter distributions against runtime traces, or downstream performance on injected real anomalies), the claim that AnomalyGen produces \"more realistic log sequences\" than AutoLog and other static-analysis tools is not substantiated.","section":"Section 5.3 / RQ2"},{"comment":"The paper acknowledges LLM reasoning uncertainty and randomness in Section 6 but supplies no evaluation of the CoT verification's false-acceptance or false-rejection rates, nor of the fraction of generated paths that pass manual expert review. Because the LLM is a load-bearing component for path completion, parameter simulation, and anomaly annotation, the framework needs at least a reliability or error analysis on a sample of generated sequences so that the impact of LLM mistakes on the final dataset can be assessed.","section":"Section 6 / Phase III"}],"minor_comments":[{"comment":"The header contains a typo: \"Wtihout/With AG\" should be \"Without/With AG\".","section":"Table 3"},{"comment":"The reported coverage increments are inconsistent across the text: the abstract and intro say \"38-95x\", Section 5.2 says \"37 to 94 times\" and \"95.3 times\", and Table 2 lists 38X, 86X, 95X, and 191X; these should be aligned.","section":"Section 5.2 / Table 2"},{"comment":"There is duplicated text in Section 3.1 (the overview paragraph appears twice) and similar repetition in the introduction; the writing needs careful editing for conciseness.","section":"Section 2.3 / Section 3.1"},{"comment":"The citation to Busse et al. appears as \"Busse et al.[? ]\" and should be completed with the proper reference key.","section":"Section 1"},{"comment":"The paper uses both \"D-Hadoop\"/\"D-HDFS\" and \"R-Hadoop\"/\"R-HDFS\" for the same benchmark datasets; unify the naming to avoid confusion.","section":"Table 1 and throughout"},{"comment":"The statement that \"All artifacts and datasets have been released\" is not accompanied by a URL or repository identifier; please provide this information.","section":"Section 1 / Section 8"},{"comment":"The caption contains a typo: \"Four Satges\" should be \"Four Stages\".","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The central claims are plausible and the topic fits the journal, but the two quantitative pillars (coverage and detection improvement) rest on fragile evidence: a self-defined denominator and an unseeded single-run evaluation. Both are fixable within the manuscript's scope but require substantial additional experimentation and re-analysis. The paper also makes several novelty claims ('first automated log synthesis framework for anomaly detection', 'first approach to automated log generation using LLM for anomaly detection') that should be contextualized against existing work such as AutoLog and UniLog; the related work section lists these but does not clearly differentiate the claims. I would like the editor to weigh whether this is a venue for empirical software engineering or for datasets; if the latter, the artifact release and reproducibility warrants closer scrutiny than the current text provides."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: AnomalyGen is a plausible idea—use LLM chain-of-thought to enrich static call-graph analysis and generate anomaly-labeled log sequences without running the system. That combination is genuinely new relative to AutoLog (pure static) and UniLog/LANCE (log statement generation). The architecture is coherent, the expert annotation check (Krippendorff's α≈0.8) is good practice, and the recall check against LogHub event types (93/107 and 14/15) is a useful sanity check. There is real work here.\n\nThe soft spot is the headline coverage claim. Table 2 reports 38x–95x improvement in log event coverage, but the denominator (9,662 for Hadoop, 2,889 for HDFS) comes from the authors' own static analysis of Hadoop 3.3.6. The baseline datasets were collected on older versions and different workloads. A control-flow-expanded enumeration will naturally include log sites no real workload ever hits, making the baseline coverage look artificially tiny. The paper even acknowledges version-driven filtering in Section 5.2. The R-coverage numbers only validate recall of known LogHub event types; they say nothing about whether the denominator is right. So the main quantitative claim is not yet substantiated. This is not a nitpick; it is the headline result.\n\nThe downstream evaluation is thin: one system (HDFS), no error bars, no repeated runs, no statistical test, and the average F1 gain is 1.8 points. The realism argument rests on code examples, not measurements. The promised artifacts are not linked. The writing has inconsistencies (37x vs 38x vs 95x; missing citation for Busse et al.).\n\nNone of this makes the idea bad. The framework is coherent, the motivation is real, and the recall check suggests some genuine coverage. But the evidence as reported does not support the 38–95x coverage claim. A revision with a version-matched, independently verifiable event enumeration, confidence intervals, an AutoLog augmentation baseline, and released code would be a solid contribution. As is, it deserves peer review but not acceptance on current evidence.","headline":"A genuinely new combination of static analysis and LLM reasoning for log generation, but the headline coverage metric rests on a self-defined denominator and the downstream gain is thin; deserves a serious referee, not acceptance yet.","tokens_in":16170,"tokens_out":2383,"would_cite":false,"duration_ms":22477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnomalyGen shows that LLM-guided static analysis can synthesize log datasets with 38–95 times more event types than current benchmarks, lifting anomaly-detection F1 by up to 3.7 points.","keywords":["log anomaly detection","log sequence generation","large language models","chain-of-thought","static program analysis","control flow graph","synthetic training data"],"falsifier":"A direct check is to independently enumerate all reachable logging statements in the same two system versions—for example, by instrumenting the systems and comparing runtime log events with the paper's totals of 9,662 and 2,889—and recompute coverage; if the independently established totals are substantially lower, the coverage claims fail. A second test is to train a detector on generated data alone and measure its F1 on held-out real production logs; if it drops sharply, the simulated parameter distributions are not representative.","tokens_in":15163,"feed_emoji":"🤖","tokens_out":13136,"duration_ms":108941,"temperature":0.7,"pith_summary":"The paper argues that the scarcity and narrowness of public log datasets can be solved by generating synthetic log sequences without executing the target system. It presents AnomalyGen, a four-phase pipeline that prunes a program's call graph to logging-related code, builds fine-grained control-flow graphs with help from a large language model, recursively merges per-method log sequences under chain-of-thought verification, and labels anomalies using explicit and implicit rules. On two widely used distributed systems, the framework generates logs covering 38 to 95 times more log event types than the existing benchmark datasets, with an average reported coverage of 97.48% of all log events in the code. Adding these generated logs to the benchmarks' training data improves F1 scores across three anomaly detection architectures, with an average gain of 1.8 percentage points and a maximum of 3.7. The significance, if the claims hold, is a low-cost path to richer and more realistic training data for log-based anomaly detection.","feed_headline":"Synthetic logs cover 95x more event types; anomaly F1 rises 3.7 points","feed_subtitle":"No system execution needed: LLM-guided static analysis produces richer logs, boosting detection F1 by up to 3.7 points.","key_machinery":"The load-bearing mechanism is the enhanced control-flow graph (Enhanced_CFG), a per-method graph built from the ternary tuple $\\langle\\text{Source Code}, \\text{Call Path}, \\text{Log-Oriented CFG}\\rangle$. An LLM refines it through five verification steps: call matching and completion, exception-path augmentation, log-flow association, path-constraint handling, and execution-log consistency checking. The second load-bearing component is the chain-of-thought recursive merge, which joins child-method sequences into parent sequences bottom-up while checking control-flow consistency, data-flow integrity, and call-point correlation at each merge. This pairing of a code-grounded graph with LLM reasoning is what allows realistic log sequences to be produced without executing the system.","core_discovery":"The central discovery is that the blind spots of static analysis can be filled by a large language model without giving up the structural guarantees of a call graph: the model supplies dynamic dispatch, exception paths, and runtime parameter values, while the graph structure keeps the generated sequences tied to real code layout. AnomalyGen computes, for each logging-related method, an enhanced control-flow graph from a ternary of source code, call path, and log-oriented control-flow graph; it then simulates a call stack from the bottom up, asking the LLM to verify control-flow consistency, data-flow integrity, and call-point correlation at every merge. The paper reports that this execution-free process covers 97.48% of the log events it identifies in the code, expands event-type coverage 38-fold to 95-fold over the baselines, and, when added to benchmark training sets, improves F1 by up to 3.7 points (average 1.8) across LSTM, Transformer, and CNN detectors.","pith_inferences":["The coverage ratios are computed against a denominator the paper derives from its own static analysis; an independent enumeration of reachable logging statements could yield a smaller 'total log events' count and lower the reported 38–95x improvements.","Because the LLM simulates runtime parameters instead of observing them, generated datasets may carry distributional biases; a testable extension would be to measure how well a detector trained solely on AnomalyGen logs transfers to real production logs.","The same chain-of-thought verified recursive merge could be applied beyond logging, for instance to synthesize API call traces or event sequences for testing, whenever source code and a call graph are available.","The rule-based anomaly annotation is tied to explicit keywords and error codes; extending it with patterns mined from past incident reports would likely capture more implicit anomaly types than the current hand-crafted rules."],"forward_implications":["Training anomaly detectors on benchmark data augmented with AnomalyGen logs improves F1 scores on the HDFS anomaly detection task: LSTM rises from 0.917 to 0.954, Transformer from 0.852 to 0.867, and CNN from 0.969 to 0.971.","The generated datasets provide 38–95 times more log event types than the existing benchmarks, which should let detectors train on a wider variety of anomaly-prone paths and reduce failures on unseen log events.","Because each generated sequence carries its execution context and annotated anomaly label, the dataset supports root-cause tracing; the paper reports that matching a log to its control flow takes less than one-fifth of the time it took with manually reconstructed mappings.","The framework needs no system execution, only source code and a call graph, so it can be applied to other Java systems with manageable configuration effort."],"supporting_citations":[{"why":"The earlier static-analysis-based log sequence synthesis framework that AnomalyGen extends and uses as the main baseline for coverage and realism.","marker":"[13]"},{"why":"The Hadoop benchmark log dataset that serves as the baseline for event coverage and as the training set in augmentation experiments.","marker":"[25]"},{"why":"The HDFS benchmark log dataset used as the baseline for event coverage and augmentation experiments.","marker":"[44]"},{"why":"The large public collection of real system logs used to compute R-coverage against logs collected from actual executions.","marker":"[50]"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coverage figures rest on the paper's own count of how many log events exist in each system's source code; if that total is inflated, the claimed 38–95x coverage improvements shrink.","fun_headline_variants_meta":{"error":"Client error '402 Payment Required' for url 'https://api.deepseek.com/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:34:02.620352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to independently enumerate all reachable logging statements in the same two system versions—for example, by instrumenting the systems and comparing runtime log events with the paper's totals of 9,662 and 2,889—and recompute coverage; if the independently established totals are substantially lower, the coverage claims fail. A second test is to train a detector on generated data alone and measure its F1 on held-out real production logs; if it drops sharply, the simulated parameter distributions are not representative.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier static-analysis-based log sequence synthesis framework that AnomalyGen extends and uses as the main baseline for coverage and realism."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The HDFS benchmark log dataset used as the baseline for event coverage and augmentation experiments."}],"review_version":1}