{"id":"cc50898b-8527-423a-99c3-118df6176901","arxiv_id":"2412.01377","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"SuperLog, a LLaMA-2-7B model continually pre-trained on an LLM-generated natural-language QA corpus about logs, claims state-of-the-art results on four log analysis tasks.","lead":"This paper trains an open-source 7B language model for log analysis by first pre-training it on 250,000 question-answer pairs that explain log events in plain language, then fine-tuning it on a small amount of task data. The resulting SuperLog model reports large gains across log parsing, anomaly detection, fault diagnosis, and log interpretation, but those gains may be inflated because its training data overlaps with the benchmark domains.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NLPLog CPT data is drawn from the same LogHub domains (HDFS, Hadoop, Zookeeper, Linux, Proxifier, BGL) as the evaluation sets for log parsing and anomaly detection, and the paper reports no overlap or deduplication check; this contamination risk directly undermines the headline 12.01% improvement…","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the NLPLog CPT corpus and the evaluation sets share source domains and the paper provides no overlap analysis. I verified this from the manuscript: Table 1 lists HDFS, Hadoop, Zookeeper, Linux, Proxifier, and BGL among NLPLog domains; Section 4.4.1 tests log parsing on those same HDFS/Hadoop/Zookeeper/Linux/Proxifier domains and anomaly detection on BGL; Section 3.1.1 deduplicates at template level; and Section 4.4.1 evaluates on the last 90% of LogHub_2k logs. No train/test separation between CPT and downstream evaluation is described anywhere, and no deduplication check is reported. The unseen-domain experiment is not a valid control because it uses LLM-generated references and the paper itself flags this limitation in Section 5.2.1. Credit is due for releasing the dataset and code and for running a human-calibration phase, but those do not resolve the overlap issue. A concrete template-overlap audit with a retraining experiment would settle whether the central claim is real; until that is done, the headline accuracy improvement should not be accepted.","tokens_in":16181,"tokens_out":2881,"duration_ms":27272,"concrete_test":"Run a template-level overlap audit: apply the same LogPPT template extractor used in Section 3.1.1 to the evaluation splits from Section 4.4.1 (last 90% of LogHub_2k for HDFS, Hadoop, Zookeeper, Linux, Proxifier; BGL and Spirit template-label pairs) and to the NLPLog source log events. Count the fraction of distinct test templates, and of exact log events after variable replacement, that match an NLPLog template or a near-duplicate under a small edit-distance threshold. If any overlap exists, retrain SuperLog on NLPLog with those templates removed and rerun Tables 2 and 3; the generalization claim survives only if the reported gains persist. Also report whether the 2,000 LogHub_2k annotations used for parsing fine-tuning overlap with NLPLog templates.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.1 builds NLPLog by extracting 51,590 log templates from 14 LogHub domains (Table 1), including HDFS, Hadoop, Zookeeper, Linux, Proxifier, and BGL, and then generates Q&A pairs from reconstructed log events over these templates. Section 4.4.1 evaluates log parsing on the last 90% of LogHub_2k logs for exactly HDFS, Hadoop, Zookeeper, Linux, and Proxifier, and evaluates anomaly detection on BGL template-label pairs. The paper never checks whether test logs, their templates, or near-paraphrases of them appear in the NLPLog CPT corpus. Because NLPLog deduplication is at template level and test logs come from the same LogHub domains, a large fraction of test templates are plausibly present in the training Q&A data. If so, the reported parsing and anomaly-detection gains reflect memorization of template structure rather than a generalizable domain-adaptation advantage. This also infects the ablation in Table 6, because the 'w/o IK' raw-log CPT baseline is trained on the same contaminated templates. The RQ3 unseen-domain experiments do not provide a clean control: they compare outputs against LLM-generated references (Section 4.6.1), and the paper itself concedes in Section 5.2.1 that high ROUGE similarity to proxy answers does not guarantee the model grasped the underlying task. Thus the central claim that SuperLog achieves state-of-the-art results across four log analysis tasks is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SuperLog, a LLaMA-2-7B model adapted to log analysis by continual pre-training (CPT) on NLPLog, a corpus of over 250,000 natural-language question-answer pairs distilled from LogHub logs by a proprietary LLM. The authors evaluate SuperLog on log parsing, anomaly detection, failure diagnosis/interpretation, and logs from unseen domains, reporting state-of-the-art results and an ablation that supports interpretable-knowledge CPT over raw-log CPT. The manuscript claims an average accuracy improvement of 12.01% over the second-best model and a 23% average improvement over raw-log training.","tokens_in":16448,"tokens_out":8294,"duration_ms":71069,"significance":"If the empirical results were valid, the paper would provide a practical and reusable recipe for adapting open-source LLMs to log analysis, and NLPLog would be a valuable community resource. The core idea of replacing raw-log CPT with interpretable natural-language Q&A is interesting and the dataset release is a concrete contribution. However, the validity of the empirical claims is not established: the training corpus overlaps in domain and likely in templates with the main evaluation benchmarks, the unseen-domain evaluation is a self-agreement check against an LLM-generated reference, and several reported numbers are internally inconsistent. The potential significance is therefore conditional on a clean, leakage-free evaluation.","major_comments":[{"comment":"The log-parsing and anomaly-detection evaluations are not protected against training/evaluation overlap. NLPLog is constructed from templates extracted from the full LogHub domains (Section 3.1.1), including HDFS, Hadoop, Zookeeper, Linux, Proxifier, and BGL (Table 1). The log-parsing test set is the last 90% of LogHub_2k for the same five domains, and the anomaly-detection test uses BGL template-label pairs (Section 4.4.1). The paper reports no check for exact template matches, near-duplicates, or paraphrases between NLPLog training entries and the test logs/templates. Because NLPLog deduplication only removes duplicate templates within the CPT corpus, it cannot prevent test templates from being present in training. If test templates were seen during CPT, parsing F1 values near 0.99 and the anomaly-detection gains reflect memorization rather than a generalizable domain-adaptation advantage. The same problem affects the w/o IK ablation in Section 4.5.1, which trains on raw logs from the same domains, so Table 6 does not establish generalization beyond memorized templates. Please report exact overlap statistics at template and event level, and re-run all affected experiments after excluding overlapping entries, or evaluate on domains disjoint from NLPLog.","section":"§3.1, Table 1; §4.4.1, Tables 2-3"},{"comment":"The unseen-domain evaluation cannot support the claimed generalization. Table 7 measures ROUGE similarity to references produced by an unnamed 'advanced LLM' rather than to ground-truth templates or human annotations. The paper itself concedes in Section 5.2.1 that high surface similarity to these proxy answers does not guarantee that the model grasped the underlying task. A model that imitates the style of the reference LLM could score well on ROUGE without performing correct parsing. This self-agreement check is circular as evidence of capability, so the claim of 'exceptional performance on logs from unseen domains' is unsupported. Please evaluate on labeled data (e.g., human-annotated templates) or report task-specific accuracy against ground truth, and justify that the reference is a reliable gold standard.","section":"§4.6, Table 7; §5.2.1"},{"comment":"The headline '23% average performance improvement' is not reproducible from the reported ablation. Table 6 reports log interpretation as 3.895 for SuperLog, but Table 4 gives usefulness 4.430 and readability 4.780, whose average is 4.605, and Section 4.5.2 states that the interpretation score is the average of usefulness and readability. In addition, relative improvements of SuperLog over w/o IK computed from Table 6 are approximately 1.5% for parsing, 21.9% for anomaly detection, 30.9% for diagnosis, and 27.5% for interpretation, whose average is about 20.5%, not 23%. Please clarify the exact aggregation formula, reconcile the two tables, and state precisely how the 23% and 12.01% figures are computed.","section":"§1 and §4.5, Table 6"},{"comment":"All comparisons are based on single runs; no standard deviations, confidence intervals, or significance tests are reported. This matters because several differences are small (e.g., BGL session-level anomaly detection F1: 0.147 for SuperLog vs. 0.129 for LogPrompt in Table 3), and few-shot fine-tuning on a few thousand examples is stochastic. Please report multiple seeds and statistical tests, or explicitly justify that the reported differences exceed run-to-run noise.","section":"Tables 2-7"}],"minor_comments":[{"comment":"There are typos such as 'pathes' in the abstract, 'trainging' and 'propietary' in Section 1, and 'decipted' in Section 1.","section":"Abstract and §1"},{"comment":"Several entries lack separating spaces, e.g., '0.9480.889' for LogPrompt and '0.9980.815' for SuperLog; these should be fixed for readability.","section":"Table 2"},{"comment":"The dataset name is inconsistent: 'AlpaCar_1k' in Section 4.1.2 versus 'Alpaca-1k' in Section 4.5.1.","section":"§4.1.2 and §4.5.1"},{"comment":"The 'state-of-the-art LLM' used for knowledge generation and the 'advanced LLM' used for unseen-domain references are never named; please identify the model and version for reproducibility.","section":"§3.1.3 and §4.6.1"},{"comment":"The 'human-calibration phase' is described only qualitatively; please report how many responses were filtered, by how many annotators, and the inter-annotator agreement.","section":"§3.1.3"},{"comment":"The log-interpretation evaluation uses four experts on 100 logs but reports no inter-annotator agreement; a mean score without agreement information is difficult to interpret.","section":"§4.4.2"},{"comment":"The phrase 'average accuracy improvement of 12.01%' is misleading because the metrics are RI, F1, ROUGE, and human scores rather than accuracy; please use task-appropriate terminology.","section":"Abstract and §5.1"}],"recommendation":"reject","confidential_remarks":"The training/evaluation overlap issue is fundamental: NLPLog is built from the same LogHub domains and templates used for the main benchmarks, and no leakage check is reported. Combined with the circular unseen-domain evaluation and the internal numerical inconsistencies in the ablation table, I do not think the current evidence supports the central claims. A positive recommendation would require a cleanly split evaluation and a ground-truth-based unseen-domain study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is NLPLog, a 250k QA dataset for log analysis, and the direct comparison showing continual pre-training on interpretable QA beats CPT on raw logs. The recipe itself is standard, but the resource and that particular ablation are new and worth knowing about if you work in AIOps. The Spirit anomaly-detection and fault-diagnosis experiments use data outside NLPLog, so they give some clean evidence that the approach has legs.\n\nThe soft spot is real and load-bearing. NLPLog is built from LogHub domains that include HDFS, Hadoop, Zookeeper, Linux, Proxifier, and BGL. The log-parsing evaluation uses exactly those domains, and the BGL anomaly detection uses BGL templates. The paper deduplicates at template level but never checks whether evaluation templates or near-paraphrases appear in the CPT corpus. Given how the dataset was built, overlap is not just possible; it is probable. That makes the 12.01% average improvement and the parsing gains uninterpretable as evidence of generalization. The ablation in Table 6 suffers from the same issue, because the raw-log baseline is trained on the same contaminated templates.\n\nTwo smaller issues: there are no error bars anywhere, so we have no sense of variance; and the unseen-domain experiments compare model outputs to an LLM-generated reference, which is a similarity check, not ground truth. The paper itself concedes this in Section 5.2.1. These are not fatal to the whole paper, but they are fatal to the headline claim as stated.\n\nWhat holds up: the fault-diagnosis results (on Alibaba/China Mobile logs) and the Spirit anomaly-detection results are on domains not in NLPLog, and SuperLog still does well there. That is genuine evidence the approach transfers. The NLPLog dataset itself is a reusable asset, and releasing it is a real contribution.\n\nWho this is for: people building log-specialized LLMs or studying domain-adaptation strategies for industrial text. It deserves a serious referee because the resource and the core comparison are valuable, but the revision will be substantial: a strict overlap audit, human-labeled unseen-domain evaluation, and variance reporting. If those come through, the paper could support a modest version of its claim. As it stands, I would not repeat the 12.01% or the 23% numbers in my own work.","headline":"Worth a serious look for the NLPLog resource and the interpretable-vs-raw CPT comparison, but the headline accuracy claims are undermined by likely train/test overlap and by LLM-referenced unseen-domain evaluation.","tokens_in":17057,"tokens_out":2237,"would_cite":false,"duration_ms":21011,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continual pre-training on natural-language Q&A pairs about logs, instead of raw log text, is claimed to adapt open-source LLMs to log analysis and to yield the best reported results across four tasks.","keywords":["log analysis","continual pre-training","large language models","domain adaptation","interpretability","question-answer dataset","log parsing","anomaly detection"],"falsifier":"Take the 51,590 NLPLog templates and the evaluation templates from LogHub_2k and BGL/Spirit, and run exact and near-duplicate matching after masking variables. If a large fraction of the test templates or their paraphrases appear in the continual-pretraining corpus, the 12.01% average gain is explained by memorization rather than by domain adaptation; in that case, retraining on domains entirely absent from the evaluation set would show whether the parsing and anomaly-detection gains persist.","tokens_in":15951,"feed_emoji":"🪵","tokens_out":6571,"duration_ms":54787,"temperature":0.7,"pith_summary":"The paper argues that the right way to adapt a general-purpose open-source LLM to log analysis is to continue pre-training it on natural-language question-answer pairs that explain logs, rather than on raw log lines. To do this the authors build NLPLog, a dataset of over 250,000 Q&A pairs covering five dimensions of log knowledge across 14 domains, and use it to continual-pretrain LLaMA2-7B into SuperLog. They report that SuperLog outperforms prior methods on log parsing, anomaly detection, fault diagnosis, and log interpretation, with an average improvement of 12.01% over the second-best baseline. An ablation replacing the Q&A corpus with raw logs gives about a 23% lower average performance, supporting the claim that interpretable knowledge, not just domain text, drives the gain.","feed_headline":"Log Q&A pretraining beats raw-log training by 23 percent","feed_subtitle":"A 250k-question log corpus adapts an open LLM to four log-analysis tasks at once","key_machinery":"The load-bearing object is NLPLog, a corpus of 250,000+ Q&A pairs in natural language, each pairing a real log event with answers along five knowledge dimensions: Grok pattern parsing, log event insights, root cause analysis, component correlation, and potential failure forecast. The construction pipeline starts from LogHub logs, extracts 51,590 unique templates with LogPPT, reconstructs representative events by recombining templates with variable groups, and generates the Q&A pairs with a large proprietary LLM followed by human calibration. Continual pre-training then runs LLaMA2-7B on these pairs for 1.5 epochs at a learning rate of 1e-5, followed by task-specific or general instruction tuning.","core_discovery":"SuperLog is a LLaMA2-7B model that first undergoes continual pre-training on NLPLog, a dataset of over 250,000 natural-language question-answer pairs built from real logs, and then lightweight instruction tuning. The paper's discovery is that this order, interpretable Q&A before task fine-tuning, transfers log knowledge without the distribution mismatch and catastrophic forgetting observed when continual pre-training is done on raw logs. The authors report the best performance on log parsing, anomaly detection, failure diagnosis, and log interpretation, with an average 12.01% improvement over the second-best model and strong outputs on Apache and OpenStack logs not seen during training.","pith_inferences":["The paper's evaluation domains overlap its pre-training domains, so the cleanest test of the generalization claim would be to pre-train on one set of log domains and evaluate on disjoint ones with human ground truth; the current unseen-domain test uses ROUGE against another LLM's outputs.","The same Q&A-injection scheme could be tried on other semi-structured operational data, such as network traces, Kubernetes events, or cloud billing records, where raw tokens are far from natural language.","Because the Q&A answers come from a proprietary LLM, the dataset's ceiling is set by that generator; a cheaper open loop would distill explanations from the base model itself or from retrieval over documentation.","A direct comparison of continual-pretraining token budgets would disentangle 'more data helps' from 'Q&A format helps'; the ablation holds the data source fixed but does not necessarily hold the token count fixed."],"forward_implications":["If the central claim holds, continual pre-training on interpretable Q&A pairs is a general recipe for adapting open-source LLMs to log analysis, replacing raw-log continual pre-training.","Log interpretation quality improves under Q&A continual pre-training, whereas raw-log continual pre-training lowers it relative to no continual pre-training, meaning interpretability is not sacrificed for task accuracy.","Performance transfers to Apache and OpenStack logs outside the training domains, suggesting the learned knowledge is not limited to the 14 training domains.","The 23% average gain in the ablation over raw-log training quantifies the value of the interpretable-knowledge format itself, not just the extra data.","The 12.01% average improvement over the second-best model across four tasks implies that the method is competitive without relying on proprietary APIs at inference time."],"supporting_citations":[{"why":"Supplies the 14 real-world LogHub domains and the LogHub_2k parsing ground truth that NLPLog and the log parsing evaluation are built on.","marker":"[19]"},{"why":"Provides the prompt-based template extraction method used to deduplicate logs into 51,590 unique templates for NLPLog.","marker":"[28]"},{"why":"Defines the continual pre-training paradigm that the paper adapts for injecting interpretable domain knowledge.","marker":"[17]"},{"why":"Identifies LLaMA2-7B as the base model that becomes SuperLog after continual pre-training and instruction tuning.","marker":"[52]"},{"why":"Supplies the LogEval benchmark and the Alibaba Cloud and China Mobile failure logs used for zero-shot failure detection and diagnosis evaluation.","marker":"[6]"},{"why":"Supplies the BGL and Spirit log datasets used for the anomaly detection evaluation.","marker":"[40]"},{"why":"Provides the LogPrompt baseline and prompt-strategy reference for log parsing and anomaly detection comparisons.","marker":"[32]"},{"why":"Provides the Alpaca-1k instruction-following examples used for zero-shot instruction tuning of SuperLog and baselines.","marker":"[14]"}],"fun_headline_variants":["Log Q&A pretraining beats raw-log training by 12%","LLMs adapt to logs via 250k interpretable Q&As","SuperLog: Q&A-pretrained LLM wins log tasks by 12%","Four log tasks, one Q&A corpus: SuperLog gains 12%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains assume that NLPLog's training entries are disjoint from the logs and templates used in evaluation, since NLPLog is built from the same LogHub domains (HDFS, Hadoop, Zookeeper, Linux, Proxifier, BGL) that appear in the parsing and anomaly-detection tests and no overlap check is reported.","fun_headline_variants_meta":{"raw":{"variants":["Log Q&A pretraining beats raw-log training by 12%","LLMs adapt to logs via 250k interpretable Q&As","SuperLog: Q&A-pretrained LLM wins log tasks by 12%","Four log tasks, one Q&A corpus: SuperLog gains 12%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2129,"prompt_tokens":905,"completion_tokens":1224,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1142}},"tokens_in":521,"tokens_out":1224,"duration_ms":11143,"temperature":1.0,"reasoning_tokens":1142,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:24:30.136693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 51,590 NLPLog templates and the evaluation templates from LogHub_2k and BGL/Spirit, and run exact and near-duplicate matching after masking variables. If a large fraction of the test templates or their paraphrases appear in the continual-pretraining corpus, the 12.01% average gain is explained by memorization rather than by domain adaptation; in that case, retraining on domains entirely absent from the evaluation set would show whether the parsing and anomaly-detection gains persist.","supporting_citations":[{"cited_title":"Le and H","cited_arxiv_id":null,"evidence_quote":"Provides the prompt-based template extraction method used to deduplicate logs into 51,590 unique templates for NLPLog."},{"cited_title":"Oliner and J","cited_arxiv_id":null,"evidence_quote":"Supplies the BGL and Spirit log datasets used for the anomaly detection evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LogPrompt baseline and prompt-strategy reference for log parsing and anomaly detection comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Alpaca-1k instruction-following examples used for zero-shot instruction tuning of SuperLog and baselines."}],"review_version":1}