{"id":"2287d356-a0ec-4320-8ac4-b9b6dba3fdff","arxiv_id":"2411.08561","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LogLLM combines BERT message embeddings with Llama sequence classification via a learned projector and regex preprocessing, reporting higher F1 scores than nine baselines on four log datasets.","lead":"LogLLM is a new method that reads software system logs and decides whether a sequence of log messages indicates a system problem. It combines a text-understanding model with a larger language model and reports better detection scores than nine existing methods on four public datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is under-tested because the closest decoder-only LLM baseline, Hadadi et al. (2024), is cited but never evaluated; without it, 'best existing method' is only best among smaller models.","rationale":"I do not find the labeling assumption to be the most load-bearing attack on the central claim. The OR-ing of message-level labels inside 100-message windows is applied uniformly to LogLLM and all baselines, and it is the standard benchmark definition used by the original NeuralLog work, so it affects external validity but not the internal comparative claim. The internal F1 inconsistency for LogAnomaly on HDFS is real but does not change the ranking, since LogLLM still exceeds NeuralLog on every dataset. The absence of error bars is a reproducibility concern, but the reported margins are large enough that variance would need to be extremely high to overturn the per-dataset ranking. The missing GPT-based baseline is different: it is a closely related method of comparable model scale that is cited in the paper and specifically aimed at unstable logs, which the abstract claims as a strength. Without evaluating it, the statement that LogLLM 'outperforms state-of-the-art methods' is not established. The reader's rationale already mentions this omitted GPT baseline, so my concern partially agrees with the reader, even though the reader's stated weakest assumption is the labeling rule. Since the reader's verdict is CONDITIONAL and this concern reinforces rather than overturns that conditional status, I recommend no change to the verdict.","tokens_in":17648,"tokens_out":9077,"duration_ms":88954,"concrete_test":"Reproduce Hadadi et al. (2024) under exactly the same protocol used for Table II: 100-message non-overlapping windows for BGL, Liberty, and Thunderbird; session windows for HDFS; OR-based sequence labels; and the same 80/20 chronological (or random for HDFS) splits. Use a decoder-only GPT-family model of comparable scale to Llama-3-8B, fine-tuned with the same QLoRA budget and preprocessing as the original method. Compute per-dataset precision, recall, and F1 and compare with LogLLM's values in Table II. If this omitted baseline reaches or exceeds LogLLM's average F1 of 0.959, the 'best existing method' claim fails; if LogLLM still leads by a clear margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section V-E is that LogLLM outperforms state-of-the-art methods, with NeuralLog named as the best existing method. This conclusion depends entirely on the comparison set in Table II. That set omits the one closely related decoder-only LLM fine-tuning baseline the paper itself cites in Section II-B: Hadadi et al. (2024), 'Anomaly detection on unstable logs with GPT models'. The paper dismisses that line of work with two asserted challenges (unclear template boundaries and excessive token counts) and claims these are demonstrated in Section V-G, but Section V-G only tests directly concatenating raw log text into Llama, not Hadadi et al.'s actual method. This matters because NeuralLog uses BERT-base plus a small transformer classifier, while LogLLM uses BERT-base plus Llama-3-8B. A 6.6% F1 advantage over a much smaller model is therefore not yet evidence that the proposed architecture or three-stage training is state-of-the-art among LLM-based detectors; it may be evidence about model scale. The missing baseline is the most load-bearing issue because it directly concerns whether the headline 'outperforms state-of-the-art methods' is true, rather than only how the result should be interpreted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LogLLM, a supervised log-sequence anomaly detection framework that replaces log parsing with regex-based normalization, embeds each log message with BERT, projects the embeddings with a linear layer, and feeds them as soft prompts to Llama for sequence classification. Training relies on minority-class oversampling and a three-stage QLoRA procedure. Experiments on HDFS, BGL, Liberty, and Thunderbird compare against nine baselines and report the highest F1 on all four datasets, with additional ablations on preprocessing, embedder use, Llama size, training stages, and the oversampling parameter β. The paper claims that LogLLM outperforms state-of-the-art methods and that it handles unstable logs without log parsers.","tokens_in":17905,"tokens_out":7851,"duration_ms":70659,"significance":"If the empirical claims hold, the paper makes a useful practical contribution: it avoids fragile log parsers, keeps sequence lengths manageable by embedding individual messages before feeding Llama, and provides a concrete three-stage training recipe. The code is publicly linked, the evaluation uses four real public datasets, and the ablation study is instructive. However, the headline state-of-the-art claim is currently under-supported because the most relevant decoder-only LLM baseline cited in the paper is not evaluated, and at least one baseline number in the main table is internally inconsistent. The comparison is also confounded by model scale, and the unstable-logs claim is not directly measured. With a strengthened comparison and corrected tables, the method could become a solid empirical contribution.","major_comments":[{"comment":"Section V-E states that LogLLM achieves the highest F1 across all datasets and is 6.6% better than the best existing method, NeuralLog. This comparison omits Hadadi et al. [40], the fine-tuning-based GPT baseline discussed in Section II-B. The two challenges used to dismiss that work (unclear template boundaries and excessive token counts) are claimed to be demonstrated in Section V-G, but Section V-G only tests concatenating raw log messages into Llama-3.2-1B; it does not test Hadadi et al.'s template-based GPT fine-tuning pipeline. Because LogLLM uses Llama-3-8B while NeuralLog uses BERT-base with a small classifier, the observed margin may reflect model scale rather than the proposed architecture. Please add the missing baseline or restrict the state-of-the-art claim to the compared methods.","section":"V-E, Table II"},{"comment":"In the LogAnomaly HDFS row, Precision=0.886 and Recall=0.893 give F1=0.889, not the printed 0.966. This is an arithmetically impossible result and undermines confidence in the transcription or computation of the baseline table. All baseline F1 values should be recomputed from raw predictions and re-verified.","section":"V-E, Table II"},{"comment":"On Thunderbird, the test set contains only 29 anomalous sequences (0.15%). The reported F1=0.966 therefore corresponds to roughly 28 true positives, 1 false positive, and 1 false negative. The average F1 advantage claimed in Section V-E is thus sensitive to a handful of test instances. Please report per-dataset confidence intervals, repeated runs with different random seeds, or a larger test sample before drawing comparative conclusions.","section":"V-D, V-E, Table I"},{"comment":"The preprocessing comparison applies Drain to the entire dataset, including the test split, to avoid OOV degradation. This leaks test information into the parser in exactly the unstable-log condition the paper claims to address, and it biases the comparison against parser-based preprocessing. Please fit Drain on the training data only, or separately quantify the number of OOV templates that appear in the test split.","section":"V-F, Table IV"},{"comment":"The oversampling proportion β is set to 30% in Section V-B after Section V-J reports its test-set F1 curve and concludes that values between 30% and 50% are acceptable. No separate validation set is described, so β selection appears to use test labels. Please introduce a validation split for hyperparameter selection, or present the sensitivity analysis as post-hoc rather than as model selection.","section":"V-B, V-J"},{"comment":"The abstract's claim that LogLLM handles unstable logs is not directly measured. The chronological splits for BGL, Thunderbird, and Liberty ensure that training precedes test, but the paper reports no statistics on how many new log templates or OOV tokens appear in the test windows, nor how template drift degrades parser-based competitors. Please add a direct drift analysis, such as template novelty rate or vocabulary overlap between train and test, to substantiate this claim.","section":"V-D, Abstract"}],"minor_comments":[{"comment":"The stated GPU memory increase of 7.7 GB when moving from Llama-3.2-1B to Llama-3-8B does not match Table V; the pairwise average of the reported memory numbers is 6.7 GB. Please check the calculation.","section":"V-H, Table V"},{"comment":"The architecture is described with a 'projector' in Section IV-B, but Section V-G calls the same component an 'adapter'. Please unify the terminology.","section":"IV-B, V-G"},{"comment":"Stage 1 is said to require 'only a few data samples' in Section IV-C3, while Section V-B states that 1,000 samples are used. Please clarify how these Stage 1 samples are selected and whether they are drawn from the training set.","section":"IV-C3, V-B"},{"comment":"For the 'Template ID' row, the description says template IDs are encoded with an embedding layer instead of BERT; please clarify whether the same LogLLM architecture and three-stage training procedure are used for all four preprocessing variants.","section":"V-F, Table IV"},{"comment":"The computational-cost table reports timings averaged across datasets but does not specify whether the numbers include GPU warm-up, evaluation, or data loading. Please state the measurement protocol.","section":"Table III"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents a plausible architecture and useful ablations, but the empirical claims are currently overreaching relative to the evidence. The paper cites Hadadi et al. (2024) as related work but omits it from the experimental comparison, and Section V-G does not test that baseline; this should be addressed before publication. The impossible LogAnomaly F1 in Table II suggests that the baseline numbers need a careful audit. There is also no repeated-run variance, and β appears to be selected after observing test-set behavior. These issues are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real engineering contribution with a plausible result, but the headline \"outperforms state-of-the-art\" is under-supported until the closest LLM baseline is actually included. If you need a strong log-anomaly detector that avoids log parsers, LogLLM is credible; if you need to trust the SOTA claim, keep reading.\n\nWhat's new: combining BERT per-message embeddings fed through a projector into Llama-8B for sequence classification is a natural composition, but the paper executes it carefully. The three-stage training procedure (answer-template warm-up, embedder training, full fine-tune) is well motivated, and the ablations show each stage earns its keep. The preprocessing comparison (raw vs. template vs. template ID vs. regex) is practical and useful; regex wins convincingly. The results on four public datasets, including the chronological split for BGL/Thunderbird/Liberty, are broadly consistent with the claims.\n\nSoft spots: the Table II F1 for LogAnomaly on HDFS is arithmetically impossible (precision 0.886, recall 0.893 cannot give F1 0.966). That kind of internal inconsistency makes me wonder about the rest of the table. More importantly, the paper cites Hadadi et al. (2024), the closest decoder-only LLM baseline, but never evaluates it. Section V-G only tests directly concatenating raw log text into Llama, not Hadadi's actual pipeline (parsed templates into GPT), so the claimed two challenges are not demonstrated against that method. Without that baseline, \"state-of-the-art\" is only best among the compared smaller models. There are also no error bars or repeated runs, and the unstable-log claim lacks template-drift statistics, so the chronological split alone doesn't prove robustness.\n\nThat said, the architecture, training recipe, and ablations are solid engineering. The beta sensitivity analysis is informative and shows the method isn't brittle to that hyperparameter. The code exists though not pinned to a commit.\n\nBottom line: this deserves a serious referee who can push for the missing baseline, corrected numbers, and variance reporting. With those revisions, it becomes a useful, citable contribution to the log-anomaly literature.","headline":"Solid engineering result in log anomaly detection, but the SOTA claim is under-tested until the closest LLM baseline is added and the Table II arithmetic is fixed.","tokens_in":18418,"tokens_out":2652,"would_cite":true,"duration_ms":24558,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims LogLLM, a BERT-to-Llama log anomaly detector with no log parser, outperforms nine prior methods on four public datasets and beats the strongest baseline by 6.6% average F1.","keywords":["log-based anomaly detection","large language models","BERT","Llama","semantic embedding","log preprocessing","three-stage training","unstable logs"],"falsifier":"Re-label BGL, Thunderbird, and Liberty using only sequences that can be tied to documented failure incidents (rather than any window containing a flagged message), retrain and retest LogLLM against NeuralLog under the same chronological split; if the average F1 advantage disappears or reverses, the paper's headline claim is an artifact of the window-labeling rule.","tokens_in":17461,"feed_emoji":"🤖","tokens_out":5494,"duration_ms":48889,"temperature":0.7,"pith_summary":"The paper sets out to show that a supervised log anomaly detector built from two pretrained language models can beat both parser-based and parser-free predecessors. LogLLM masks variable parameters with regular expressions, encodes each log message into a semantic vector with BERT, projects those vectors into Llama's embedding space, and fine-tunes Llama to classify each sequence as normal or anomalous. On HDFS, BGL, Liberty, and Thunderbird, the authors report the highest F1-score among ten methods, with average F1 6.6% above NeuralLog, and they argue that the design removes log-parser dependence and handles unstable logs.","feed_headline":"LogLLM beats top log anomaly detectors by 6.6 percent F1","feed_subtitle":"Parser-free BERT-plus-Llama system tops HDFS, BGL, Liberty, and Thunderbird.","key_machinery":"The load-bearing object is the learned embedding bridge: a frozen BERT encoder with a trainable [CLS]-based semantic-vector head, followed by a single linear projector that maps each message vector into Llama's token-embedding space. The projected vectors are inserted between two textual prompt pieces (\"Below is a sequence of system log messages:\" and \"Is this sequence normal or anomalous?\") and fed to Llama. This design gives each log message a clean boundary, avoids concatenating the raw log into a long string, and so sidesteps out-of-memory failures while keeping the semantic knowledge of both pretrained models.","core_discovery":"On the paper's own terms, the central discovery is that a transformer encoder and a transformer decoder can be joined into one log anomaly detector: BERT summarizes each log message into one vector, a learned linear projector aligns that vector to Llama's token-embedding space, and Llama classifies the resulting sequence. Trained in three stages (answer-template fine-tuning, embedder training, then full fine-tuning), LogLLM is reported to achieve F1-scores of 0.997 on HDFS, 0.916 on BGL, 0.958 on Liberty, and 0.966 on Thunderbird, outperforming all nine compared methods on every dataset. On average the paper computes a 6.6% F1 improvement over the best existing method, NeuralLog.","pith_inferences":["The 6.6% advantage is measured against window labels built by OR-ing message-level labels; if operators care about incidents rather than any flagged message, the practical gap may differ.","Because the method is supervised and benefits from labeled anomalies, its comparison with semi-supervised and training-free alternatives mixes in the effect of supervision, not only the LLM architecture.","Regex masking assumes parameters have recognizable formats; logs with free-text variable content would test whether the parser-free advantage survives.","The claim of robustness to unstable logs would be sharpened by reporting how many unseen templates appear in the test windows and how LogLLM's F1 varies with that quantity."],"forward_implications":["Log parsers are unnecessary: regex masking of parameters yields higher F1 than Drain templates or template IDs on all four datasets.","Message-level embedding lets a decoder-only LLM handle sequences of hundreds of messages without out-of-memory errors, which direct raw-text input cannot do.","The three-stage training procedure is load-bearing: dropping Stage 1 lowers average F1 by 29.7%.","Supervised anomaly labels matter: methods trained with labeled anomalies outperform reconstruction-based and retrieval-based methods on these datasets.","Chronological splits make the result relevant to unstable logs, where new templates appear after training."],"supporting_citations":[{"why":"NeuralLog, the strongest baseline; supplies the parser-free idea LogLLM builds on and the 6.6% average-F1 comparison target.","marker":"[3]"},{"why":"BERT, the encoder whose [CLS] vectors become per-message semantic embeddings.","marker":"[43]"},{"why":"Llama 3, the decoder model that classifies the projected embedding sequences.","marker":"[26]"},{"why":"QLoRA, the efficient fine-tuning method used to train BERT and Llama at reduced memory.","marker":"[55]"},{"why":"Drain, the log parser used in the preprocessing ablation that regex masking outperforms.","marker":"[51]"},{"why":"HDFS dataset, source of session-window sequences and labels for the largest benchmark.","marker":"[49]"},{"why":"BGL, Thunderbird, and Liberty datasets, providing message-level labels and sliding-window sequences.","marker":"[50]"},{"why":"Chronological train/test split protocol for unstable logs, which LogLLM adopts for BGL, Thunderbird, and Liberty.","marker":"[6]"},{"why":"LogHub, the public collection from which the four datasets are taken.","marker":"[57]"}],"fun_headline_variants":["LogLLM joins BERT and Llama to spot log anomalies","Parser-free BERT and Llama raise log anomaly detection","LogLLM: BERT encoder plus Llama decoder for log anomalies","LogLLM achieves top log anomaly detection with BERT and Llama","LogLLM fuses BERT and Llama, outpacing 9 log detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on treating a 100-message window as anomalous when at least one message inside it is labeled anomalous; if that window rule is not what system operators mean by an anomaly, the reported F1 improvement is an improvement on that labeling rule rather than on real incident detection.","fun_headline_variants_meta":{"raw":{"variants":["LogLLM joins BERT and Llama to spot log anomalies","Parser-free BERT and Llama raise log anomaly detection","LogLLM: BERT encoder plus Llama decoder for log anomalies","LogLLM achieves top log anomaly detection with BERT and Llama","LogLLM fuses BERT and Llama, outpacing 9 log detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2598,"prompt_tokens":910,"completion_tokens":1688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1593}},"tokens_in":526,"tokens_out":1688,"duration_ms":10997,"temperature":1.0,"reasoning_tokens":1593,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:31:48.985573+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-label BGL, Thunderbird, and Liberty using only sequences that can be tied to documented failure incidents (rather than any window containing a flagged message), retrain and retest LogLLM against NeuralLog under the same chronological split; if the average F1 advantage disappears or reverses, the paper's headline claim is an artifact of the window-labeling rule.","supporting_citations":[{"cited_title":"Online system problem detection by mining patterns of console logs,","cited_arxiv_id":null,"evidence_quote":"HDFS dataset, source of session-window sequences and labels for the largest benchmark."}],"review_version":1}