{"id":"e82fca92-9ad5-41c4-ace5-e8c103843a9d","arxiv_id":"2501.12166","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"TempoLog replaces fixed-size log windows with continuous-time dynamic graphs and link prediction to detect anomalies at individual event level.","lead":"This paper proposes TempoLog, a log anomaly detection method that builds a time-dependent graph of log templates instead of chopping logs into fixed-size windows. It reports state-of-the-art event-level F1 scores on three HPC log datasets, but the evaluation protocol has unresolved fairness and reproducibility gaps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol is undefined: no window-to-event label mapping for baselines and internally inconsistent train/test splits make the reported state-of-the-art F1 gains uninterpretable.","rationale":"The paper's core idea is coherent: construct continuous-time dynamic graphs from parsed log templates, initialize node memory with BERT embeddings, and detect anomalies via link prediction. If implemented as described, the event-level output of TempoLog itself is well-defined. The least secure part of the argument is the comparative evaluation. The reader's weakest_assumption identified the missing window-to-event mapping for baselines, and I agree; the reader also noted the split inconsistency, which is confirmed by Table 1 and Sections 4.1/4.2. I do not find an internal flaw in the model architecture, but the empirical foundation of the state-of-the-art claim is incomplete. Two secondary points reinforce the conditional verdict: the 'multi-scale' claim is tested only for H={0,1}, so the reported results do not demonstrate the benefit of more than two scales, and the authors acknowledge unstable training in Section 4.6 yet report no multiple-seed variance or error bars. A single re-evaluation with a stated event-level mapping and consistent splits would settle the main concern; until then, CONDITIONAL is appropriate.","tokens_in":10369,"tokens_out":4851,"duration_ms":53551,"concrete_test":"Ask the authors to release the evaluation script and recompute Table 2 with one explicit event-level mapping: all events inside a flagged window are labeled positive, and evaluate all methods on the same 50/50 chronological split stated in Section 4.1 (correcting the Spirit row to match the actual split). If any reported baseline F1 changes by more than 0.05, or if TempoLog no longer ranks first on BGL or Thunderbird, the current comparison protocol is the cause and the 'significantly outperforming' claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, stated in the Abstract and Section 4.2, is that TempoLog achieves state-of-the-art event-level anomaly detection. Section 4.1 defines precision/recall/F1 at event level ('TP are the anomalous event that are accurately detected'), but the seven baselines are window-based: Section 4.2 says 'for the other window-based models, we utilize a fixed-size window to group logs and the window size is 100.' No rule is given for converting a window-level alarm into event-level TP/FP. If every event in a flagged window is counted as positive, normal events inside that window become false positives; if only the anomalous event inside a flagged window is counted, the baseline numbers change substantially. DeepLog and LogAnomaly predict the next template, so their reported event-level F1 scores (0.226 and 0.224 on BGL) cannot be reproduced from the text without a stated localization rule. Independently, the split is inconsistent: Section 4.1 says 50% of logs are training and the rest testing, Section 4.2 says 40% of data is test, and Table 1 lists Spirit #Train logs as 25,000,000 out of 500,000,000 total, which is 5%, not 50%. The link-prediction decision threshold for the score sigma(f(zi,zj)) in Eq. (8) is also never specified. Each of these omissions affects the numerical comparison and prevents verification of the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TempoLog, an event-level log anomaly detection framework that replaces fixed-size window grouping with continuous-time dynamic graphs. Log templates are embedded with BERT and represented as nodes; directed temporal edges are created at multiple hop distances and enriched with hand-crafted edge features (semantic similarity, co-occurrence frequency, time interval, log level). A TGN-style memory/attention encoder produces node embeddings, and a link-prediction head (Eq. 8) is trained to decide whether an edge should exist; violations are flagged as event-level anomalies. On BGL, Spirit, and Thunderbird, the authors report F1 scores of 0.986, 0.993, and 0.987, outperforming seven window-based baselines, and they include ablations, a case study, efficiency measurements, and a threats-to-validity section.","tokens_in":10576,"tokens_out":6699,"duration_ms":64162,"significance":"If the evaluation is made fully transparent, TempoLog would be a meaningful contribution: event-level detection directly addresses the fuzzy-localization problem, and the multi-scale CTDG with semantic memory is a reasonable way to avoid window-size sensitivity. The ablation study and the efficiency discussion are useful, and the central claims are falsifiable. However, the paper currently provides no code and the reported state-of-the-art comparison rests on an underspecified evaluation protocol; the missing window-to-event mapping and the internally inconsistent data statistics prevent verification of the headline performance claims. The modeling idea is plausible and worth publishing after the evaluation issues are resolved.","major_comments":[{"comment":"The baselines are window-based (window size 100, Section 4.2), yet all metrics are defined at event level in Section 4.1. The paper never states how a window-level alarm is converted into event-level TP/FP/FN. For example, DeepLog and LogAnomaly predict the next log template, and their reported event-level F1 scores of 0.226 and 0.224 on BGL cannot be reproduced from the text without a stated localization rule. Please specify the exact mapping used (e.g., whether every event in an alerted window counts as a positive, or only the first mismatching event) and confirm that the same rule was applied to every baseline.","section":"Section 4.1/4.2, Table 2"},{"comment":"The data split is internally inconsistent. Section 4.1 says 50% of logs are training and the rest are testing; Section 4.2 says 40% of the data is test; Table 1 has a caption 'Training set ratio=0.5' but lists Spirit #Train logs as 25,000,000 out of 500,000,000, which is 5%, not 50%. In addition, the parenthetical anomaly percentages for Spirit (15.30%, 11.82%, 3.47%) do not match the listed counts: 764,891/500,000,000 is 0.153%, not 15.30%. These discrepancies must be reconciled because the baseline comparison and the discussion of anomaly ratios depend on the actual sample size and split.","section":"Section 4.1, Section 4.2, Table 1"},{"comment":"The link-prediction decision rule is underspecified. The text says an anomaly is detected when the predicted link relationship is inconsistent with the real link relationship, but no decision threshold on sigma(f(zi,zj)) is given, and there is no description of how negative edges are sampled for training or for the test-time decision. Since event-level precision and recall will vary strongly with this threshold and with the negative-edge construction, please report the threshold, the procedure used to select it, and the negative-edge sampling scheme.","section":"Section 3.5, Eq. (8)"},{"comment":"The H=0 graph, which is used in the default setting H={0,1} and in the H=0 ablation, is never defined. Section 3.3 gives examples only for H=1 and H>1. Without a formal definition of the nodes and edges for H=0, the model and the ablation comparing H=0 with H={0,1} are not reproducible.","section":"Section 3.3, Table 2"}],"minor_comments":[{"comment":"The sentence defining FN says 'the reset are FN'; this should be 'the rest are FN'.","section":"Section 4.1"},{"comment":"The text says 'It includes four core modules: memory, message function, message aggregator, memory updater, and node embedding' but five items are listed; please correct the count or the list.","section":"Section 3.4"},{"comment":"The text refers to 'y_hat_ij' as the predicted probability, but Eq. (8) defines L(Theta) = BCE(y_ij, sigma(f(zi,zj))); please align the notation and explicitly define y_ij as the ground-truth edge indicator.","section":"Section 3.5, Eq. (8)"},{"comment":"The checkmark layout is garbled in the submitted version, so it is unclear which feature is removed in each ablation row; please label rows explicitly (e.g., 'w/o C1') and fix the 'Precison' header.","section":"Table 3"},{"comment":"The y-axis label 'Log10(Time)(ms)' is ambiguous; please specify the units and whether lower values are better, and fix the 'T est' typo in panel (b).","section":"Figure 6"},{"comment":"The log-level feature LL(vi) is defined for a single node, but it is used as an edge feature; please clarify how the log levels of the two endpoints are combined into the edge representation.","section":"Section 3.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-protocol omissions and the data-table inconsistencies are likely fixable, but they are currently decisive; I would not accept the paper before seeing the corrected tables and an explicit window-to-event mapping applied uniformly to all baselines. After the evaluation is resolved, the novelty claim about being the first to use multi-scale temporal graphs for log data should be checked carefully against prior graph-based log anomaly detection work such as Log2Graph, LogGD, and LogGC."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper has a genuinely new idea in log anomaly detection—replace fixed-size windows with continuous-time dynamic graphs and detect anomalies via link prediction at the event level. If the numbers hold, it's a meaningful step forward; but the current evaluation is under-specified enough that the headline F1s cannot be taken at face value.\n\nWhat's actually new: the paper is the first among its cited literature to build CTDGs directly from raw log sequences, rather than windowed subsequences. The design is sensible: BERT-initialized memory per template, TGN-style message passing, and edge features (semantic similarity, co-occurrence, time interval, log level). The ablation study mostly supports the design choices: removing any single feature hurts, and semantic information matters. The authors also acknowledge unstable training as a threat to validity, which is honest.\n\nSoft spots, in order of severity. First, there is no description of how window-based baselines are converted to event-level TP/FP. Section 4.1 defines TP as 'the anomalous event that is accurately detected,' but DeepLog/LogAnomaly output next-template predictions, not event-level decisions. Without a stated localization rule, the comparison in Table 2 is uninterpretable. Second, the train/test split is internally inconsistent: Section 4.1 says 50% train/rest test, Section 4.2 says 40% test, and Table 1's Spirit row shows 25M train logs out of 500M total—5%, not 50%. Third, the link-prediction threshold is never stated; Eq. (8) uses a sigmoid but the decision rule is vague. Fourth, only H=0 and H={0,1} are evaluated, so calling the model 'multi-scale' without testing H>1 is an overclaim. Finally, no error bars or multiple seeds are reported, and the paper admits training instability.\n\nNone of these are fatal to the underlying method—the direction is worth pursuing and the architecture is reasonable—but the state-of-the-art claim cannot be verified as written.\n\nWho's this for: anyone working on log anomaly detection or temporal graph learning for event sequences. It deserves a serious referee, with the expectation of major revision. The authors should release code and configs, state the threshold, align the split descriptions, specify the baseline-to-event conversion rule, and report variance. If they do that, this could be a solid paper.\n\nRecommendation: send it to peer review. Don't desk reject—the core idea is novel and the execution is mostly careful, but the evaluation needs a thorough going-over before the results can be trusted.\n\nBest,","headline":"A genuinely novel idea for event-level log anomaly detection, but the evaluation protocol is under-specified enough that the headline F1 scores are currently unverifiable.","tokens_in":11183,"tokens_out":3438,"would_cite":false,"duration_ms":31311,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TempoLog claims that log anomaly detection can be done event-by-event on a continuous-time dynamic graph, removing fixed-size windows and reaching F1 scores of 0.986 on BGL, 0.993 on Spirit, and 0.987 on Thunderbird, beating seven…","keywords":["log anomaly detection","temporal graph networks","continuous-time dynamic graphs","event-level anomaly detection","window-free detection","link prediction","semantic log embeddings","multi-scale graphs"],"falsifier":"Re-run the seven baselines under the paper's own protocol with the same event-level labeling rule as TempoLog and the same 40% test split; if any baseline's per-event F1 on BGL, Spirit, or Thunderbird moves materially from Table 2, or if TempoLog is instead scored at window level, the 'significantly outperforming' claim collapses. The simplest check is to count the number of true positives the baselines would admit under an explicit localization rule.","tokens_in":10111,"feed_emoji":"🕸️","tokens_out":6616,"duration_ms":57248,"temperature":0.7,"pith_summary":"TempoLog is a log-anomaly detection framework that does away with the fixed-size windows used by prior methods, which the paper argues introduce context bias and make it hard to localize which event is anomalous. Instead, it builds a continuous-time dynamic graph whose nodes are log templates and whose edges are temporal dependencies at several hop distances, then trains a temporal graph network with semantic node memory to decide, event by event, whether the next link between templates is expected. The paper claims this yields state-of-the-art event-level F1 scores of 0.986, 0.993, and 0.987 on the BGL, Spirit, and Thunderbird datasets, beating seven baseline models in accuracy while keeping training and inference costs competitive. The payoff, if correct, is that engineers see the exact anomalous log event instead of a window to search through.","feed_headline":"Event-level log anomaly detection hits 0.993 F1 without windows","feed_subtitle":"A temporal graph network replaces fixed-size windows, flagging each anomalous log event and beating seven baselines on three large datasets.","key_machinery":"The central object is the multi-scale continuous-time dynamic graph (CTDG) built directly from the log stream: nodes are log-template semantic vectors, edges are temporal dependencies at hop distances $H=0$ and $H=1$ by default, and edge features encode semantic similarity, co-occurrence frequency, time interval, and log level. On top of this graph sits a temporal graph network with a memory module initialized with each template's BERT vector, a message function and aggregator that propagate the most recent neighbor memory, and a temporal graph attention embedding module; a link-prediction head trained with binary cross-entropy decides whether an edge should exist between two nodes at a given timestamp, and any mismatch flags the event as anomalous. Parameter sharing across hop scales keeps training and inference efficient, which is what makes the event-level prediction practical on large logs.","core_discovery":"On its own terms, the central claim is that log anomaly detection can and should be framed as link prediction on a continuous-time dynamic graph rather than as classification of fixed-size windows. TempoLog represents every log template as a node carrying a BERT-derived semantic vector, creates directed temporal edges between nodes that appear within a chosen hop distance, enriches those edges with features such as semantic similarity, co-occurrence frequency, time interval, and log level, and then uses a temporal graph network—with memory initialized from the semantic vectors and updated by message passing—to produce node embeddings at each timestamp. An event is flagged anomalous when the predicted existence of an edge between two nodes at that time disagrees with the real link. The paper reports that this event-level protocol achieves F1 scores of 0.986 on BGL, 0.993 on Spirit, and 0.987 on Thunderbird, outperforming seven window-based baselines, and that ablations show both the multi-scale (multi-hop) structure and the semantic memory contribute substantially to the gains.","pith_inferences":["Because the graph is defined purely by event order and timestamps, the same construction could in principle be applied to other discrete event streams—clickstreams, financial transactions, or sensor events—where fixed-window grouping also thresholds context.","The link-prediction criterion is a local decision per event, so TempoLog's event-level outputs could be aggregated into window-level or session-level alerts without re-training, which window-based methods cannot do in reverse.","A direct test of the framework's generality would be to sweep the hop distance $H$ beyond $\\{0,1\\}$ and measure how F1 changes; the paper does not report this sweep, so it remains open whether larger hops help or hurt on denser log streams.","The reported efficiency comparison covers training and inference time, but not end-to-end operational cost including memory growth from unseen templates, which the paper itself flags as a threat to validity."],"forward_implications":["A log-anomaly detector can operate at event granularity, so on-site engineers get the specific anomalous event rather than a window of logs to inspect.","Because no fixed window is used, the choice of window size can no longer distort detection accuracy the way the paper shows it does for DeepLog, LogAnomaly, PLELog, LogRobust, and CNN.","Semantic memory and multi-scale edges are load-bearing: removing semantic initialization drops F1 by roughly 26% on BGL, 75% on Spirit, and 48% on Thunderbird, and using only the 0-hop graph drops it by 15%, 63%, and 61%.","The same framework handles unseen log templates at test time by growing the memory, which supports dynamic log updates in practice.","The window-free design removes the need for the manual localization step that window-based methods impose on engineers."],"supporting_citations":[{"why":"Supplies the temporal graph network architecture—memory, message function, aggregator, and temporal graph attention—that TempoLog adapts with semantic initialization.","marker":"Rossi et al. [2020]"},{"why":"Provides the Drain log parsing method that converts raw log messages into the templates that become graph nodes.","marker":"He et al. [2017]"},{"why":"Provides the BERT-based semantic vector extraction used for node initialization and as a key baseline (LogBert).","marker":"Guo et al. [2021]"},{"why":"Documents the strong dependence of window-based methods on window size, which is the motivating evidence for the window-free design.","marker":"Le and Zhang [2022]"},{"why":"Represents the window-based sequence-model baseline (DeepLog) that TempoLog claims to surpass at event level.","marker":"Du et al. [2017]"},{"why":"Represents the window-based robustness-focused baseline (LogRobust) used in the comparison.","marker":"Zhang et al. [2019]"}],"fun_headline_variants":["Log anomaly detection as link prediction beats window methods","TempoLog: graph-centric log anomaly detection hits 0.993 F1","Graph links, not windows, for event-level log anomaly detection","Event-level log anomalies caught by temporal graph, no windows needed","Continuous-time graphs snag log anomalies at 0.993 F1, no fixed windows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority assumes the seven baselines were scored at the same event level and on the same train/test splits as TempoLog, but the paper never specifies how window-level baseline outputs were mapped to event-level labels and its reported splits are inconsistent (the caption says 50% training while Section 4.2 says 40% testing, and Spirit's listed train size is only 5% of its total).","fun_headline_variants_meta":{"raw":{"variants":["Log anomaly detection as link prediction beats window methods","TempoLog: graph-centric log anomaly detection hits 0.993 F1","Graph links, not windows, for event-level log anomaly detection","Event-level log anomalies caught by temporal graph, no windows needed","Continuous-time graphs snag log anomalies at 0.993 F1, no fixed windows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1334,"prompt_tokens":909,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":525,"tokens_out":425,"duration_ms":4575,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:26:51.182494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the seven baselines under the paper's own protocol with the same event-level labeling rule as TempoLog and the same 40% test split; if any baseline's per-event F1 on BGL, Spirit, or Thunderbird moves materially from Table 2, or if TempoLog is instead scored at window level, the 'significantly outperforming' claim collapses. The simplest check is to count the number of true positives the baselines would admit under an explicit localization rule.","supporting_citations":[{"cited_title":"Logbert: Log anomaly detection via bert","cited_arxiv_id":null,"evidence_quote":"Provides the BERT-based semantic vector extraction used for node initialization and as a key baseline (LogBert)."},{"cited_title":"Loggd: Detecting anomalies from system logs with graph neural networks","cited_arxiv_id":null,"evidence_quote":"Documents the strong dependence of window-based methods on window size, which is the motivating evidence for the window-free design."},{"cited_title":"Robust log-based anomaly de- tection on unstable log data","cited_arxiv_id":null,"evidence_quote":"Represents the window-based robustness-focused baseline (LogRobust) used in the comparison."}],"review_version":1}