{"id":"9628d4b0-c742-4cc6-baa3-8a931e538cf0","arxiv_id":"2411.15354","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"UTLParser merges logs from multiple sources into one causal graph using semantic analysis and delay-tolerant time queries.","lead":"This paper presents UTLParser, a tool that turns logs from several computer systems into a single causal graph of actions and connections, such as who contacted which address and when. A security analyst could use this graph to trace how an attack unfolded across different services.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I rests on hand-corrected logs after the parser's first run, so the central outperformance claim is not independently established; the causal-graph half of the claim has no quantitative evaluation.","rationale":"The reader's verdict identifies the hand-crafted correction as a soundness and circularity problem, and I agree that this is the most load-bearing empirical issue. The paper's own text admits the correction step after the initial parsing trial, which directly undermines the only benchmark where UTLParser is claimed to outperform all baselines. I differ slightly from the reader's 'weakest_assumption' field, which focuses on the dependency-parsing assumption in Section III-D; that assumption is relevant to the causal-graph half of the paper, but the more immediate threat to the central claim is that Table I is not an independent measurement. Even if the dependency assumption were satisfied, the evaluation still would not validate causal graph correctness. The paper does provide source code and a data link, which is positive for reproducibility, but no commit hash or exact command pipeline is given, so the correction step cannot currently be audited. My proposed test directly checks whether the reported parsing results survive on the original, uncorrected data. If they do not, the rejection is warranted. If they do, the parsing claim would be credible, but the causal-graph/attribution claim would still require a separate graph-level evaluation, which is currently entirely absent. This reasoning supports maintaining the reader's REJECT verdict without change.","tokens_in":6977,"tokens_out":3844,"duration_ms":39767,"concrete_test":"Using the GitHub/Zenodo repository linked in reference [5], recover the exact raw AIT dataset v2.0 files corresponding to the 748 auth, 2000 audit, 2000 dns, and 1045 syslog samples. Re-run the full evaluation pipeline from [20] on the unmodified raw logs, without applying the 'hand-crafted correction' described in Section V, and recompute Table I for Logram, NuLog, ULP, Brain, and UTLParser. If the corrected and uncorrected inputs or labels differ, report both sets of numbers; if UTLParser no longer exceeds all baselines on every metric, the central outperformance claim fails. Independently, inspect the git history of the repository for the commit that introduced the correction to determine whether the change affected input logs or ground-truth labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The only quantitative support for the headline claim is Section V.A, Table I, where UTLParser reports near-perfect parsing accuracy and F1 (0.998/0.9984). The evaluation section states: 'The process involves hand-crafted correction after initial parsing trail.' This means the input logs, or the ground-truth labels derived from them, were modified after an initial run of UTLParser. Since all four baselines are then scored against this parser-informed reference, the comparison is not an independent benchmark. UTLParser is being judged on data that was adjusted in response to its own output, making the 'outperforms all other methods' claim circular rather than demonstrated. The absence of error bars and a single run further weakens the comparison. Separately, even if Table I were fully trustworthy, it measures only token-level template/parameter matching. Section V reports no metric for the fused causal graph, entity-triple correctness, or attack attribution that the title and abstract promise. Section III-E's graph fusion is a set union with last-writer-wins attributes, and Section V.B reports only processing time for that stage. Thus the central claim has a double gap: the one strong empirical result rests on a potentially parser-dependent correction step, and the causal-graph/attack-attribution capability has no experimental validation at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes UTLParser, a framework for converting multi-source logs into fused causal graphs. It classifies logs as general, key-value, or request logs, applies specialized parsing, extracts Points of Interest, uses spaCy dependency parsing and SemgrexPattern to obtain entity triples (subject, action, object), constructs per-source causal subgraphs, fuses them by union with last-write-wins attribute updates, and provides a temporal graph query interface with a delay tolerance. The evaluation in Section V reports parsing accuracy and F1 on auth, audit, dns, and syslog samples from the AIT dataset, plus processing times for larger logs. The paper claims that UTLParser outperforms four existing log parsers on every dataset and that it precisely extracts causal threat information for attack attribution.","tokens_in":7101,"tokens_out":8181,"duration_ms":74219,"significance":"If the claims were established, UTLParser would be a useful contribution: a single tool that handles several log formats and outputs temporal causal graphs would address a real gap, and the public code release is commendable for reproducibility. However, the significance is currently not demonstrated. The core parsing comparison is undermined by a hand-corrected evaluation sample, no semantic/triple-based baselines are included, no metrics are reported for entity-triple or causal-graph correctness, and the attack-attribution promise is entirely unquantified. The strengths are the clear architecture, the public artifact, and the use of real AIT/IoT-32 data, but these do not compensate for the missing validation of the central claims.","major_comments":[{"comment":"The sentence 'The process involves hand-crafted correction after initial parsing trail' indicates that the evaluation logs or their ground-truth labels were modified after an initial run of UTLParser. Because all baselines in Table I are then scored against this parser-informed reference, the claim in Section V.A that 'UTLParser outperforms all other methods across all datasets' is circular and cannot be accepted as an independent benchmark. Please re-evaluate on unmodified logs with pre-registered ground truth, describe any corrections before evaluation, and report multiple runs with error bars. In addition, Table I reports identical F1 values across all four datasets for each baseline (e.g., ULP 0.9750), which needs explanation.","section":"V.A, Table I"},{"comment":"There is no quantitative evaluation of the causal graph construction or attack attribution, although these are central to the title and abstract. Table II reports only running times; the conclusion's statement of a 'low missing rate of critical IOCs information after transformation' is not supported by any reported metric. The authors should measure entity-triple precision/recall against ground truth, evaluate fused-graph edge correctness on known attack scenarios, and apply a downstream attribution task (or provide a clear proxy) to substantiate the claimed capability.","section":"V, Tables I-II"},{"comment":"The dependency-extraction step rests on an explicit assumption that entity triples consist of a verb action and noun subject/object tokens and that these tokens align with predefined IOCs. The manuscript provides no evaluation of how often spaCy dependency parsing yields correct triples from fragmented log entries, and no comparison with semantic parsers such as SemParser or CyberEntRel that are discussed in Section II. If subject/object identification is effectively IOC matching, the semantic-generalization claim is untested. Please report dependency-parsing and triple-extraction accuracy separately.","section":"III-D"},{"comment":"The fusion operation is a set union with last-write-wins attribute overwriting, and the paper does not justify that this preserves the causal semantics needed for downstream analysis. The delay-tolerance selection in Algorithm 1 uses undefined quantities `inte score` and `inde score`, and line 9 appears to compute the score from the full graph G rather than the extracted temporal graph GT_t. No experiments validate the fused graph's correctness or the sensitivity to the delay tolerance. At minimum, these definitions and a correctness analysis or ablation should be provided.","section":"III-E, III-F, Algorithm 1"},{"comment":"The comparative evaluation uses only general-purpose template/parameter parsers (Logram, NuLog, ULP, Brain). None of the semantic relation extraction systems introduced in Section II are included as baselines, so the evaluation does not test the semantic parsing or causal-graph contribution that distinguishes UTLParser. The comparison should be extended to semantic extractors, or the claims should be narrowed accordingly.","section":"V, Table I"}],"minor_comments":[{"comment":"Section V.A says 'hand-crafted correction after initial parsing trail'; 'trail' should be 'trial', and the sentence needs to specify exactly what was corrected (logs, labels, or parser output).","section":"V.A"},{"comment":"Table I would benefit from error bars, the number of runs, and a clearer separation between accuracy and F1 columns; the repeated F1 values for each baseline across datasets are difficult to interpret.","section":"Table I"},{"comment":"The score functions `inte score` and `inde score` are not defined anywhere in the paper, and the relationship between `G` and `GT_t` in line 9 is unclear.","section":"III-F, Algorithm 1"},{"comment":"The citation [12] for SemgrexPattern appears to be the wrong reference; the cited CIKM 2011 paper is about regular expression inference, not Semgrex. Please add the correct Semgrex citation.","section":"III-D"},{"comment":"Section VI acknowledges that manual IOC/POI configuration is necessary; this should be reconciled with the claim of a generic/extensible framework, and the manual configuration effort should be quantified.","section":"VI"},{"comment":"In Section V.A, 'across four art algorithms' appears to be a typo for 'state-of-the-art algorithms'.","section":"V.A"}],"recommendation":"reject","confidential_remarks":"The paper would be better positioned as a tool/system description. The provided public code and dataset are useful, but the central empirical claims need to be re-established with an independent benchmark and with metrics for the causal graph. If the authors can do that, a substantially revised version could be considered. I do not see a need for further external review of the current version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nUTLParser is a pipeline that turns heterogeneous logs into a fused causal graph: it classifies log type, parses general/key-value/request formats, extracts POIs, runs spaCy dependency parsing to get subject-action-object triples, builds sub-graphs, fuses them, and supports temporal queries with a delay tolerance parameter. The genuinely new piece is the combination of multi-source fusion with delay-tolerant querying; each component borrows from prior work (Drain, regex, spaCy, Semgrex). That's a legitimate engineering contribution, and the modular design is sensible. The authors also promise code and data on Zenodo, which is good.\n\nThe soft spots are mostly in the evaluation, and they're serious. Table I reports near-perfect parsing accuracy/F1 across auth, audit, dns, syslog, but Section V.A says the process involves 'hand-crafted correction after initial parsing trail.' That means the reference or the logs were adjusted after seeing UTLParser's first output. Whatever the details, the comparison against Logram, NuLog, ULP, and Brain is no longer an independent benchmark. There are also no error bars, no repeated runs, and no description of how baselines were configured. Separately, the paper's title and abstract promise causal graph construction and attack attribution, but there is no quantitative evaluation of graph correctness, entity-triple validity, or attribution quality—only processing times. The graph fusion is a set union with last-writer-wins attributes, which is fine as plumbing but doesn't by itself establish causality.\n\nThe assumption in Section III-D—that actions are verbs and subjects/objects are nouns aligned with IOCs—is plausible but untested on fragmented log entries. Algorithm 1 references functions inte_score and inde_score that are never defined, and Table II is malformed. These are fixable but need attention.\n\nThe core idea is worth pursuing, and the paper is not incoherent. But as submitted, the load-bearing outperformance claim is not independently demonstrated, and the causal-graph half is a claim without measurement. I'd send it to a software/security venue with a serious referee and artifact evaluation, expecting heavy revision: re-run the log parsing benchmark on untouched ground truth, add variance estimates, and report at least one metric on the fused graph or downstream attribution. For a reading group, it's a useful example of how evaluation choices can undercut a good systems story.\n\nBest.","headline":"UTLParser is a plausible multi-source log fusion framework, but the paper's only strong empirical claim rests on parser-informed hand-corrected reference logs, and the causal-graph half is unevaluated.","tokens_in":7742,"tokens_out":2571,"would_cite":false,"duration_ms":24943,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single semantic parser aims to turn multi-source logs into causal attack graphs.","keywords":["Semantic Log Parsing","Multi-Source Logs Fusion","Temporal Causal Graph","Points of Interest","Attack Attribution","Dependency Parsing","Graph Fusion"],"falsifier":"Run UTLParser on a labeled multi-source log set with known ground-truth attack steps and count how many fused graph edges match true causal relations. If parsing accuracy and F1 remain near the reported values while edge-level precision is only slightly above chance, the causal-graph claim fails even though the parsing claim stands.","tokens_in":6631,"feed_emoji":"🔎","tokens_out":6047,"duration_ms":50580,"temperature":0.7,"pith_summary":"The paper proposes UTLParser, a unified framework that parses labeled logs from several formats and fuses the results into temporal causal graphs for attack attribution. Its aim is to overcome two gaps: existing parsers extract templates and parameters without semantic context, and semantic entity-extraction approaches usually handle one log type. UTLParser instead combines log-type-specific parsing with Points of Interest extraction, dependency-based subject-action-object triple detection, and subgraph fusion, so one event that writes multiple log types can be reconstructed as one graph. On auth, audit, dns, and syslog samples, the paper reports average parsing accuracy of 0.9826 and an F1 score of 0.9984, claiming it outperforms all compared baselines. The practical payoff would be cross-log threat attribution that tolerates timestamp skew between sources.","feed_headline":"UTLParser unifies four log types at 98% accuracy","feed_subtitle":"The same pipeline that reaches 0.9984 F1 builds causal graphs from auth, audit, dns, and syslog entries.","key_machinery":"The load-bearing object is the eight-step pipeline: log-type detection dispatches to one of three parsers—a Drain-style tree parser for general logs, a key-value parser, and a regex-based request parser—then Points of Interest extraction normalizes fields into unified columns such as Time, Src_IP, Dest_IP, and IOCs. Semantic dependency analysis via spaCy identifies verb tokens as actions and noun tokens as subjects/objects, and SemgrexPattern, a regular-expression language over dependency graphs, infers the relationships that become causal edges. Graph fusion merges subgraphs with last-write-wins attributes, and temporal graph querying uses a delay tolerance $\\Delta t$ chosen by scoring graph integrity against graph independence. The machinery works because Points of Interest define where Indicators of Compromise are located, so the dependency triples inherit threat-relevant meaning.","core_discovery":"UTLParser converts a labeled multi-source log collection into a directed multigraph by parsing each log according to its category, mapping extracted fields to unified Points of Interest columns, running semantic dependency analysis to find action/subject/object triples, and fusing all subgraphs while preserving node and edge attributes. The paper's central empirical claim, stated in the performance evaluation, is that UTLParser outperforms every compared parser on all four data types, with average parsing accuracy of 0.9826 and an average F1 score of 0.9984. The claim is specifically about token-level template and parameter parsing; the paper does not report a quantitative metric for whether the resulting graph edges match ground-truth causal attack relations.","pith_inferences":["A natural next benchmark is edge-level causal precision: compare fused graph edges against manually labeled attack steps, something the paper does not report.","The delay-tolerance scoring could transfer to any temporal graph extraction setting, since integrity versus independence is a general analyst trade-off.","If dependency parsing degrades on fragmented or non-English log dialects, token-level accuracy could stay high while causal edges become unreliable; stress tests on abbreviated logs would map this boundary."],"forward_implications":["One event that triggers multiple log sources can be reassembled into a single fused causal subgraph, making cross-log attack steps visible.","Temporal queries with optimized delay tolerance include related timestamp-skewed records without merging independent events.","The fused directed multigraph carries timestamps, IOCs, and Points of Interest as attributes, so downstream pattern-, scoring-, or temporal-snapshot-based anomaly detectors can consume it directly.","Because the parser handles general, key-value, and request logs, the same framework applies to network traffic and OS/auth logs without per-attack retraining."],"supporting_citations":[{"why":"Supplies the labeled AIT semi-structured log data and the ground-truth labelling rules from which Points of Interest and Indicators of Compromise are defined.","marker":"[13]"},{"why":"Supplies the labeled IoT-23 structured network-traffic logs used as the second data source.","marker":"[14]"},{"why":"Drain's fixed-depth tree parser is the base implementation that the general log parser extends.","marker":"[15]"},{"why":"SemgrexPattern is the dependency-pattern language used to infer relationships between entity tokens.","marker":"[12]"},{"why":"Defines parsing accuracy and F1 metrics that the experimental evaluation follows.","marker":"[20]"},{"why":"Logram is a baseline parser in the comparison; its scores anchor the claimed improvement.","marker":"[1]"},{"why":"Brain is a baseline parser in the comparison; UTLParser's scores are measured against it.","marker":"[2]"},{"why":"NuLog is a self-supervised baseline in the comparison table.","marker":"[18]"},{"why":"ULP is the large-log parsing baseline in the comparison table.","marker":"[19]"}],"fun_headline_variants":["UTLParser builds causal graphs from four log types","Semantic parser turns multi-source logs into causal graphs","UTLParser unifies auth, audit, dns, syslog at 98% accuracy","From raw logs to attack graphs: UTLParser"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The graph's causality is only as sound as the assumption that a dependency parser can extract verb/noun subject-action-object triples from short, fragmented log lines and that those tokens align with predefined IOCs.","fun_headline_variants_meta":{"raw":{"variants":["UTLParser builds causal graphs from four log types","Semantic parser turns multi-source logs into causal graphs","UTLParser unifies auth, audit, dns, syslog at 98% accuracy","From raw logs to attack graphs: UTLParser"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000309,"raw_usage":{"total_tokens":1723,"prompt_tokens":860,"completion_tokens":863,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":791}},"tokens_in":476,"tokens_out":863,"duration_ms":7390,"temperature":1.0,"reasoning_tokens":791,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:23:29.588407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run UTLParser on a labeled multi-source log set with known ground-truth attack steps and count how many fused graph edges match true causal relations. If parsing accuracy and F1 remain near the reported values while edge-level precision is only slightly above chance, the causal-graph claim fails even though the parsing claim stands.","supporting_citations":[{"cited_title":"An effective approach for parsing large log files,","cited_arxiv_id":null,"evidence_quote":"ULP is the large-log parsing baseline in the comparison table."},{"cited_title":"Ait log data set v2.0,","cited_arxiv_id":null,"evidence_quote":"Supplies the labeled AIT semi-structured log data and the ground-truth labelling rules from which Points of Interest and Indicators of Compromise are defined."},{"cited_title":"IoT-23: A labeled dataset with malicious and benign IoT network traffic,","cited_arxiv_id":null,"evidence_quote":"Supplies the labeled IoT-23 structured network-traffic logs used as the second data source."},{"cited_title":"Drain: An online log parsing approach with fixed depth tree,","cited_arxiv_id":null,"evidence_quote":"Drain's fixed-depth tree parser is the base implementation that the general log parser extends."},{"cited_title":"En- abling information extraction by inference of regular expres- sions from sample entities,","cited_arxiv_id":null,"evidence_quote":"SemgrexPattern is the dependency-pattern language used to infer relationships between entity tokens."},{"cited_title":"Tools and benchmarks for automated log parsing,","cited_arxiv_id":null,"evidence_quote":"Defines parsing accuracy and F1 metrics that the experimental evaluation follows."},{"cited_title":"Logram: Efficient log parsing using nn-gram dictionaries,","cited_arxiv_id":null,"evidence_quote":"Logram is a baseline parser in the comparison; its scores anchor the claimed improvement."},{"cited_title":"Self-supervised log parsing,","cited_arxiv_id":null,"evidence_quote":"NuLog is a self-supervised baseline in the comparison table."}],"review_version":1}