{"id":"a60ab20a-9e07-4de6-823a-80707743e6ef","arxiv_id":"2508.09594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LLMLog uses edit-distance similarity, confidence-weighted annotation selection, and adaptive demonstration coverage to improve LLM-based log template generation.","lead":"A new framework helps large language models turn raw system logs into structured templates by picking the most useful logs to label and the right examples to show the model. It reports higher parsing accuracy on 16 log datasets at lower cost than previous LLM-based methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8) and the Appendix 6.2 proof define different objectives; the stated 1-1/e guarantee for Algorithm 1 is not established.","rationale":"The reader's weakest_assumption — that cosine>=0 word coverage transfers label information — is plausible, but it is not the most actionable defect: the paper's own parameter sensitivity in Section 4.4.4 reports robustness from threshold 0 to 1 on two datasets, which weakens the force of that concern for the tested cases. The clearest internal inconsistency is between Eq. (8) and Appendix 6.2. Eq. (8) puts the coverage term inside a sum over selected logs, making the greedy marginal depend on |L| and producing a set function g(L)=|L|·coverage(L) that is not submodular; the appendix's Lemma 1 proves submodularity for a different objective with coverage outside the sum. Algorithm 1 line 8 explicitly invokes Eq. (8), so the paper's approximation guarantee is either unproven for the stated objective or the objective is misprinted. This is load-bearing for the formal contribution, but it does not by itself invalidate the empirical comparisons, so the reader's CONDITIONAL verdict remains appropriate. A code-level check of the implemented objective would settle which version is real.","tokens_in":33204,"tokens_out":8497,"duration_ms":106585,"concrete_test":"Inspect the released code at https://github.com/XinTT/LLMLog and implement IS(Lr) exactly as written in Eq. (8). Compute the marginal ΔIS(s|Lr) and the greedy selections on a small dataset (e.g., Proxifier). Then recompute with the appendix objective F(L)=(1-λ)coverage(L)/|U| + λΣC(s_i). If the selected sets differ, the stated algorithm and proof refer to different problems. Independently verify submodularity for the written objective by evaluating F({a}), F({b}), F({a,b}) on logs with overlapping I-sets: if the marginal of the second element exceeds the marginal of the first, Eq. (8) is not submodular and Theorem 2 cannot apply. This settles whether the 1-1/e guarantee holds for Algorithm 1 as published.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (8) defines IS(Lr) = max Σ_{s_i∈Lr}[(1-λ)|∪_{i=0}^{|Lr|} I_{s_i}|/|U| + λC(s_i,...)], placing the coverage term inside the summation. Under this written objective, the marginal gain of adding s is λC(s) + (1-λ)((|L|+1)·coverage(L∪{s}) - |L|·coverage(L))/|U|, not the expression used in Lemma 1, which is λC(s) + (1-λ)·Δcoverage/|U|. The set function g(L)=|L|·coverage(L) is not submodular: adding a second element can give marginal larger than the singleton marginal, so monotone-submodularity and the 1-1/e bound in Theorem 2 do not follow for Eq. (8). Appendix 6.2 instead proves the bound for the different objective (1-λ)coverage(L)/|U| + λΣC(s_i), where coverage is taken once. Since Algorithm 1 line 8 says it computes Eq. (8) and selects by ΔIS derived from it, the paper either implements the written objective and the approximation guarantee is false, or it implements the appendix objective and the central formal statement is mis-specified. The greedy guarantee is a stated contribution ('a greedy algorithm with theoretical guarantees'), so this is load-bearing for the formal claims even if the empirical tables are unaffected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"LLMLog proposes an LLM-driven multi-round annotation framework for log template generation. It introduces a semantic edit-distance (SED) similarity, a representative score, an LLM prediction confidence score, a greedy annotation selection with claimed approximation guarantees, an adaptive demonstration-selection algorithm based on word coverage, and an adaptive per-round budget. The paper reports extensive experiments on 16 datasets with GPT-4o and Qwen2.5-7B-Instruct, showing that LLMLog matches or beats DivLog, AdaICL, Drain, and LogPPT on MLA, PTA, and RTA while reducing inference time and API cost. The main formal claims are a 1-1/e approximation for the annotation selection and a 1+ln(n) approximation for adaptive demonstration selection.","tokens_in":33559,"tokens_out":8171,"duration_ms":78030,"significance":"If the formal results are repaired, this would be a solid empirical contribution: the evaluation spans 16 datasets, two LLM backbones, three accuracy metrics, and cost/time measures, and a public artifact link is provided. The ablation study isolates the main components and shows large drops when SED, representativeness, confidence, adaptive demonstrations, or adaptive budget are removed. I also agree with the reader's circularity assessment: selection uses LLM probabilities and word consistency, not benchmark labels, so there is no obvious feedback loop of fitted constants. However, the theoretical guarantee for the annotation algorithm is not established as written: the objective in Eq. (8) and the proof in Appendix 6.2 are different set functions, and the proof itself omits a term in the marginal. One pseudocode loop is also non-terminating as written. These issues are load-bearing for the paper's stated contributions, though they appear fixable without changing the empirical core.","major_comments":[{"comment":"The formal guarantee of Algorithm 1 is not established. Equation (8) defines IS(L_r)=max \\sum_{s_i\\in L_r} [(1-\\lambda)|\\cup_{i=0}^{|L_r|} I_{s_i}|/|U| + \\lambda C(s_i,\\hat t_i,\\hat s_i)], i.e., the coverage ratio sits inside the summation. Under this objective the marginal gain of adding s is \\lambda C(s)+(1-\\lambda)((|L|+1)\\cdot cov(L\\cup{s}) - |L|\\cdot cov(L))/|U|, not the appendix's \\lambda C(s)+(1-\\lambda)\\Delta cov/|U|. The function |L|\\cdot cov(L) is not submodular, so the claimed 1-1/e ratio in Theorem 2 does not follow. Appendix 6.2 analyzes a different objective, (1-\\lambda)cov(L)/|U| + \\lambda\\sum C(s_i); moreover Eqs. (15)-(16) omit the -cov(L'_r) term in the marginal, so the submodularity proof is invalid for that objective too. Since Algorithm 1 line 8 states it computes Eq. (8), the approximation guarantee as stated is unsupported.","section":"3.3.3, Eq. (8); Appendix 6.2"},{"comment":"Algorithm 2's loop condition is 'while B >= 0'. Whenever the remaining budget B is smaller than the computed B_r, line 11 sets B_r = min(B_r,B)=B and then B = B - B_r = 0, but the loop does not exit. At the next iteration B_r = min(B_r,0)=0 and B remains 0, so the loop never terminates (and would repeatedly call AnnotationSelection with budget 0). The condition should be 'while B > 0', with an explicit break if B_r = 0. As written, the multi-round procedure is not a terminating algorithm.","section":"3.3.5, Algorithm 2"},{"comment":"The word-coverage condition in Problem 1 and Algorithm 3, cosine(w_i, w_k) >= 0, is extremely permissive. With typical text embeddings, most (even unrelated) word pairs have cosine similarity >= 0; then any labeled log can 'cover' almost any word, and the set-cover constraint is vacuous. The parameter-sensitivity experiment in Figure 4 reports no change over thresholds 0, 0.25, 0.5, 0.75, 1, which is more consistent with an inactive condition than with robustness. Please report the distribution of pairwise cosine similarities between words from different templates and the number of demonstrations selected as a function of the threshold. Without this, the mechanism by which adaptive demonstration selection improves accuracy is not established. The same threshold also appears in the SED definition, Eq. (3).","section":"3.4, Eq. (14); Figure 4"}],"minor_comments":[{"comment":"Definition 1 appears twice: once for the Log-Word Bipartite Graph in Section 3.2 and again for the LLM-driven Log Annotation Problem in Section 3.3.3. Renumber one of them.","section":"3.2 / 3.3.3"},{"comment":"The proof sketch says 'We prove Theorem 3 by reduction from the Max Coverage problem', but it should refer to Theorem 1.","section":"3.3.4, Theorem 1 proof sketch"},{"comment":"The union \\cup_{i=0}^{|L_r|} I_{s_i} is written inside the summation over s_i. If a global coverage term is intended, it should be placed outside the sum, and the index/domain should be clarified.","section":"Eq. (8)"},{"comment":"The sentence defining c(w1,w2) has a typo: 'if the word similarity between w1 and is greater than 0' is missing the second argument w2.","section":"Eq. (3)"},{"comment":"The approximation ratio 1+ln(n) needs n defined. Also, the standard set-cover bound is H(max set size) <= 1+ln n; please cite the precise form being used.","section":"Theorem 4 / Appendix 6.4"},{"comment":"The text says the effect of the cosine threshold 'has converged'. Please quantify what is shown: are all curves identical, or within some tolerance? Reporting the actual values would make the claim checkable.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is strong and the artifact link is a plus. The main obstacle is the formal approximation guarantee: as written, Eq. (8) and Appendix 6.2 define different objectives, and the proof has a missing marginal term. This is not a reason to reject the whole paper, because the fix is local: either correct Eq. (8) and the algorithm to match a submodular objective and re-prove the bound, or remove/qualify the 1-1/e claim. Please also fix the non-terminating loop in Algorithm 2. I would not raise circularity concerns; the selection loop is active learning, not result leakage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the system is sensible and the evaluation is credible, but the 1-1/e guarantee is not proven as written. Eq. (8) puts the coverage term inside the summation, so the objective is effectively |L| times the coverage fraction plus the confidence sum. The appendix proof instead treats coverage as a single global term. Those are different functions. The marginal gain in Lemma 1 does not match the written objective, and the set function g(L)=|L|·coverage(L) is not submodular. So Theorem 2 does not follow. This is a real formal gap, not a nitpick.\n\nWhat is actually good: the combination of SED-based similarity, confidence-weighted annotation, and adaptive demonstration coverage is a genuine improvement over DivLog, AdaICL, and IDEAL. The evaluation on 16 datasets with two LLM backbones is thorough, and the accuracy gains are consistent. The cost savings from adaptive demonstration selection are meaningful and well demonstrated. Code and data are linked, which helps verifiability.\n\nThe soft spots beyond the proof: no error bars or repeated-run statistics, and hyperparameters like alpha in Eq. (7) and the embedding checkpoint are not specified. Those are minor compared to the proof issue, but they would matter for reproduction.\n\nMy read: this is an applied paper for log parsing and active learning. The empirical core deserves a serious referee, but the theoretical claim needs repair. I would send it out, asking the authors to either fix Eq. (8) to match the appendix objective or remove the approximation guarantee and present the greedy algorithm as a heuristic. The empirical results may well stand either way.","headline":"Solid empirical system with a broken theoretical guarantee as stated; worth reviewing but needs a fix.","tokens_in":34037,"tokens_out":2645,"would_cite":true,"duration_ms":27997,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Log-template AI hits perfect scores on 11 of 16 datasets","keywords":["log template generation","LLM in-context learning","multi-round annotation","semantic edit distance","adaptive demonstration selection","log parsing","active learning","cost-efficient LLM prompting"],"falsifier":"Construct or find a set of logs where every word of an unlabeled log has cosine similarity $\\geq 0$ to words in a labeled log that actually carries a different template label; if the LLM then copies the wrong template instead of the correct one, the coverage condition is not sufficient to guarantee accuracy.","tokens_in":33089,"feed_emoji":"🧩","tokens_out":4517,"duration_ms":44248,"temperature":0.7,"pith_summary":"This paper is trying to establish that log template generation—turning raw system logs like \"2024-11-14 192.168.1.1 GET /index.html 200\" into structured templates like \"[DATE] [IP] <GET> [RESOURCE] [STATUS]\"—can be automated more accurately and cheaply than current heuristic, neural, and LLM baselines by letting the LLM choose its own examples. The proposed framework, LLMLog, runs in rounds: it measures log similarity with a semantic edit distance that emphasizes keywords rather than long timestamps, it spends each round's human-annotation budget on logs that are both representative and hard for the LLM, and it feeds each unlabeled log a minimal set of fully labeled demonstration logs that cover every word in the input. Experiments on 16 LogPAI datasets report LLMLog reaching or beating all baselines on message-level and template-level accuracy, often at 100 percent, while using fewer tokens and lower API cost. If the claim holds, it means accurate templates can be produced with much less human labeling and cheaper LLM prompting than the previous state of the art.","feed_headline":"Log-template AI hits perfect scores on 11 of 16 datasets","feed_subtitle":"Multi-round annotation plus adaptive examples cut cost while beating prior log parsers.","key_machinery":"Semantic Edit Distance (SED): a dynamic-programming edit distance on the remaining words of two logs (after removing words already recognized by the labeled set), with replacement cost 0 when two word embeddings have cosine similarity $\\geq 0$ and 1 otherwise. It carries the argument because it defines both the representative score (which logs cover many peers) and the coverage objective for demonstration selection; two submodular greedy algorithms with approximation guarantees run on top of it.","core_discovery":"The paper claims that log template generation is best done by an iterative loop in which (1) unlabeled logs are compared by a semantic edit distance (SED) that removes words already identified by the current labeled set, (2) each round's annotation budget is spent on logs that are simultaneously representative and hard for the LLM, and (3) each input log receives a minimal set of demonstrations that covers all of its words with embeddings having non-negative cosine similarity. On the 16 LogPAI datasets, this design is reported to yield higher MLA, PTA, and RTA than DivLog, AdaICL, Drain, and LogPPT on every dataset, reaching 100% on a majority of them, while using fewer tokens and lower API","pith_inferences":["The coverage condition $\\cos(w_i, w_j) \\geq 0$ is very permissive for typical embedding spaces, so the real driver of the gains may be the greedy minimum-covering selection rather than the semantic threshold; a control experiment using random word embeddings would separate the two.","If the mechanism holds, adaptive demonstration selection should transfer to other token-level sequence annotation tasks (e.g., column type annotation, NER-style slot filling), where fixed $k$-shot contexts are also standard.","The adaptive budget rule (shrink budget as identified words grow) creates a self-referential loop: small early rounds may under-cover words and inflate later budgets, so stability depends on the hand-set first two rounds' budgets more than the paper's sensitivity analysis shows."],"forward_implications":["Log template generation can run with far fewer human annotations because each round targets the most representative and most uncertain logs.","Accuracy becomes less dependent on the choice of $k$ in $k$-shot prompting and on embedding-based similarity, since demonstrations are chosen adaptively per log.","The same framework lowers API cost and inference time because each unlabeled log gets only as many demonstrations as needed to cover its words.","The performance ceiling is tied to the annotation budget; accuracy stabilizes once the budget covers the distinct words, making tuning predictable.","The reported gains appear on both GPT-4o and a 7B open model, suggesting the mechanism is not tied to one LLM's strength."],"supporting_citations":[{"why":"DivLog is the LLM-based state-of-the-art baseline that LLMLog must beat, and its DPP-initialized labeled set and fixed top-k demonstrations are the main comparison point.","marker":"[74]"},{"why":"AdaICL is the multi-round annotation baseline that LLMLog extends with confidence-aware selection and adaptive demonstrations.","marker":"[45]"},{"why":"LogPAI supplies the 16 benchmark datasets on which all experiments are run.","marker":"[87]"},{"why":"Drain is the heuristic baseline whose handcrafted rules motivate the need for LLM-based methods.","marker":"[23]"},{"why":"LogPPT is the neural-network baseline requiring labeled training data, representing the supervised approach the framework avoids.","marker":"[33]"},{"why":"Determinantal Point Process is used to initialize the labeled log set with diverse logs, following DivLog.","marker":"[10]"},{"why":"LLM2Vec provides the word embeddings used in SED's replacement cost and in the coverage constraint.","marker":"[5]"},{"why":"Adaptive submodularity theory supplies the approximation guarantee for the greedy demonstration-selection algorithm.","marker":"[21]"},{"why":"The max coverage problem is the reduction source proving the annotation selection problem NP-hard.","marker":"[32]"},{"why":"The set cover problem is the reduction source proving the adaptive demonstration selection problem NP-hard.","marker":"[13]"}],"fun_headline_variants":["LLMLog beats every log parser on all 16 datasets","Multi-round LLM annotation yields perfect templates on most logs","LLMLog: smarter, cheaper log templates that set new SOTA","Adaptive LLM examples make log templating flawless","Edit-distance driven LLM context conquers log templates"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The method's gains rest on the assumption that any two words with non-negative embedding cosine similarity carry the same template label, so covering each input word with such a labeled word transfers the correct label; if that transfer fails, the selected demonstrations and annotations need not help.","fun_headline_variants_meta":{"raw":{"variants":["LLMLog beats every log parser on all 16 datasets","Multi-round LLM annotation yields perfect templates on most logs","LLMLog: smarter, cheaper log templates that set new SOTA","Adaptive LLM examples make log templating flawless","Edit-distance driven LLM context conquers log templates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000353,"raw_usage":{"total_tokens":1777,"prompt_tokens":779,"completion_tokens":998,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":915}},"tokens_in":523,"tokens_out":998,"duration_ms":10431,"temperature":1.0,"reasoning_tokens":915,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:58:11.580834+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or find a set of logs where every word of an unlabeled log has cosine similarity $\\geq 0$ to words in a labeled log that actually carries a different template label; if the LLM then copies the wrong template instead of the correct one, the coverage condition is not sufficient to guarantee accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LogPAI supplies the 16 benchmark datasets on which all experiments are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adaptive submodularity theory supplies the approximation guarantee for the greedy demonstration-selection algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The max coverage problem is the reduction source proving the annotation selection problem NP-hard."}],"review_version":1}