{"id":"9744a19c-ab31-465d-a50d-4a49a0b11e27","arxiv_id":"2412.01066","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Practitioners want log anomaly detectors that are accurate, fast, explainable, and customizable, but current research rarely delivers on the last three.","lead":"This paper surveys 312 software practitioners and interviews 15 professionals to learn what they expect from automated log anomaly detection tools. It then compares those expectations with 36 recent research papers, finding gaps in interpretability, customizability, and privacy protection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'only one / none' gap counts in Table I rest on un-audited binary coding of 36 papers; reproducibility of the central claim is the weakest link.","rationale":"I focus on the literature-review coding rather than sampling because the strongest claim is a comparison claim: even granting the survey's external validity, the gap size ('only one', 'none') is determined by how the 36 papers are coded. The survey representativeness concern affects generalizability but not the internal comparison; the coding concern affects the headline numbers themselves. The 51% reuse issue already flagged by the reader is real but not part of the strongest claim as stated; it supports the CONDITIONAL verdict rather than overturning the gap. The proposed re-coding test settles whether the gap is as large as claimed. If the re-coding reproduces Table I with high agreement, the central claim is robust; if not, the paper needs to soften the quantitative claims. The paper otherwise shows good-faith methodological effort, including a pilot survey, snowballing, and reporting interview agreement, so the appropriate posture remains conditional acceptance with a reproducibility requirement for the literature-review coding.","tokens_in":33379,"tokens_out":4949,"duration_ms":50243,"concrete_test":"Recruit two independent raters with log anomaly detection expertise, blind them to survey results, and have them re-code all 36 papers against an operational rubric derived verbatim from the survey definitions: customization = user-adjustable thresholds, algorithms, or configurations; interpretability = a mechanism (intrinsic or post-hoc) that explains why a specific log/sequence is anomalous; privacy = any stated measure for protecting sensitive log data. Report Cohen's kappa, a per-paper evidence table with quotes or figures, and a pre-registered rule for threshold-tunable detectors. If kappa is below substantial, or if threshold-tunable detectors qualify as customization, Table I's 'none' entries and the abstract's gap claim must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that research underdelivers rests on Table I's binary judgments: exactly one paper [33] supports interpretability, and zero support customization or privacy. These counts are produced by two authors reading each paper (Section II-C) with no reported inter-rater agreement and no per-paper evidence trail, unlike the interview coding which reports Cohen's kappa = 0.72 (Section II-A). The 'none' entries are especially fragile because the survey's customization definition explicitly includes 'adapting thresholds for defining anomalies' (Section II-B1), and many log anomaly detection techniques decide by thresholding a score; the paper does not state why threshold tunability in, e.g., threshold-based detectors was not counted. Interpretability is similarly a judgment call: attention-based and Transformer-based methods in the corpus could be argued to produce feature-attribution rationales, yet the paper marks only Zhao et al. [33] as interpretable, without showing that the other 35 papers lack any explanation mechanism or that [33]'s report was validated with users. Because the quantitative headline ('only one of 36', 'none') is the core evidence for the gap, the coding must be auditable. The lack of a rubric, example codings, or disagreement resolution for Stage 3 is a missing support that directly threatens the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a mixed-methods empirical study: 15 semi-structured interviews and a survey of 312 practitioners from 36 countries, followed by a literature review of 36 log anomaly detection papers from 2014–2024. The authors compare practitioners' expectations (granularity, effectiveness, efficiency, scalability, interpretability, generalizability, customization, privacy) with the capabilities of the reviewed papers, concluding that current research largely meets accuracy and scalability expectations but underdelivers on interpretability, generalizability, customization, and privacy. The main contribution is a descriptive map of practitioner expectations and a gap analysis against recent research.","tokens_in":33556,"tokens_out":6415,"duration_ms":56382,"significance":"If the RQ4 comparison is made auditable, the paper provides a useful and timely resource for the log analysis community: it quantifies practitioner thresholds (e.g., 95.5% view automated tools as essential or worthwhile, 70.5% prefer sequence-level analysis, 84.5% demand interpretability) and highlights under-addressed design dimensions. The survey design is carefully reported: pilot testing, interview coding with Cohen's kappa = 0.72, a publicly accessible survey instrument, explicit inclusion criteria, and snowballing for the literature review. The practitioner-expectation findings are likely to influence future tool and technique design. However, the literature-review comparison in RQ4 has methodological gaps—notably the lack of an auditable coding rubric—that must be addressed before the central gap claims can be fully accepted.","major_comments":[{"comment":"The central claim that only one of 36 papers supports interpretability and none support customization or privacy rests on binary capability coding by two authors with no reported inter-rater reliability, no coding rubric, and no per-paper evidence trail, in contrast to the interview coding which reports Cohen's kappa = 0.72. Because the survey defines customization to include 'adapting thresholds for defining anomalies' (Section II-B1), the conclusion that zero papers support customization is not self-evident for threshold-based detectors and requires an explicit coding justification. The authors should provide an appendix or online repository with the coding rubric, example codings, and a per-paper justification list, and report agreement (e.g., Cohen's kappa) for the Stage 3 coding.","section":"Section II-C, Table I, Finding 10"},{"comment":"The statement that 'around half of the surveyed practitioners (51%) do not prioritize these techniques' reuses the 51% compatibility-issue statistic from Figure 4, which was reported as an issue with existing log monitoring tools, not as a reason to deprioritize research techniques. This reuses a different quantity and does not support the sentence. Please either report the actual percentage derived from the free-text concerns (categorized as 26X, 19X, and 12X) or rephrase the sentence to refer directly to the survey question about willingness to use deep-learning-based techniques.","section":"Section III-D, 'Challenges of adopting existing log anomaly detection techniques'"},{"comment":"Scalability satisfaction is inferred from the size of the evaluation dataset (e.g., '35 papers ... evaluated ... on a public dataset containing more than 1,000,000 logs'), but the survey defines scalability as the capacity to process a specified number of logs while maintaining accurate and timely detection (Section II-B1). Dataset size is not a measurement of a technique's throughput or capacity; a paper may evaluate on HDFS without reporting end-to-end processing time or memory behavior at that scale. The comparison should be re-framed as 'evaluated on datasets with at least 1M logs,' or the authors should identify actual scalability measurements in the reviewed papers.","section":"Section III-D, Scalability"},{"comment":"Finding 9 states that 89% of studies rely on historical labeled normal log data and contrasts this with 'surveyed data availability (39.3%)'. However, 39.3% is the proportion of practitioners for whom such data are 'always' available; adding the 'sometimes' responses gives roughly 80% (Section III-A, Figure 3). The paper itself uses the 'sometimes or always' interpretation in Finding 3. Using only the 'always' figure in RQ4 makes the gap appear larger than the survey supports. Please use a consistent availability definition or justify why 'always' is the relevant threshold for training-data availability.","section":"Section III-D, Data Resource and Finding 9"}],"minor_comments":[{"comment":"There is a typo in 'log aonmlay detection' (should be 'log anomaly detection') in the paragraph introducing RQ3; please correct.","section":"Section III-C, first paragraph"},{"comment":"Finding 6 states that 'more than 78% of surveyed practitioners consider using automated log anomaly detection tools if they can be customized to process logs with different structures and provide a rationale for the detected anomalies,' but the underlying Figure 7 reports 78.2% for customization, 83.8% for generalizability, and 84.5% for interpretability as separate factors. The 'more than 78%' applies only to customization alone, not to the conjunction, so the finding as worded is misleading.","section":"Section III-C, Finding 6"},{"comment":"The search protocol would be easier to reproduce if the authors reported the date of the search, the number of hits per database, and a flow diagram or table showing how the 88 initial papers were reduced to 24 and then to the final 36; the current textual description is clear but lacks these quantitative search details.","section":"Section II-C, Literature search"},{"comment":"For papers [33], [50], and [47], the table marks recall/precision as '?' but footnote 1 says their F1 values are 'likely to be in the 80-100% range.' This speculation conflicts with the 'unknown' classification; either remove the 'likely' phrase or place those papers in the 80-100% bin.","section":"Table I, Effectiveness footnotes"},{"comment":"The title and several conclusions speak broadly of 'practitioners' expectations,' but the sample is a self-selected convenience sample (professional networks plus GitHub contributors) as acknowledged in Section IV-B. Adding a consistent hedge such as 'surveyed practitioners' in the title and key conclusions would more accurately reflect the external-validity limits the authors themselves state.","section":"Title and Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical survey that fits TSE's scope, and the practitioner-side results are valuable. The main obstacle is the auditability of the Stage 3 literature coding, which supports the paper's headline RQ4 claim; the other major issues (the reused 51% statistic, the scalability conflation, and the inconsistent data-availability definition) are fixable with careful re-analysis. I see no grounds for rejection if the authors can provide the coding artifact and correct these points. The paper would also benefit from a slightly more cautious framing of external validity in the title and abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I've read the paper. The honest one-liner: this is the first systematic attempt I know to ask practitioners what they actually want from log anomaly detection, and the headline – research overdelivers on raw recall/precision but underdelivers on interpretability, customization, privacy, and cross-system generalization – looks credible despite some wobbly supporting numbers.\n\nWhat's genuinely new: the survey of 312 practitioners yields concrete adoption thresholds (recall/precision above 60%, detection within 5 seconds, setup under 1 hour, handling 100k+ logs) that tool builders can use as target specs. The mixed-methods design is standard but executed carefully: interviews with 15 professionals, pilot survey, two language versions, Cohen's kappa 0.72 on interview coding, and the survey instrument is posted on Figshare. That is reproducible evidence, and the authors deserve credit for it.\n\nThe soft spots are real but not fatal. The 51% claim in Section III-D is sloppy: the same number is reused from the compatibility statistic in Figure 4 and applied to \"do not prioritize these techniques,\" which the free-text tallies (26X / 19X / 12X) do not support. The authors need to fix that mislabeled statistic and clarify the cumulation method for satisfaction rates. The bigger issue is Table I. The \"only one paper supports interpretability, zero support customization or privacy\" counts are binary judgments by two authors with no rubric, no example codings, and no inter-rater agreement for Stage 3. That matters because customization explicitly includes adapting anomaly thresholds, and many detectors do threshold tuning; the paper never states why that doesn't count. I'd want the coding made auditable before this becomes a canonical citation. On the other hand, the stress-test concern about interpretability is partially answered in the paper itself – it explicitly notes that Zhao et al. has no human evaluation, which is the right kind of honesty. The convenience sample (GitHub contributors + professional networks) is a limitation the authors acknowledge; it tempers the percentages but doesn't undermine the direction of the findings.\n\nWho's this for: software engineering researchers working on log anomaly detection, and tool builders in observability. It's a serious paper with a flawed section, not a flawed paper. I'd send it to referees and ask them to focus on the coding audit and the statistic repair. I'd probably cite it for the thresholds once the numbers are fixed.","headline":"First real map of practitioner expectations for log anomaly detection, with concrete adoption thresholds; the gap analysis is more judgment-call than the authors let on, but the central finding holds up.","tokens_in":34121,"tokens_out":1605,"would_cite":true,"duration_ms":15481,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated log anomaly detection research is optimizing for the wrong outcomes: it delivers accuracy but not the interpretability, customization, and privacy practitioners say they need.","keywords":["log anomaly detection","practitioners' expectations","empirical survey","literature review","interpretability","customization","privacy protection","software maintenance"],"falsifier":"A replication that samples practitioners through a probability-based method instead of professional networks and open-source contributor emails, and finds that only a minority value interpretability and customization, would undercut the claimed gap; equally, a re-review of the same venues with the same inclusion criteria that located many papers offering rationales, customization, or privacy protections would show the gap is smaller than reported.","tokens_in":33155,"feed_emoji":"📊","tokens_out":7502,"duration_ms":61349,"temperature":0.7,"pith_summary":"This paper tries to establish that there is a large gap between what software practitioners want from automated log anomaly detection and what a decade of research has delivered. It bases this claim on a survey of 312 practitioners from 36 countries and a structured review of 36 papers from premier venues published from 2014 to 2024. The central numbers: 95.5% of practitioners call automated tools essential or worthwhile; 84.5% require interpretability and 78.2% require customization, yet only one reviewed paper offers interpretable output and none offer customization or privacy protection. More than 74% of practitioners report that metrics and traces are available, but only four reviewed studies use them. If the paper is right, benchmark-driven accuracy gains are not enough: adoption depends on factors the literature has mostly skipped, and research should shift toward interpretable, generalizable, customizable, privacy-aware, and user-friendly tools.","feed_headline":"Only 1 of 36 log-anomaly papers explains its detections","feed_subtitle":"Practitioners rank interpretability and customization above raw accuracy; research barely delivers either.","key_machinery":"The machinery is a three-stage mixed-methods design: (1) 15 semi-structured interviews with professionals produce a set of issues and expectations; (2) an online survey of 312 practitioners turns those into percentages and minimum thresholds; (3) a structured literature review of 36 papers from premier venues (2014-2024) rates each technique against nine factors—data resource, granularity, effectiveness (recall and precision), efficiency, scalability, interpretability, generalizability, customization, and privacy protection—using the comparison table as the central instrument. The satisfaction-rate column is the load-bearing device: for any capability range a paper reports, the authors look up what percentage of surveyed practitioners would accept that level, converting research capabilities into adoption likelihoods.","core_discovery":"The paper's central claim, stated in its own terms, is that current automated log anomaly detection research does not meet practitioners' expectations, and the mismatch is systematic. Across nine comparison dimensions, the review finds that most techniques perform well only on the dimensions practitioners care least about as adoption gatekeepers: 30 of 36 papers report recall in the 80-100% range and 26 report precision there, satisfying all surveyed practitioners, while interpretability, customization, and privacy protection—valued by 84.5%, 78.2%, and 66.6% respectively—are addressed by one paper, no papers, and no papers. The paper also reports a data-availability mismatch: 88.9% of studies train on historical labeled normal logs, yet only 39.3% of practitioners always have that data, whereas metrics and traces are available to at least 74% and used by only four studies. The paper's overall conclusion is that accuracy on public benchmarks is necessary but not sufficient for adoption.","pith_inferences":["Beyond the paper: because practitioners already have metrics and traces, a natural next step is to benchmark multi-modal detectors that consume logs plus metrics and traces; the review suggests only a handful of techniques currently do this.","Beyond the paper: the complete absence of customization and privacy in the reviewed literature implies that public benchmarks reward only accuracy; adding adoption-oriented metrics such as time-to-explain, configurability, and data-protection guarantees would test whether these properties can be engineered into detectors.","Beyond the paper: practitioners want rationales integrated into existing tools rather than as add-ons, which points to LLM-based explanation as a concrete direction: generate natural-language rationales for flagged logs and measure whether they cut manual investigation time, not just whether practitioners say they like them.","Beyond the paper: the finding that about half of practitioners still would not prioritize research techniques despite high reported accuracy suggests that offline F1 gains may not transfer to industry; field deployments that track adoption decisions would be the direct test."],"forward_implications":["Researchers should prioritize interpretable output; currently 1 of 36 reviewed papers provides a rationale for detected anomalies, while 84.5% of practitioners say interpretability shapes their acceptance.","The reviewed techniques should incorporate metrics and traces, since at least 74% of practitioners have such data available but only 4 of 36 studies use them.","Customization and privacy protection, with zero coverage in the reviewed literature, are adoption conditions for 78.2% and 66.6% of practitioners, so tool-building efforts need to address them.","Detection-time reporting needs to become standard: 63.9% of reviewed papers do not report anomaly detection time, while roughly 80% of practitioners want detection within one minute.","Cross-project generalizability deserves evaluation as a first-class property: only 4 of 36 papers test across domains, yet 83.8% of practitioners expect tools to handle diverse log structures."],"supporting_citations":[{"why":"The only reviewed paper whose technique is classified as offering interpretable output; it anchors the interpretability-gap finding.","marker":"[33]"},{"why":"One of the four reviewed papers evaluating cross-project generalizability, via a pretrain-and-tune Transformer pipeline.","marker":"[19]"},{"why":"One of the four cross-project generalizability papers; it also supplies the claim that high accuracy requires target-domain similarity.","marker":"[20]"},{"why":"One of the four cross-project generalizability papers, using meta-learning for cross-system detection.","marker":"[22]"},{"why":"One of the four cross-project generalizability papers, using transfer learning across systems.","marker":"[36]"},{"why":"A reviewed technique that integrates metrics and traces; it also notes industrial datasets are access-restricted, grounding the data-availability discussion.","marker":"[5]"},{"why":"DeepLog, the sequence-level LSTM baseline used in the review's granularity and efficiency comparisons.","marker":"[6]"},{"why":"The HDFS benchmark dataset, one of the two public datasets driving the scalability and sequence-level grouping analysis.","marker":"[51]"},{"why":"The BGL benchmark dataset; its single-source content illustrates the limited log diversity behind generalizability concerns.","marker":"[52]"}],"fun_headline_variants":["Only 1 of 36 log-anomaly papers explains detections","Practitioners want interpretability; research delivers accuracy","Log studies train on labeled logs most teams never have","Accuracy alone won't drive log anomaly tool adoption"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 312 practitioners recruited through the authors' professional networks and emails to open-source contributors represent software practitioners generally, so the reported percentages (95.5% importance, 84.5% interpretability, 78.2% customization) hold beyond this convenience sample.","fun_headline_variants_meta":{"raw":{"variants":["Only 1 of 36 log-anomaly papers explains detections","Practitioners want interpretability; research delivers accuracy","Log studies train on labeled logs most teams never have","Accuracy alone won't drive log anomaly tool adoption"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3053,"prompt_tokens":872,"completion_tokens":2181,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":2116}},"tokens_in":488,"tokens_out":2181,"duration_ms":16821,"temperature":1.0,"reasoning_tokens":2116,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:42:35.235559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication that samples practitioners through a probability-based method instead of professional networks and open-source contributor emails, and finds that only a minority value interpretability and customization, would undercut the claimed gap; equally, a re-review of the same venues with the same inclusion criteria that located many papers offering rationales, customization, or privacy protections would show the gap is smaller than reported.","supporting_citations":[{"cited_title":"Heteroge- neous anomaly detection for software systems via semi-supervised cross- modal attention,","cited_arxiv_id":null,"evidence_quote":"A reviewed technique that integrates metrics and traces; it also notes industrial datasets are access-restricted, grounding the data-availability discussion."},{"cited_title":"Deeplog: Anomaly detection and diagnosis from system logs through deep learning,","cited_arxiv_id":null,"evidence_quote":"DeepLog, the sequence-level LSTM baseline used in the review's granularity and efficiency comparisons."},{"cited_title":"Detecting large-scale system problems by mining console logs,","cited_arxiv_id":null,"evidence_quote":"The HDFS benchmark dataset, one of the two public datasets driving the scalability and sequence-level grouping analysis."}],"review_version":1}