Pith. sign in

REVIEW 3 major objections 5 minor 83 references

AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AdaptiveLog claims that routing only uncertain logs to a large language model beats full-LLM analysis at 27% of the LLM cost.

desk verdict A practical SLM/LLM cascade for log analysis with a genuinely useful error-case retrieval prompt, but the uncertainty routing rule in §3.3 is inverted as written and must be resolved before the 73% cost claim can be trusted. read the letter →

arxiv 2501.11031 v1 pith:HB4KCH6B submitted 2025-01-19 cs.SE cs.AIcs.CL

Lipeng Ma , Weidong Yang , Yixuan Li , Ben Fei , Mingjie Zhou , Shuhao Li , Sihang Jiang , Bo Xu
show 1 more author
Yanghua Xiao
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords adaptiveloganalysislargelanguagemodelsmalluncertaintyestimationMonteCarlodropoutin-contextlearningcase-basedreasoningcost-efficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaptiveLog tries to prove that automated log analysis can get large-language-model accuracy at a fraction of the cost by letting a cheap, fine-tuned small model answer most logs and invoking an expensive LLM only when the small model is uncertain. The paper also claims that the LLM's hard-case answers improve when its prompt includes retrieved examples of the small model's past mistakes, together with reasoning and pitfalls. If these claims hold, operators of large systems could cut LLM inference expense by about 73% while improving overall accuracy, and the same collaboration recipe could be reused for other tasks with a cheap model and an expensive one.

What carries the argument

The central mechanism is the uncertainty estimate $p(C|(x,y'))$ computed after the SLM predicts: the SLM is run $N=10$ times with dropout, the deviation of each run from the mean is compared with the variation seen on correct validation samples, and the count of 'uncertain' observations updates a Beta prior $\mathrm{Beta}(err,1-err)$ to a posterior whose expectation is the routing score. The LLM is invoked when that score exceeds 0.5. The second mechanism is the Error-Case Reasoning Enhancement (ECR), a prompt strategy that retrieves similar error-prone cases: validation errors of the SLM are converted by the LLM into cases containing reasoning and pitfalls, stored in a key-value database, retrieved by cosine similarity using an embedding model, and placed in the prompt in ascending similarity order to counter recency bias.

What would settle it

Take any held-out test set from the paper (for example, LDSM Huawei-Switches), run the SLM with ten dropout passes per sample, and compare the estimated uncertainty probability with whether the SLM's prediction is actually wrong; if uncertainty is not systematically higher for misclassified logs than for correct logs, or if the Section 3.3 rule selects the opposite subset, the routing mechanism is not doing the work and the framework's gains would disappear.

Watch

Extended reading notes

Core claim

The paper's central claim is that an adaptive router can give a large language model's accuracy at a small model's cost: on six log-analysis tasks (anomaly detection, failure identification, module classification, level prediction, log-description semantic matching, and log-cause ranking), AdaptiveLog outperforms both the fine-tuned BERT small model and a ChatGPT baseline that analyzes every sample, while invoking the LLM for only about 27% of logs on average. The router uses a Bayesian posterior over the small model's error probability, updated by repeated Monte Carlo dropout passes, to decide when the LLM is needed; the LLM's hard-sample answers are then improved by the ECR prompt, which retrieves similar past mistakes of the small model along with their reasoning and pitfalls. The paper reports that ECR beats standard in-context learning on hard samples and that the same recipe improves results across different SLMs, low-resource settings, and cross-vendor transfer.

Load-bearing premise

The load-bearing premise is that the Section 3.3 counting rule, which treats a small deviation between repeated predictions as an uncertain event, correctly identifies logs the small model will get wrong; if that sign convention is wrong, the router sends the wrong samples to the LLM and the claimed cost-accuracy trade-off collapses.

Editorial extensions

If this is right

  • Operators can cut LLM API spend by about 73% on similar workloads while keeping or improving accuracy, because only about 27% of logs on average are routed to the LLM.
  • In low-resource settings with 10% of the training labels, the framework's advantage over all baselines grows, since the weaker SLM sends more samples to the LLM.
  • The framework is not tied to BERT: combining it with Biglog, KnowLog, Llama-3.2-3B, or ChatGLM-6B improves hard-sample accuracy over each base model alone.
  • Prompting the LLM with similar error-prone cases that include a reasoning process and potential pitfalls beats standard in-context learning and random error cases on hard samples.
  • The SLM uncertainty estimation is cheap enough (a few percent of total runtime) that the framework remains faster overall than analyzing every log with ChatGPT.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond logs, the same two-model recipe should transfer to any high-volume classification task where a cheap fine-tuned model handles most inputs and an expensive model is reserved for the uncertain tail; the paper does not test that, but nothing in the method is log-specific.
  • The reported 73% saving is measured against a ChatGPT baseline that analyzes every sample; a fairer operational comparison would also count the GPU time of the SLM's ten dropout passes, which the paper reports as about 4.68% of runtime but does not fold into the dollar cost.
  • A natural stress test is to replace the Monte Carlo dropout variation rule with a calibrated confidence score (softmax entropy or a temperature-scaled probability) and check whether the same cost-accuracy frontier holds; the paper's own learned-router ablation suggests routing is hard, so a calibrated simple baseline would be informative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AdaptiveLog, a hybrid log-analysis framework that first obtains a prediction from a fine-tuned small language model (SLM), estimates the SLM's uncertainty via Monte Carlo dropout and Bayesian updating, and invokes a large language model (LLM) only when the estimated uncertainty exceeds certainty. For LLM-routed samples, the framework retrieves similar error-prone cases from a validation-derived mistake database and prompts the LLM with their reasoning processes and potential pitfalls. Experiments are reported on six log-analysis tasks (anomaly detection, failure identification, module classification, level prediction, log-description semantic matching, and log-cause ranking), including low-resource and transfer settings, with a claimed 73% reduction in LLM query cost relative to querying the LLM on all samples.

Significance. If the central claims hold, AdaptiveLog would be a practically useful contribution: it combines the efficiency of a fine-tuned SLM with the reasoning ability of an LLM, and the idea of retrieving error-prone cases to prompt the LLM is plausible and well motivated. The paper is also commendable for its breadth of evaluation across six tasks, its explicit ablation studies of both the routing mechanism and the prompt strategy, and its provision of source code and detailed experimental data. However, the manuscript as written contains an internal inconsistency in the uncertainty-counting rule that is load-bearing for the headline cost-accuracy claim, and the baseline comparisons are weakened by a potentially unfair re-pre-training procedure for Biglog and KnowLog. These issues must be resolved before the empirical conclusions can be accepted.

major comments (3)
  1. [§3.3, Eq. (8) and following text] The uncertainty-counting rule is inverted as written. The manuscript defines Variation as the mean absolute deviation of Monte-Carlo-dropout probabilities on correct validation samples and then sets alpha = #{n : Observation_n <= Variation}. Because correct samples are stable, Variation is small, and an unstable or error-prone input should exhibit Observation_n > Variation on most passes. The written inequality therefore assigns high alpha to confident samples and low alpha to uncertain ones, inverting the routing decision in Eq. (9). This contradicts the paper's own partition: Table 3 labels hard samples as high-uncertainty and reports BERT hard-sample F1 of 0.0 on BGL while easy samples reach 99.84, a separation that can only arise if the implemented code counts the opposite inequality. Section 5.1 further states that models are 'prone to errors when their uncertainty probability is low' immediately after reporting that error samples cluster at uncertainty 0.8–1.0. The authors must correct the inequality, or explicitly explain the intended semantics, and reconcile these statements; as written, the 73% cost-saving and superior-results claims are unsupported.
  2. [§3.3, Eqs. (4)–(6)] Equation (4) writes the likelihood as p((x,y')|C) ≈ p(C)^alpha (1-p(C))^{N-alpha}, substituting the prior probability for the Bernoulli parameter. The likelihood of observing alpha uncertain outcomes given parameter theta should be theta^alpha (1-theta)^{N-alpha}, and the posterior in Eq. (5) is indeed the correct conjugate update for that likelihood. As written, however, Eq. (4) conflates the prior with the likelihood parameter and is mathematically inconsistent. In addition, setting p(C) = err from Eq. (1) assumes that the unobservable 'uncertain' event C coincides with classification error; this equivalence is asserted rather than justified. The final update formula in Eq. (6) is standard, but the derivation needs to be rewritten and the assumption stated explicitly.
  3. [§4.3, Tables 3–5] The baseline comparison for Biglog and KnowLog is potentially unfair as described. The text says these models are 'reproduced with the same pre-training setting on our log corpus (all training sets),' and the training sets are very small (e.g., 196 samples for OpenStack and roughly 400–7,000 samples for the network-device tasks). The documented advantage of Biglog and KnowLog comes from large-scale log pre-training, so re-pre-training them from scratch on the task corpora likely removes their main strength and makes the 'state-of-the-art' claim less convincing. Please clarify whether the original pre-trained checkpoints were used or whether the models were trained from scratch on the small corpora, and if the latter, add comparisons with the original pre-trained versions or justify why this setting is appropriate.
minor comments (5)
  1. [§5.1, Table 9 and Fig. 14] The text references 'Table ??' when discussing the effect of the proportion of error samples on ECR; this should point to Fig. 14 or a properly numbered table.
  2. [§5.3] The threats-to-validity paragraph on randomness says 'Randomness in the selection of error cases in RQ4,' but the random-case experiment appears in RQ6 (Fig. 10), not in the RQ4 study of different SLMs; the cross-reference should be corrected.
  3. [Tables 3–5] Several entries in Tables 3–5 have formatting problems, such as '86.0 /96.0' and similarly missing spaces around slashes; these should be cleaned up for readability.
  4. [§3.3, Eq. (6)] The phrase 'the first term is a constant' is imprecise: err/(N+1) is constant with respect to the observation count alpha, but it depends on the validation error rate; this should be stated more carefully.
  5. [§1 and §3.4] The paper calls the approach 'simple and non-parametric,' but several hyperparameters are set empirically, including N (number of dropout observations), the routing threshold of 0.5 in Eq. (9), k (number of retrieved cases), and the similarity threshold for case quality; the 'non-parametric' phrasing should be softened or qualified.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: uncertainty routing and ECR prompting are fixed rules evaluated on disjoint test splits.

full rationale

AdaptiveLog's claimed derivation chain is not circular. The uncertainty probability in Eq. (6) is p(C|(x,y'))=(err+alpha)/(N+1), where err is the SLM's validation error rate (Eq. 1) and alpha is a count of MC-dropout variation observations (Eqs. 7-8) computed on the new input. Neither quantity is fitted to test labels or to the reported test metrics; the threshold in Eq. (9) is a fixed decision rule. The measured "hard sample" split and the 73% cost saving are consequences of applying this fixed rule to the test split, not quantities whose values were used to define the rule. The error-prone case database (Section 3.2) is built from validation error samples with validation labels; this is a legitimate use of held-out data, and the final metrics are reported on a separate chronological test split. The only self-citation of substance is KnowLog [45], used as a baseline and as an alternative SLM; it does not supply a premise of the uncertainty derivation or a uniqueness argument, so it is not load-bearing. The sign convention in Section 3.3 (alpha counting Observation_n <= Variation, while later text says error samples cluster at high uncertainty) is internally inconsistent and may be a correctness risk, but it is not circularity: either reading still defines a fixed, non-fitted routing rule rather than a parameter fitted to the target result. No equation is equivalent by construction to the claimed outcome.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a handful of manually chosen hyperparameters (N, k, threshold) and three domain assumptions about uncertainty calibration and retrieval quality.

free parameters (4)
  • N = 10
    Number of Monte Carlo dropout forward passes for uncertainty estimation; set by hand in Section 3.3.
  • k = 5
    Number of retrieved error-prone cases in ECR prompt; set by hand in Section 3.4.1.
  • routing_threshold = 0.5
    Threshold for invoking LLM (uncertainty must exceed 0.5), set by hand in Equation (9).
  • dropout_variation_threshold = computed from validation correct samples
    Threshold for counting an observation as uncertain; estimated from validation set, not a learned parameter but a data-dependent cutoff.
assumptions (4)
  • domain assumption Monte Carlo dropout variation is a reliable proxy for SLM predictive uncertainty on log data
    Section 3.3 relies on this to count uncertain observations; no evidence given that this holds for fine-tuned BERT on logs.
  • domain assumption Validation error rate err is a suitable prior for the probability that the SLM is uncertain on a new sample
    Equation (1) and (3) set p(C)=err; assumes i.i.d. and stationarity between validation and test.
  • standard math Dropout passes are i.i.d. observations
    Used to justify the binomial likelihood in Equation (4).
  • domain assumption Error-prone cases retrieved by cosine similarity in embedding space improve LLM reasoning
    ECR strategy in Section 3.4; supported by ablations but not a proven general property.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model." pith.science (2026). https://pith.science/paper/HB4KCH6B

@misc{pith2026250111031,
  author       = {Pith},
  title        = {Pith review of: AdaptiveLog: An Adaptive Log Analysis Framework with the Collaboration of Large and Small Language Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HB4KCH6B}},
  note         = {Machine review of arXiv:2501.11031}
}
read the original abstract

Automated log analysis is crucial to ensure high availability and reliability of complex systems. The advent of LLMs in NLP has ushered in a new era of language model-driven automated log analysis, garnering significant interest. Within this field, two primary paradigms based on language models for log analysis have become prominent. Small Language Models (SLMs) follow the pre-train and fine-tune paradigm, focusing on the specific log analysis task through fine-tuning on supervised datasets. On the other hand, LLMs following the in-context learning paradigm, analyze logs by providing a few examples in prompt contexts without updating parameters. Despite their respective strengths, we notice that SLMs are more cost-effective but less powerful, whereas LLMs with large parameters are highly powerful but expensive and inefficient. To trade-off between the performance and inference costs of both models in automated log analysis, this paper introduces an adaptive log analysis framework known as AdaptiveLog, which effectively reduces the costs associated with LLM while ensuring superior results. This framework collaborates an LLM and a small language model, strategically allocating the LLM to tackle complex logs while delegating simpler logs to the SLM. Specifically, to efficiently query the LLM, we propose an adaptive selection strategy based on the uncertainty estimation of the SLM, where the LLM is invoked only when the SLM is uncertain. In addition, to enhance the reasoning ability of the LLM in log analysis tasks, we propose a novel prompt strategy by retrieving similar error-prone cases as the reference, enabling the model to leverage past error experiences and learn solutions from these cases. Extensive experiments demonstrate that AdaptiveLog achieves state-of-the-art results across different tasks, elevating the overall accuracy of log analysis while maintaining cost efficiency.

Figures

Figures reproduced from arXiv: 2501.11031 by the authors.

Figure 1
Figure 1. Different log analysis frameworks for anomaly detection with language models, where SLMs are efficient but low [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The framework of AdaptiveLog, which comprises three key steps and a preparation stage. It first analyzes logs with [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The prompt template example of ECR on the anomaly detection task, where the error-prone cases including reasoning [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Results in the low-resource scenario. The experimental results are shown in [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Results in the transfer learning scenario. Left side of [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Efficieny of AdaptiveLog and ChatGPT on different tasks. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Time with different number of observations on LDSM and LPCR tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Results with different number of observations on LDSM and LPCR tasks. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Results of different error case numbers on the LDSM. [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Results on different case selection strategies. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Ablation studies on Bayesian inference. these mistakes and thus avoid the same errors. In conclusion, choosing suitable examples is important for the LLM. Similar error cases are more conducive to improving the reasoning of LLM on logs. Hence, we encourage users to co…
Figure 12
Figure 12. Figure 12: Percentage of hard samples selected correctly. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Proportion of LLMs invoked by different tasks, where we count the proportion of hard samples for all datasets on [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: ECR results based on different proportions of error-prone cases on the LDSM. [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Uncertainty distribution chart on LDSM task. [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Qualitative example on the Level Prediction task. [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

83 extracted references · 52 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large Language Models for Mathematical Reasoning: Progresses and Challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop. 225–237

    2024

  3. [3]

    Crispin Almodovar, Fariza Sabrina, Sarvnaz Karimi, and Salahuddin Azad. 2024. LogFiT: Log anomaly detection using fine-tuned language models. IEEE Transactions on Network and Service Management (2024)

    2024

  4. [4]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165 (2020)

    arXiv 2020

  5. [5]

    Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al

  6. [6]

    Laming Chen, Guoxin Zhang, and Eric Zhou. 2018. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems 31 (2018)

    work page 2018

  7. [7]

    Rui Chen, Shenglin Zhang, Dongwen Li, Yuzhe Zhang, Fangrui Guo, Weibin Meng, Dan Pei, Yuzhi Zhang, Xu Chen, and Yuqing Liu

    work page

  8. [8]

    Song Chen and Hai Liao. 2022. Bert-log: Anomaly detection for system logs based on pre-trained language model. Applied Artificial Intelligence 36, 1 (2022), 2145642

    work page 2022

Show all 83 references
  1. [9]

    Domenico Cotroneo, Luigi De Simone, Pietro Liguori, Roberto Natella, and Nematollah Bidokhti. 2019. How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. In Proceedings of the 2019 27th ACM Joint Meeting on European Softwa...

    2019

  2. [10]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  3. [11]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234 (2022)

    2022 arXiv

  4. [12]

    Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security . 1285–1298

    2017

  5. [13]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou

  6. [14]

    Chris Egersdoerfer, Di Zhang, and Dong Dai. 2022. Clusterlog: Clustering logs for effective log-based anomaly detection. In 2022 IEEE/ACM 12th Workshop on Fault Tolerance for HPC at eXtreme Scale (FTXS) . IEEE, 1–10

    2022

  7. [15]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1469–1481

    2023

  8. [16]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. , Vol. 1, No. 1, Article . Publication date: January 2018. 28 • L. Ma et al

    2018

  9. [17]

    Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning . PMLR, 10421–10430

    2023

  10. [18]

    Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning . PMLR, 1050–1059

    2016

  11. [19]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations

    2022

  12. [20]

    Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, Hongyu Zhang, and Michael R Lyu. 2023. What makes good in-context demonstrations for code intelligence tasks with llms?. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 761–773

    2023

  13. [21]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations

    2024

  14. [22]

    Jiechao Gao, Haoyu Wang, and Haiying Shen. 2020. Task failure prediction in cloud data centers using deep learning. IEEE transactions on services computing 15, 3 (2020), 1411–1422

    2020

  15. [23]

    Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM computing surveys (CSUR) 54, 6 (2021), 1–37

    2021

  16. [24]

    Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. 2024. A deep dive into large language models for automated bug localization and repair. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1471–1493

    2024

  17. [25]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al . 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024)

    2024 arXiv

  18. [26]

    Shaohan Huang, Yi Liu, Carol Fung, He Wang, Hailong Yang, and Zhongzhi Luan. 2023. Improving log-based anomaly detection by pre-training hierarchical transformers. IEEE Trans. Comput. 72, 9 (2023), 2656–2667

    2023

  19. [27]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of code language models on automated program repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 1430–1442

    2023

  20. [28]

    Shaohan Huang, Yi Liu, Carol Fung, Rong He, Yining Zhao, Hailong Yang, and Zhongzhi Luan. 2020. Hitanomaly: Hierarchical transformers for anomaly detection in system log. IEEE transactions on network and service management 17, 4 (2020), 2064–2076

    2020

  21. [29]

    James Joyce. 2003. Bayes’ theorem. (2003)

    2003

  22. [30]

    Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning an...

    2023

  23. [31]

    Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2023. LILAC: Log Parsing using LLMs with Adaptive Parsing Cache. arXiv preprint arXiv:2310.01796 (2023)

    2023 arXiv

  24. [32]

    Janet L Kolodner. 1992. An introduction to case-based reasoning. Artificial intelligence review 6, 1 (1992), 3–34

    1992

  25. [33]

    Sawan Kumar and Partha Talukdar. 2021. Reordering Examples Helps during Priming-based Few-Shot Learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 . 4507–4518

    2021

  26. [34]

    Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems 30 (2017)

    2017

  27. [35]

    Van-Hoang Le and Hongyu Zhang. 2022. Log-based anomaly detection with deep learning: How far are we?. In Proceedings of the 44th international conference on software engineering . 1356–1367

    2022

  28. [36]

    Van-Hoang Le and Hongyu Zhang. 2023. Log parsing with prompt-based few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2438–2449

    2023

  29. [37]

    Van-Hoang Le and Hongyu Zhang. 2021. Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 492–504

    2021

  30. [38]

    Yukyung Lee, Jina Kim, and Pilsung Kang. 2023. Lanobert: System log anomaly detection based on bert masked language model. Applied Soft Computing 146 (2023), 110689

    2023

  31. [39]

    Xiaoyun Li, Pengfei Chen, Linxiao Jing, Zilong He, and Guangba Yu. 2020. Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 92–103. , Vol. 1, N...

    2020

  32. [40]

    Van-Hoang Le and Hongyu Zhang. 2024. PreLog: A Pre-trained Model for Log Analytics. Proceedings of the ACM on Management of Data 2, 3 (2024), 1–28

    2024

  33. [41]

    Yichen Li, Yintong Huo, Renyi Zhong, Zhihan Jiang, Jinyang Liu, Junjie Huang, Jiazhen Gu, Pinjia He, and Michael R Lyu. 2024. Go Static: Contextualized Logging Statement Generation. arXiv preprint arXiv:2402.12958 (2024)

    2024 arXiv

  34. [42]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  35. [43]

    Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, and Michael R Lyu. 2023. Exploring the effectiveness of llms in automated logging generation: An empirical study. arXiv preprint arXiv:2307.05950 (2023)

    2023 arXiv

  36. [44]

    Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang. 2018. Detecting anomaly in big data system logs using convolutional neural network. In 2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl C...

    2018

  37. [45]

    Lipeng Ma, Weidong Yang, Bo Xu, Sihang Jiang, Ben Fei, Jiaqing Liang, Mingjie Zhou, and Yanghua Xiao. 2024. Knowlog: Knowledge enhanced pre-trained language model for log understanding. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13

    2024

  38. [46]

    Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yuhang Chen, Yanqing Zhao, Hao Yang, and Yanfei Jiang. 2024. Interpretable online log analysis using large language models with prompt strategies. In Proceedings of the 32nd IEEE/ACM International Conference on Progr...

    2024

  39. [47]

    Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 10572–10601

    2023

  40. [48]

    Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. 2024. Llmparser: An exploratory study on using large language models for log parsing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

    2024

  41. [49]

    Xiaoxue Ma, Huiqi Zou, Jacky Keung, Pinjia He, Yishu Li, Xiao Yu, and Federica Sarro. 2024. On the Influence of Data Resampling for Deep Learning-Based Log Anomaly Detection: Insights and Recommendations. arXiv preprint arXiv:2405.03489 (2024)

    2024 arXiv

  42. [50]

    Haibo Mi, Huaimin Wang, Yangfan Zhou, Michael Rung-Tsong Lyu, and Hua Cai. 2013. Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems. IEEE Transactions on Parallel and Distributed Systems 24, 6 (2013), 1245–1255

    2013

  43. [51]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196 (2024)

    2024 arXiv

  44. [52]

    Neil Mercer. 2008. Talk and the development of reasoning and understanding. Human development 51, 1 (2008), 90–100

    2008

  45. [53]

    Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, Hanlu Li, Danfeng Zhu, and Depei Qian. 2023. Logencoder: Log-based contrastive representation learning for anomaly detection. IEEE Transactions on Network and Service Management 20, 2 (2023), 1378–1391

    2023

  46. [54]

    Taly Reich, Alex Kaju, and Sam J Maglio. 2023. How to overcome algorithm aversion: Learning from mistakes. Journal of Consumer Psychology 33, 2 (2023), 285–302

    2023

  47. [55]

    Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. 2022. Spt-code: Sequence-to-sequence pre-training for learning source code representations. In Proceedings of the 44th international conference on software engineering . 2006–2018

    2022

  48. [56]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024)

    2024 arXiv

  49. [57]

    Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2023. Cocosoda: Effective contrastive learning for code search. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2198–2210

    2023

  50. [58]

    Devjeet Roy, Xuchao Zhang, Rashi Bhave, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan. 2024. Exploring llm-based agents for root cause analysis. arXiv preprint arXiv:2403.04123 (2024)

    2024 arXiv

  51. [59]

    Shimin Tao, Yilun Liu, Weibin Meng, Zuomin Ren, Hao Yang, Xun Chen, Liang Zhang, Yuming Xie, Chang Su, Xiaosong Oiao, et al

  52. [60]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering (2024)

    2024

  53. [61]

    Jiahao Shi, Sihang Jiang, Bo Xu, and Yanghua Xiao. 2023. ServerRCA: Root Cause Analysis for Server Failure using Operating System Logs. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 486–496

    2023

  54. [62]

    Ian Watson and Farhi Marir. 1994. Case-based reasoning: A review. The knowledge engineering review 9, 4 (1994), 327–354

    1994

  55. [63]

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846 (2023)

    2023 arXiv

  56. [64]

    Thorsten Wittkopp, Philipp Wiesner, and Odej Kao. 2024. LogRCA: Log-based Root Cause Analysis for Distributed Services. arXiv preprint arXiv:2405.13599 (2024). , Vol. 1, No. 1, Article . Publication date: January 2018. 30 • L. Ma et al

    2024 arXiv

  57. [65]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  58. [66]

    Junjielong Xu, Ruichun Yang, Yintong Huo, Chengyu Zhang, and Pinjia He. 2024. DivLog: Log Parsing with Prompt Enhanced In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–12

    2024

  59. [67]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116 (2024)

    2024 arXiv

  60. [68]

    Boxi Yu, Jiayi Yao, Qiuai Fu, Zhiqing Zhong, Haotian Xie, Yaoliang Wu, Yuchi Ma, and Pinjia He. 2024. Deep Learning or Classical Machine Learning? An Empirical Study on Log-Based Anomaly Detection. In Proceedings of the 46th IEEE/ACM International Conference on Software Engine...

    2024

  61. [69]

    Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, et al . 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12

    2024

  62. [70]

    Beichen Zhang, Kun Zhou, Xilin Wei, Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2024. Evaluating and improving tool-augmented computation-intensive math reasoning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  63. [71]

    Shenglin Zhang, Ying Liu, Weibin Meng, Zhiling Luo, Jiahao Bu, Sen Yang, Peixian Liang, Dan Pei, Jun Xu, Yuzhi Zhang, et al. 2018. Prefix: Switch failure prediction in datacenter networks. Proceedings of the ACM on Measurement and Analysis of Computing Systems 2, 1 (2018), 1–29

    2018

  64. [72]

    Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, et al

  65. [73]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Sof...

    2024

  66. [74]

    Nengwen Zhao, Honglin Wang, Zeyan Li, Xiao Peng, Gang Wang, Zhu Pan, Yong Wu, Zhen Feng, Xidao Wen, Wenchi Zhang, et al

  67. [75]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning . PMLR, 12697–12706

    2021

  68. [76]

    Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R Lyu. 2023. Loghub: A large collection of system log datasets for ai-driven log analytics. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 355–366

    2023

  69. [77]

    Armin Zirak and Hadi Hemmati. 2024. Improving automated program repair with domain adaptation. ACM Transactions on Software Engineering and Methodology 33, 3 (2024), 1–43. , Vol. 1, No. 1, Article . Publication date: January 2018

    2024

  70. [78]

    Xu Zhang, Yong Xu, Si Qin, Shilin He, Bo Qiao, Ze Li, Hongyu Zhang, Xukun Li, Yingnong Dang, Qingwei Lin, et al . 2021. Onion: identifying incident-indicating logs for cloud systems. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and S...

    2021

  71. [2019]

    In Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering

    Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering . 807–817

    2019

  72. [2020]

    In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE)

    Logtransfer: Cross-system log anomaly detection for software systems with transfer learning. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 37–47

    2020

  73. [2021]

    In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

    An empirical investigation of practical log anomaly detection for online service systems. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1404–1415

  74. [2023]

    In 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS)

    Biglog: Unsupervised large-scale pre-training for a unified log representation. In 2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS) . IEEE, 1–11

    2023

  75. [2024]

    World Wide Web 27, 4 (2024), 42

    When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web 27, 4 (2024), 42

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.