Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned small open-source language models under 14 billion parameters, especially Qwen2.5-coder-14B with LoRA and RAG, beat existing tools and larger proprietary LLMs at predicting where to log and what to write.

desk verdict Useful empirical study of fine-tuned small models for logging generation; the 'larger is not always better' headline is not established because model size is confounded with fine-tuning. read the letter →

arxiv 2505.16590 v3 pith:EMDHQUA2 submitted 2025-05-22 cs.SE

Renyi Zhong , Yichen Li , Guangba Yu , Wenwei Gu , Jinxi Kuang , Yintong Huo , Michael R. Lyu This is my paper
classification cs.SE
keywords automatedloggingstatementgenerationsmallopen-sourcelanguagemodelsparameter-efficientfine-tuningLow-RankAdaptation(LoRA)retrieval-augmentedempiricalstudyAL-Benchsoftwaremaintenance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the first large-scale empirical study of small open-source language models (SOLMs, under 14 billion parameters) for automated logging statement generation. The authors test four SOLMs on the AL-Bench dataset, varying prompting strategies and parameter-efficient fine-tuning, and find that a specific recipe—an instruction-tuned model, LoRA fine-tuning, and retrieval-augmented generation with a similar-example prompt—lets Qwen2.5-coder-14B exceed both specialized logging tools and much larger proprietary LLMs on location accuracy and statement quality. If the finding holds, enterprises could deploy locally hosted logging assistants that preserve code privacy, lower cost, and still reach top results. The claim rests on the assumption that in-distribution fine-tuning, not model scale, is what the comparison actually isolates.

What carries the argument

The load-bearing mechanism is the LoRA+RAG synergy: LoRA injects low-rank adapter matrices that specialize a frozen base model to the logging task using only a small fraction of trainable parameters, while RAG retrieves the single most similar code snippet (via BM25) from a validation pool and prepends it as a one-shot example during inference. The paper shows this combination beats either technique alone and remains the best configuration across code lengths, and the AL-Bench benchmark (10 Java repositories, 33,224 instances split at file level to avoid leakage) supplies the measuring instrument, with position accuracy, level accuracy, variable precision and F1, BLEU-4, ROUGE-L, and an LLM-based judge as the metrics.

What would settle it

Fine-tune a large (e.g., 70B-parameter) open model with the same LoRA+RAG protocol on the same 26,713 AL-Bench training instances and evaluate it on the same test set; if it matches or exceeds Qwen2.5-coder-14B's 66.20% position accuracy, the claim that smaller models are the better choice collapses. A supporting check is to repeat the evaluation on logging statements collected after these models' training cutoffs to rule out memorization of AL-Bench.

Watch

Extended reading notes

Core claim

The central discovery is that a well-optimized sub-14B open-source model can be the top performer in automated logging. On AL-Bench, fine-tuned SOLMs with LoRA and RAG prompts outperform all five existing tools and four larger LLM baselines across position accuracy, level accuracy, variable matching, and text-generation metrics, and the LLM-as-a-judge evaluation agrees. The best configuration is Qwen2.5-coder-14B with LoRA and RAG, reaching 66.20% position accuracy, 69.92% level accuracy, 59.93% variable F1, and the highest judge scores; the paper reads this as evidence that for specialized logging tasks targeted adaptation outweighs raw model scale.

Load-bearing premise

The load-bearing premise is that fine-tuning on 26,713 in-distribution AL-Bench examples is not the true cause of the small models' wins; if a large model fine-tuned the same way also surpassed them, the 'larger is not always better' conclusion would fall.

Editorial extensions

If this is right

  • A sub-14B model fine-tuned for under six hours on a single A100 GPU can match or beat large proprietary LLMs at automated logging, enabling local, privacy-preserving deployment.
  • The LoRA+RAG combination is synergistic: task-level specialization from fine-tuning and instance-level context from retrieval together define the best-performing configuration.
  • Logging performance scales with size only from 3B parameters upward; models below 3B scale inconsistently, and a single fine-tuning epoch is enough to avoid overfitting.
  • SOLMs generalize to unseen repositories, and training on projects with consistent logging conventions, such as Apache-style practices, noticeably improves that generalization.
  • Longer code hurts logging-position accuracy but improves level, variable, and text quality, posing a design trade-off for real-world tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the headline comparison may reflect in-distribution fine-tuning advantage rather than model scale, since the large LLM baselines were prompted without being fine-tuned on the same 26,713 training instances.
  • Beyond the paper: the same LoRA+RAG recipe could be tested on other code-generation tasks, such as comment generation or test generation, to see whether the synergy transfers beyond logging.
  • Beyond the paper: because RAG-based retrieval quality depends on the validation pool, using a per-project knowledge base may matter more than model size in real deployments, and the observed failures to invoke out-of-context methods suggest that adding static-analysis call-graph context could improve the approach further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents an empirical study of small open-source language models (SOLMs, defined as models under 14B parameters) for automated logging statement generation on the AL-Bench benchmark. It evaluates four SOLMs with four prompting strategies and four parameter-efficient fine-tuning techniques, and reports that RAG prompting and LoRA fine-tuning are the most effective choices. The main claim is that fine-tuned SOLMs, especially Qwen2.5-coder-14B with LoRA and RAG, outperform existing logging tools and larger proprietary LLM baselines on location accuracy and statement quality, based on metrics such as PA, LA, PMR, F1, BLEU-4, ROUGE-L, and an LLM-as-a-judge scoring scheme. The paper also studies generalization across repositories and the effect of code length.

Significance. If the headline claim were established, the result would be practically valuable: a locally deployable, sub-14B model that beats GPT-4o and Claude3.7-sonnet on a well-known logging benchmark would give enterprises a privacy-preserving and cost-effective alternative. The paper has genuine strengths: it uses an external benchmark (AL-Bench) with a file-level split, explicitly addresses data leakage, releases a replication package, and conducts a fairly systematic comparison of prompting and PEFT strategies. However, the central scale claim is not currently supported by the experimental design, because the SOLM/LLM comparison in RQ3 varies model scale and fine-tuning simultaneously. The practical message about efficient, task-adapted SOLMs is credible; the theoretical message 'larger is not always better' is not yet established.

major comments (3)
  1. [Section 4.3, Table 7 and Finding 8] The headline comparison in Table 7 conflates model size with fine-tuning. Every SOLM row is LoRA-fine-tuned on 26,713 in-distribution AL-Bench instances and evaluated with a RAG prompt, while every proprietary and open LLM baseline is prompt-only. Table 4 shows that LoRA alone increases Qwen2.5-coder's position accuracy from 3.40 to 62.40, so fine-tuning is a dominant intervention. To support the title claim that 'larger is not always better' and Finding 8, the paper needs an ablation in which a large open model (e.g., Qwen2.5-coder-32B, LLaMA-3.1-70B, or an equivalent model) is fine-tuned with the identical LoRA+RAG protocol and shown not to match or exceed Qwen2.5-coder-14B. Without such an ablation, the data only support the narrower claim that a specialized, in-domain fine-tuned small model can beat generic prompted models.
  2. [Section 4.3, Tables 7 and Figure 6] The headline comparisons are single-run point estimates without confidence intervals or significance tests. Several decisive margins are very small: position accuracy is 66.20 for Qwen2.5-coder-14B versus 65.90 for Claude3.7-sonnet-RAG, and the average LLM-judge score is 1.506 versus 1.489. These gaps may be within sampling noise, especially given test-set and repository variability. The paper should report confidence intervals (e.g., bootstrap by test instance or by repository) or repeated running of the stochastic components, and it should use a statistical test before claiming that a method 'outperforms' all baselines across all metrics.
  3. [Section 4.4, Table 11, Finding 13, and Section 5.4.3] The long-code robustness result is in tension with Finding 8 and the title. Table 11 shows that the fine-tuned Qwen2.5-coder-14B loses 13.43 PA points on long code, while the Deepseek-coder-v3-RAG baseline loses only 9.97; the paper itself attributes this to 'smaller model scale may face greater challenges' in Section 5.4.3. This is evidence that the larger baseline is more robust in one important dimension, which undercuts the general claim that smaller scale is preferable. The authors should either reconcile this with the scale narrative (for example, by testing large fine-tuned models on long code) or explicitly restrict the scope of the central claim to short- to medium-length code.
minor comments (6)
  1. [Table 7] The column header prints 'LOD' but Section 3.5.1 defines and uses AOD; rename the column to AOD for consistency.
  2. [Section 4.3 approach text] The approach paragraph enumerates only LANCE, LEONID, Unilog, and Fastlog as existing methods, but Table 7 and the subsequent results include SCLogger; add SCLogger to the list.
  3. [Table 10] In the base+RAG Short row, the ROUGE-L value is reported as 15.10, which conflicts with Table 3's ROUGE-L of 37.57 for the same configuration; this appears to be a copy-paste from the BLEU-4 column.
  4. [Table 2] The column header 'LOLS' is not defined; define it as 'lines of logging statements' or rename it for clarity.
  5. [Throughout] Several typos should be fixed in copy editing: 'Tradictional evaluating metrics' (Section 3.5.1), 'Empircial' (Section 4 heading), 'investegates' (Section 3), and 'automoated' (Findings 2 and 3).
  6. [References] Reference [70] for AL-Bench is missing venue and date information ('[n. d.]'); update it to the full arXiv or publication entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central comparison is measured on an external, file-split benchmark rather than derived from fitted inputs or self-cited premises.

full rationale

The paper's derivation chain is an empirical evaluation, not a formal derivation. Test-set predictions are produced by fine-tuned SOLMs whose training data come from the AL-Bench training split (26,713 instances) with a file-level split, and RAG retrieval draws only from the validation split (3,508 instances); the test set (3,003 instances) is not used to fit any model parameter. The RQ3 comparison does vary two factors at once—all SOLM rows are LoRA-tuned on in-distribution data while LLM baselines are prompt-only—so the title claim 'Larger Is Not Always Better' is not fully established as a scale effect. This is a validity threat, not circularity: the SOLMs' reported scores are measured outputs, not quantities equal to training statistics by construction. Similarly, RQ2.4 selects the RAG prompt strategy using the test set, which risks selection bias but does not make the later RQ3 numbers definitionally forced. Self-citations (e.g., the prompt-design reference [11] and the authors' own SCLogger baseline) are not load-bearing: SCLogger is reproduced as an external comparison, and the prompting techniques are standard ICL/RAG/CoT methods from the general literature. No uniqueness theorem is imported, no ansatz is smuggled in via citation, and no metric is fitted to the target result. The honest finding is therefore no significant circularity.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the AL-Bench benchmark, on chosen hyperparameters for fine-tuning and retrieval, and on the assumption that the evaluation metrics and LLM judges reflect logging quality. The free parameters are experimental design choices rather than fitted scientific constants; no invented entities are introduced.

free parameters (7)
  • Max input length cutoff = 512 tokens
    Code snippets longer than 512 tokens are excluded from the main dataset and analyzed separately in RQ4.2; this threshold shapes the main test set and the reported baseline comparisons.
  • LoRA rank r = 16
    Used for all LoRA and QLoRA runs; it is a chosen hyperparameter from preliminary experiments, not tuned per model or per task.
  • LoRA alpha = 32
    Scaling factor for LoRA adapters; fixed across all models and runs.
  • Learning rate = 1e-4
    Fixed for all fine-tuning with a cosine scheduler; chosen without per-model tuning.
  • Fine-tuning epochs = 1 (5 for RQ2.3 epoch study)
    One epoch is used for all main runs; the base-versus-instruct comparison uses five epochs to study overfitting.
  • Global batch size = 64
    Fixed across all fine-tuning runs; a standard choice that affects optimization dynamics.
  • RAG retrieval configuration = BM25 top-1 from validation set
    RAG prompts retrieve the single most similar validation snippet with BM25; performance depends on this retrieval pool and similarity function.
assumptions (5)
  • domain assumption AL-Bench ground-truth logging statements are high-quality and representative of real-world practice.
    AL-Bench is the sole evaluation benchmark; if its ground truth is not a good proxy for quality logging, the performance comparisons may not transfer to real projects. Invoked in Section 3.1.1.
  • domain assumption Exact match to the original logging position is a valid correctness target.
    Position Accuracy counts only exact matches; the paper's own threats section acknowledges that multiple locations could be functionally equivalent, which would penalize all models similarly but may misrepresent quality. Invoked in Section 3.5.1.
  • domain assumption LLM-as-a-judge scores correlate with human developer judgments.
    The LLM judges (Claude3.7, Deepseek-coder-v3, GPT-4o) are used as the holistic quality measure without human validation; the threats section concedes possible misalignment. Invoked in Section 3.5.2 and Section 6.3.2.
  • domain assumption BLEU-4 and ROUGE-L are adequate proxies for logging text quality.
    The paper itself notes these metrics only capture lexical similarity and can miss semantic equivalence; they are used as primary text quality scores in all tables. Invoked in Section 3.5.1.
  • domain assumption File-level splitting prevents test leakage from the AL-Bench construction.
    All instances from one file are kept in one split, and AL-Bench's formatting and version choices are assumed to avoid pretraining contamination; the paper discusses this as a mitigation in Section 3.1.2 and Section 6.1.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation." pith.science (2026). https://pith.science/paper/EMDHQUA2

@misc{pith2026250516590,
  author       = {Pith},
  title        = {Pith review of: Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMDHQUA2}},
  note         = {Machine review of arXiv:2505.16590}
}
read the original abstract

Developers use logging statements to create logs that document system behavior and aid in software maintenance. As such, high-quality logging is essential for effective maintenance; however, manual logging often leads to errors and inconsistency. Recent methods emphasize using large language models (LLMs) for automated logging statement generation, but these present privacy and resource issues, hindering their suitability for enterprise use. This paper presents the first large-scale empirical study evaluating small open-source language models (SOLMs) for automated logging statement generation. We evaluate four prominent SOLMs using various prompt strategies and parameter-efficient fine-tuning techniques, such as Low-Rank Adaptation (LoRA) and Retrieval-Augmented Generation (RAG). Our results show that fine-tuned SOLMs with LoRA and RAG prompts, particularly Qwen2.5-coder-14B, outperform existing tools and LLM baselines in predicting logging locations and generating high-quality statements, with robust generalization across diverse repositories. These findings highlight SOLMs as a privacy-preserving, efficient alternative for automated logging.

Figures

Figures reproduced from arXiv: 2505.16590 by the authors.

Figure 1
Figure 1. Task formulation: given a method which missing a logging statement, the model is asked to automated [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The overview of our experimental design with four research questions. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The prompt template for automated logging. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Overlap of Correctly Logging Statement Placement Across Different SOLM Configurations. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Performance Comparison of Base and Instruct Mistral-7B Models Across Five Epochs. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Distribution of LLM Judges Scores for Generated Logging Statement Quality Across Models. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: A Case of Generated Logging Statement from Multiple Models. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: A Case Study on the Failure to Invoke Out-of-Context Methods in Log Generation. [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]
Figure 9
Figure 9. Figure 9: A Case Study on Distraction by Normal Execution Flow in Complex Methods during Log Generation. [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReLog: Execution-Aware Logging with Runtime Feedback for LLM-Oriented Debugging

    cs.SE 2026-03 conditional novelty 6.0 of 10

    ReLog iteratively writes and rewrites logging statements guided by runtime feedback, and its logs beat static logging baselines on Defects4J debugging tasks.

  2. CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A two-stage detector-plus-LLM-fixer trained on a new, LLM-filtered dataset reports state-of-the-art code-comment inconsistency detection (F1 89.54%) and 18.84% relative GLEU gain in repair.

  3. Approximating the universal thermal climate index using sparse regression with orthogonal polynomials

    physics.ao-ph 2025-08 unverdicted novelty 5.0 of 10

    The paper reports that sparse regression with Legendre polynomials yields UTCI approximations with lower mean error, lower root-mean-square error, and fewer large errors than the standard sixth-degree polynomial.

Reference graph

Works this paper leans on

98 extracted references · 48 canonical work pages · cited by 3 Pith papers

  1. [1]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified Pre-training for Program Understanding and Generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) . 2655–2668

    2021

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS) 33 (2020), 1877–1901

    2020

  3. [3]

    Boyuan Chen and Zhen Ming Jiang. 2017. Characterizing and Detecting Anti-Patterns in the Logging Code. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) . IEEE, Buenos Aires, 71–81. https: //doi.org/10.1109/ICSE.2017.15

    doi:10.1109/icse.2017.15 2017

  4. [4]

    Boyuan Chen and Zhen Ming Jiang. 2019. Extracting and studying the Logging-Code-Issue-Introducing changes in Java-based large-scale open source software systems. Empirical Software Engineering (EMSE) 24 (2019), 2285–2322

    2019

  5. [5]

    Boyuan Chen and Zhen Ming Jiang. 2021. A survey of software log instrumentation. ACM Computing Surveys (CSUR) 54, 4 (2021), 1–34

    2021

  6. [6]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (FSE) . 572–576

    2024

  7. [7]

    Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawardana. 2023. Reducing the Carbon Impact of Generative AI Inference (today and in 2035). In Proceedings of the 2nd workshop on sustainable computer systems. 1–7

    2023

  8. [8]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems (NeurIPS) 36 (2023), 10088–10115

    2023

Show all 98 references
  1. [9]

    Zishuo Ding, Heng Li, and Weiyi Shang. 2022. LoGenText: Automatically Generating Logging Texts Using Neural Machine Translation. In 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, Honolulu, HI, USA, 349–360. https://doi.org/...

    2022

  2. [10]

    Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning . PMLR, 10421–10430

    2023

  3. [11]

    Shuzheng Gao, Cuiyun Gao, Wenchao Gu, and Michael Lyu. 2024. Search-based llms for code optimization. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 254–266

    2024

  4. [12]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao

  5. [13]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al . 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  6. [14]

    translation

    David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. 2020. Code to comment" translation" data, metrics, baselining & evaluation. InProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ICSE). 746–757

    2020

  7. [15]

    Shenghui Gu, Guoping Rong, He Zhang, and Haifeng Shen. 2022. Logging practices in software engineering: A systematic mapping study. IEEE transactions on software engineering (TSE) 49, 2 (2022), 902–923

    2022

  8. [16]

    Wenwei Gu, Renyi Zhong, Guangba Yu, Xinying Sun, Jinyang Liu, Yintong Huo, Zhuangbin Chen, Jianping Zhang, Jiazhen Gu, Yongqiang Yang, et al. 2025. KPIRoot+: An Efficient Integrated Framework for Anomaly Detection and Root Cause Analysis in Large-Scale Cloud Systems. arXiv pre...

    2025 arXiv

  9. [17]

    Mehran Hassani, Weiyi Shang, Emad Shihab, and Nikolaos Tsantalis. 2018. Studying and detecting log-related issues. Empirical Software Engineering (EMSE) 23 (2018), 3248–3280

    2018

  10. [18]

    Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM computing surveys (CSUR) 54, 6 (2021), 1–37

    2021

  11. [19]

    Shilin He, Xu Zhang, Pinjia He, Yong Xu, Liqun Li, Yu Kang, Minghua Ma, Yining Wei, Yingnong Dang, Saravanakumar Rajmohan, et al. 2022. An empirical study of log analysis at microsoft. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium ...

    2022

  12. [20]

    Yi Wen Heng, Zeyang Ma, Zhenhao Li, Dong Jae Kim, et al. 2024. Studying and Benchmarking Large Language Models For Log Level Suggestion. arXiv preprint arXiv:2410.08499 (2024)

    2024 arXiv

  13. [21]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning. PMLR, 2790–2799

    2019

  14. [22]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al

  15. [23]

    Junjie Huang, Zhihan Jiang, Jinyang Liu, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Cong Feng, Hui Dong, Zengyin Yang, and Michael R Lyu. 2024. Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis. In 2024 IEEE 35th International Symposium on ...

    2024

  16. [24]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-Coder Technical Report. arXiv preprint arXiv:2409.12186 (2024)

    2024 arXiv

  17. [25]

    Aaron Imani, Iftekhar Ahmed, and Mohammad Moshirpour. 2024. Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation. arXiv preprint arXiv:2408.02502 (2024)

    2024 arXiv

  18. [26]

    Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. 2022. Codefill: Multi-token code completion by jointly learning from structure and naming sequences. In Proceedings of the 44th international conference on software engineering (ICSE) . 401–412

    2022

  19. [27]

    Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language models for code completion: A practical evaluation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE) . 1–13

    2024

  20. [28]

    Zhouyang Jia, Shanshan Li, Xiaodong Liu, Xiangke Liao, and Yunhuai Liu. 2018. SMARTLOG: Place Error Log Statement by Deep Understanding of Log Intention. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, Campobasso, 61–...

    2018

  21. [29]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  22. [30]

    Zhihan Jiang, Junjie Huang, Zhuangbin Chen, Yichen Li, Guangba Yu, Cong Feng, Yongqiang Yang, Zengyin Yang, and Michael R Lyu. 2025. L4: Diagnosing large-scale llm training failures via automated log analysis. arXiv preprint arXiv:2503.20263 (2025)

    2025 arXiv

  23. [31]

    Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2024. LILAC: Log Parsing using LLMs with Adaptive Parsing Cache.Proceedings of the ACM on Software , Vol. 1, No. 1, Article . Publication date: September ...

    2024

  24. [32]

    Suhas Kabinna, Weiyi Shang, Cor-Paul Bezemer, and Ahmed E. Hassan. 2016. Examining the Stability of Logging Statements. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER) . IEEE, Suita, 326–337. https://doi.org/10.1109/SANER.2016.29

    2016 doi

  25. [33]

    Arjun Kharpal. 2023. Samsung bans use of AI like ChatGPT for staff after misuse of chatbot.CNBC (2 May 2023). https: //www.cnbc.com/2023/05/02/samsung-bans-use-of-ai-like-chatgpt-for-staff-after-misuse-of-chatbot.html Accessed: 2025-03-21

    2023

  26. [34]

    Sangeeta Lal, Neetu Sardana, and Ashish Sureka. 2015. Two level empirical study of logging statements in open source java projects. International Journal of Open Source Software and Processes (IJOSSP) 6, 1 (2015), 49–73

    2015

  27. [35]

    Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)

    2021 arXiv

  28. [36]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  29. [37]

    Heng Li, Tse-Hsun Chen, Weiyi Shang, and Ahmed E. Hassan. 2018. Studying Software Logging Using Topic Models. Empirical Software Engineering (EMSE) 23, 5 (2018), 2655–2694. https://doi.org/10.1007/s10664-018-9595-8

    2018 doi

  30. [38]

    Heng Li, Weiyi Shang, Bram Adams, Mohammed Sayagh, and Ahmed E. Hassan. 2021. A Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives. IEEE Transactions on Software Engineering (TSE) 47, 12 (2021), 2858–2873. https://doi.org/10.1109/TSE.2020.2970422

    2021

  31. [39]

    Heng Li, Weiyi Shang, and Ahmed E. Hassan. 2017. Which Log Level Should Developers Choose for a New Logging Statement? Empirical Software Engineering (EMSE) 22, 4 (2017), 1684–1716. https://doi.org/10.1007/s10664-016-9456-2

    2017 doi

  32. [40]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161 (2023)

    2023 arXiv

  33. [41]

    Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lo...

    2021

  34. [42]

    Briand, and Michael R

    Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, Lionel C. Briand, and Michael R. Lyu

  35. [43]

    Yichen Li, Yintong Huo, Renyi Zhong, Zhihan Jiang, Jinyang Liu, Junjie Huang, Jiazhen Gu, Pinjia He, and Michael R Lyu. 2024. Go static: Contextualized logging statement generation. Proceedings of the ACM on Software Engineering (FSE) 1, FSE (2024), 609–630

    2024

  36. [44]

    Zhenhao Li, Tse-Hsun (Peter) Chen, and Weiyi Shang. 2020. Where Shall We Log?: Studying and Suggesting Logging Locations in Code Blocks. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). ACM, Virtual Event Australia, 361–372....

    2020

  37. [45]

    IEEE Transactions on Software Engineering (TSE) 50, 12 (2024), 3188–3207

    Exploring the Effectiveness of LLMs in Automated Logging Statement Generation: An Empirical Study. IEEE Transactions on Software Engineering (TSE) 50, 12 (2024), 3188–3207. https://doi.org/10.1109/TSE.2024.3475375

    2024

  38. [46]

    Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? studying and exploring variable-aware log abstraction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 830–842

    2023

  39. [47]

    Bo Lin, Shangwen Wang, Ming Wen, Liqian Chen, and Xiaoguang Mao. 2024. One size does not fit all: Multi-granularity patch generation for better automated program repair. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 1554–1566

    2024

  40. [48]

    Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. DeepLV: Suggesting Log Levels Using Ordinal Based Neural Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, Madrid, ES, 1461–1472. https://doi.org/10.1109/ICSE43902.2021.00131

    2021

  41. [49]

    Jiahao Liu, Jun Zeng, Xiang Wang, Kaihang Ji, and Zhenkai Liang. 2022. TeLL: Log Level Suggestions via Modeling Multi-Level Code Block Information. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, Virtual South Korea...

    2022

  42. [50]

    Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yuhang Chen, Yanqing Zhao, Hao Yang, and Yanfei Jiang. 2024. Interpretable online log analysis using large language models with prompt strategies. In Proceedings of the 32nd IEEE/ACM International Conference on Progr...

    2024

  43. [51]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  44. [52]

    Hassan, and Shanping Li

    Zhongxin Liu, Xin Xia, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2019. Which Variables Should I Log? IEEE Transactions on Software Engineering (TSE) (2019), 1–1. https://doi.org/10.1109/TSE.2019.2941943

    2019

  45. [53]

    Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. 2023. Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 647–658

    2023

  46. [54]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. In Proceedings of the IEEE/ACM 46th international c...

    2024

  47. [55]

    Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using deep learning to generate complete log statements. In Proceedings of the 44th International Conference on Software Engineering (ICSE) . 2279–2290

    2022

  48. [56]

    Tsuyoshi Mizouchi, Kazumasa Shimari, Takashi Ishio, and Katsuro Inoue. 2019. PADLA: A Dynamic Log Level Adapter Using Online Phase Detection. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, Montreal, QC, Canada, 135–138. https://doi.org/1...

    2019

  49. [57]

    Antonio Mastropaolo, Valentina Ferrari, Luca Pascarella, and Gabriele Bavota. 2024. Log statements generation via deep learning: Widening the support provided to developers. Journal of Systems and Software (JSS) 210 (2024), 111947

    2024

  50. [58]

    OpenAI. 2025. Advanced usage: Reproducible outputs. https://platform.openai.com/docs/advanced-usage/reproducible- outputs. Accessed: 2025-07-30

    2025

  51. [59]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. I...

    2024

  52. [60]

    Modal Labs, Inc. 2024. Modal. https://modal.com/

    2024

  53. [61]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21, 140 (2020), 1–67

    2020

  54. [62]

    Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, Michael R. Lyu. 2025. The replication package. https://anonymous.4open.science/r/Logging_Empirical_Study-9E54/

    2025

  55. [63]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (ACL) . 311–318

    2002

  56. [64]

    Guoping Rong, Yongda Yu, Song Liu, Xin Tan, Tianyi Zhang, Haifeng Shen, and Jidong Hu. 2024. Code Comment Inconsistency Detection and Rectification Using a Large Language Model. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Socie...

    2024

  57. [65]

    Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing automatic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (F...

    2021

  58. [66]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval 3, 4 (2009), 333–389

    2009

  59. [67]

    André Silva, Sen Fang, and Martin Monperrus. 2023. Repairllama: Efficient representations and fine-tuned adapters for program repair. arXiv preprint arXiv:2312.15698 (2023)

    2023 arXiv

  60. [68]

    Jacopo Soldani and Antonio Brogi. 2022. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Computing Surveys (CSUR) 55, 3 (2022), 1–39

    2022

  61. [69]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al . 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  62. [70]

    Boyin Tan, Junjielong Xu, Zhouruixing Zhu, and Pinjia He. [n. d.]. AL-Bench: A Benchmark for Automatic Logging . https://doi.org/10.48550/arXiv.2502.03160 arXiv:2502.03160 [cs]

    doi arXiv

  63. [71]

    Zhao Tian, Honglin Shu, Dong Wang, Xuejie Cao, Yasutaka Kamei, and Junjie Chen. 2024. Large Language Models for Equivalent Mutant Detection: How Far Are We?. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 1733–1745

    2024

  64. [72]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. 2024. Llm4vuln: A unified evaluation framework for decoupling and enhancing llms’ vulnerability reasoning. arXiv preprint arXiv:2401.16185 (2024)

    2024 arXiv

  65. [73]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. [n. d.]. CodeT5: Identifier-aware Unified Pre-trained Encoder- Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (Online and P...

    2021 doi

  66. [74]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems (NeurIPS) 35 (2022), 24824–24837

    2022

  67. [75]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering. arXiv preprint arXiv:2502.06193 (2025)

    2025 arXiv

  68. [76]

    Xiaoyuan Xie, Zhipeng Cai, Songqiang Chen, and Jifeng Xuan. 2024. FastLog: An End-to-End Method to Efficiently Generate and Insert Logging Statements. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 26–37

    2024

  69. [77]

    Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, et al. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICS...

    2024

  70. [78]

    Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations...

    2023

  71. [79]

    Guang Yang, Yu Zhou, Xiang Chen, Xiangyu Zhang, Terry Yue Zhuo, and Taolue Chen. 2024. Chain-of-thought in neural code generation: From and for lightweight language models. IEEE Transactions on Software Engineering (TSE) (2024)

    2024

  72. [80]

    de Pádua, Weiyi Shang, Steve Sporea, Andrei Toma, and Sarah Sajedi

    Kundi Yao, Guilherme B. de Pádua, Weiyi Shang, Steve Sporea, Andrei Toma, and Sarah Sajedi. 2018. Log4Perf: Suggesting Logging Locations for Web-based Systems’ Performance Monitoring. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering . ACM...

    2018 doi

  73. [81]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  74. [82]

    Ding Yuan, Soyeon Park, Peng Huang, Yang Liu, Michael M Lee, Xiaoming Tang, Yuanyuan Zhou, and Stefan Savage

  75. [83]

    Ding Yuan, Soyeon Park, and Yuanyuan Zhou. 2012. Characterizing Logging Practices in Open-Source Software. In 2012 34th International Conference on Software Engineering (ICSE) . IEEE, Zurich, 102–112. https://doi.org/10.1109/ICSE. 2012.6227202

    2012

  76. [84]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing (2024), 100211

    2024

  77. [85]

    Yuhao Zeng, Jinfu Chen, Weiyi Shang, et al. 2019. Studying the characteristics of logging practices in mobile apps: a case study on F-Droid. Empirical Software Engineering (ESE) 24 (2019), 3394–3434. https://doi.org/10.1007/s10664-019- 09687-9

    2019 doi

  78. [86]

    Chenxi Zhang, Xin Peng, Chaofeng Sha, Ke Zhang, Zhenqing Fu, Xiya Wu, Qingwei Lin, and Dongmei Zhang

  79. [87]

    Jianchen Zhao, Yiming Tang, Sneha Sunil, and Weiyi Shang. 2023. Studying and Complementing the Use of Identifiers in Logs. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 97–107

    2023

  80. [88]

    Ding Yuan, Jing Zheng, Soyeon Park, Yuanyuan Zhou, and Stefan Savage. 2012. Improving Software Diagnosability via Log Enhancement. ACM Transactions on Computer Systems 30, 1 (2012), 1–28. https://doi.org/10.1145/2110356.2110360

    2012

  81. [89]

    Yu Zhao, Lina Gong, Zhiqiu Huang, Yongwei Wang, Mingqiang Wei, and Fei Wu. 2024. Coding-ptms: How to find optimal code pre-trained models for code embedding in vulnerability detection?. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineer...

    2024

  82. [90]

    Renyi Zhong. 2025. Towards Quality Assurance of Natural Language in Code. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 187–189

    2025

  83. [91]

    In Proceedings of the 44th International Conference on Software Engineering (ICSE)

    DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-Based Deep Learning. In Proceedings of the 44th International Conference on Software Engineering (ICSE) . ACM, Pittsburgh Pennsylvania, 623–634. https://doi.org/10.1145/3510003.3510180

  84. [92]

    Rui Zhou, Mohammad Hamdaqa, Haipeng Cai, and Abdelwahab Hamou-Lhadj. 2020. Mobilogleak: A preliminary study on data leakage caused by poor logging practices. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 577–581

    2020

  85. [93]

    Xu Zhao, Kirk Rodrigues, Yu Luo, Michael Stumm, Ding Yuan, and Yuanyuan Zhou. 2017. Log20: Fully Automated Optimal Placement of Log Printing Statements under Specified Overhead Threshold. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP) . ACM, Shangh...

    2017 doi

  86. [96]

    Renyi Zhong, Yichen Li, Jinxi Kuang, Wenwei Gu, Yintong Huo, and Michael R. Lyu. 2025. LogUpdater: Automated Detection and Repair of Specific Defects in Logging Statements. ACM Trans. Softw. Eng. Methodol. (TOSEM) (April 2025). https://doi.org/10.1145/3731754 Just Accepted. , ...

    2025 doi

  87. [98]

    Lyu, and Dongmei Zhang

    Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R. Lyu, and Dongmei Zhang. 2015. Learning to Log: Helping Developers Make Informed Logging Decisions. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (ICSE). IEEE, Florence, Italy, 415–425. htt...

    2015 doi

  88. [2012]

    In 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12)

    Be Conservative: Enhancing Failure Diagnosis with Proactive Logging. In 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12) . 293–306

  89. [2022]

    ICLR 1, 2 (2022), 3

    Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3

    2022

  90. [2024]

    In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE)

    Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE) . 1–13

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.