Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

ReLog shows that logging statements generated through an execution-feedback loop — generate, run, evaluate, refine — are substantially more useful for LLM-based debugging than logs from static single-pass generators.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 05:35 UTC pith:CVUBAKET

load-bearing objection Real reframing of logging evaluation toward downstream debugging utility, with a plausible framework and consistent gains; but the evaluation oracle is under-specified and variance is unreported. the 4 major comments →

arxiv 2603.29122 v2 pith:CVUBAKET submitted 2026-03-31 cs.SE

ReLog: Execution-Aware Logging with Runtime Feedback for LLM-Oriented Debugging

Xin Wang , Yang Feng , Xiaoqian Jiao , Yang Zhang , Zhenhao Li , Zishuo Ding This is my paper
classification cs.SE
keywords logging statement generationLLM-powered debuggingruntime feedbackiterative refinementdefect localizationprogram repairlog utility evaluationexecution-aware logging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ReLog claims that automatic logging should be driven by how useful the resulting runtime logs are for downstream tasks, not by how closely they resemble developer-written statements. It builds a closed loop in which an LLM writes logging statements, the program is compiled and executed, an LLM critic scores the produced logs on traceability, state visibility, and causal linkage, and a refiner adjusts the statements until the logs are judged sufficient. Tested as an aid for defect localization and repair on real-world Java bugs, ReLog reports F1 scores of 0.520 (source available) and 0.408 (source hidden), and repairs 97 defects, consistently beating baselines that generate logging in a single static pass. The paper argues this reframes logging generation as a runtime-guided, task-oriented process and that evaluation should follow suit.

Core claim

The central claim is that logging statements are instruments to be refined against actual runtime behavior, not static code decorations to be matched to human preferences. ReLog operationalizes this as an iterative loop: it first generates logging statements from the code plus the observed execution outcome, then repairs any compilation errors the inserted statements introduce (keeping the original logic untouched), executes the instrumented program, and asks an LLM critic whether the logs expose the execution path, key variable states, and the causal chain behind the failure. If the critic finds observability gaps, a refiner modifies the logging list and the cycle repeats. In the paper's ev

What carries the argument

The load-bearing mechanism is a closed-loop refinement cycle orchestrated around three LLM components working on a discrete list of logging statements kept separate from the program source. After initial generation (conditioned on the code and its first execution outcome), a compilation-repair agent fixes errors that the inserted statements introduce, guaranteeing the instrumented program runs. A critic then judges the resulting logs along three axes — traceability (does the log expose the relevant execution path), state visibility (are key variables recorded), and causal linkage (does the log explain why the failure occurred) — and returns actionable feedback. A refiner applies that feedbac

Load-bearing premise

The evaluation treats LLM-based matching of a generated defect description to ground truth, and of a candidate patch to the ground-truth fix, as a valid oracle, but the paper does not say who or what performs that matching or how disagreements are resolved; if the oracle shares the critic/refiner's biases, part of ReLog's advantage could reflect optimization to the evaluator's preferences rather than genuinely better logs.

What would settle it

Have human experts adjudicate the same 311 direct-debugging and 225 indirect-debugging instances that ReLog and a strong static baseline both process — deciding which generated defect descriptions match ground truth and which patches are semantically equivalent — then recompute precision, recall, and F1. If ReLog's reported margin over baselines shrinks under human judgment, the framework's edge is in large part an artifact of an LLM scoring itself; if it holds, the logs are genuinely more useful for debugging.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Logging evaluation should move from textual similarity to developer-written logs toward measuring how much the produced logs improve downstream tasks such as debugging.
  • Iterative refinement is not optional: disabling the feedback loop drops defect-localization F1 from 0.520 to 0.388 in the direct setting.
  • Compilation repair is essential for reliability: without it, compilation failures rise to 94 cases in the direct setting and recall falls from 0.511 to 0.366.
  • The gains come from the framework rather than any single model: four different LLMs inside ReLog all outperform the static baselines.
  • Logs refined under this loop can support debugging even when source code is unavailable, pointing to use in production troubleshooting and operations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because ReLog optimizes against an LLM critic's sufficiency judgment, its logs may be tuned to what today's LLM judges prefer; swapping the critic's rubric or having humans judge the same logs would test whether the gains generalize to other consumers.
  • The repeated compile-and-execute cycle is expensive and offline oriented; a natural next step is measuring log utility against iteration budget to see whether most of the gain comes in the first refinement round.
  • The benchmark covers Java bugs only; applying the same runtime-feedback loop to Python or Go programs would test whether the mechanism transfers across language runtimes and logging ecosystems.
  • ReLog's logs run longer (about 5.5 statements per method) than human-written logs in the comparison, which may raise readability and storage concerns in production; the paper only measures automated debugging utility, not operator overhead.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ReLog, an LLM-based framework for automatic logging statement generation that replaces static, single-pass generation with an iterative loop of generation, execution, compilation repair, LLM-based log sufficiency evaluation, and refinement based on runtime feedback. The framework is evaluated not by similarity to developer-written logs but by downstream utility in defect localization and repair on two Defects4J-derived datasets: a direct setting (faulty code plus logs) with 311 instances and an indirect setting (logs and caller context only) with 225 instances. The reported results claim consistent gains over baselines, including a direct-setting F1 of 0.520 and 97 repairs, an indirect-setting F1 of 0.408, robust performance across four LLMs, and ablations supporting the value of both compilation repair and iterative refinement.

Significance. If the quantitative claims hold, the paper makes a useful contribution: it re-frames logging generation as an execution-aware, task-oriented process and introduces a downstream-utility evaluation methodology that is more meaningful than text similarity to human-written logs. The replication package is a strength, as are the ablations and the cross-LLM experiments. However, the empirical claims currently rest on an underspecified evaluation oracle, single-run stochastic evaluations, and a dataset construction process that is not fully described. These issues must be addressed before the central claim—that ReLog produces logs that are genuinely more useful for LLM-based debugging—can be considered established.

major comments (4)
  1. [§4.1.1 and §4.1.2] The headline F1 and repair counts depend on binary judgments that are never operationalized. Section 4.1.1 says a defect-localization prediction is a True Positive when the generated description 'matches the ground truth' and a repair is successful when the patch is 'identical or semantically equivalent' to the ground-truth fix; Section 4.1.2 uses analogous wording for indirect localization. The paper does not say who or what performs this matching, what prompt, rubric, or threshold is used, whether Defects4J's test suite is executed to validate candidate patches, whether human annotators are involved, or how disagreements are resolved. Because every table in the paper depends on these judgments, the central quantitative claims are not reproducible or independently verifiable. Please specify the matching oracle in detail, release the evaluation artifacts, and, if an LLM judge is used, pr
  2. [§5, Tables 1–3] All results are single-point estimates from stochastic LLM pipelines. DeepSeek-V3 and the other evaluated models are stochastic, yet no repeated runs, confidence intervals, or statistical significance tests are reported. For instance, the direct-setting gap between ReLog (F1 0.520) and UniLog (F1 0.447), and the repair gap of 97 vs. 78, could be within sampling noise. Please report variance over multiple seeds or sampling temperatures and, where appropriate, paired statistical tests for the main comparisons and ablations.
  3. [§4.1.3, §4.4, RQ2] The debugging-agent model is not specified. Section 4.1.3 states only that 'lightweight LLM-based debugging agents' are used; RQ1 says ReLog and baselines are based on DeepSeek-V3, but it does not say whether the downstream localization/repair agent is also DeepSeek-V3. If the same model family is used for the critic/refiner, the matching oracle, and the debugging agent, ReLog's refinement may be optimizing logs to the evaluator's preferences rather than to intrinsic diagnostic quality. The cross-LLM RQ2 does not resolve this unless the downstream judge is varied independently. Please state the exact judge/agent model for every condition and add an evaluation with a different judge model family or human validation.
  4. [§4.2 and §7] The dataset construction is underspecified and may be selective. Section 4.2 reports 311 direct and 225 indirect instances derived from Defects4J, and Section 7 refers to a 'curated Java dataset where the input codebases are inherently executable,' but the paper gives no inclusion/exclusion criteria, no sampling procedure, and no per-instance metadata. Without this, it is impossible to assess whether the benchmarks are representative or whether certain bug types or projects were systematically excluded, which would affect the external validity of the comparisons. Please describe the curation pipeline in detail and make the dataset publicly available with per-instance information.
minor comments (5)
  1. [§4.3 and Table 1] The baseline is listed as 'SCLogger' in Section 4.3 but referred to as 'GoStatic' throughout Section 5 and Table 1; reference [33] is titled 'Go Static.' Please standardize the name.
  2. [§4.1.1] The text contains a broken cross-reference: 'the debugging agent (cf. Section )'—the section number is missing.
  3. [Table 2] The GPT-5-mini row has a formatting/alignment problem: '0.5470.55990' appears where precision/F1/repair values should be clearly separated.
  4. [RQ3] The text says removing the refinement loop 'reduces the framework to a traditional static logging generator,' but Stage 1 still uses an initial execution outcome to guide logging generation; this is not purely static. Please rephrase to avoid overstating the ablation.
  5. [Reference [3]] The replication-package reference title is misspelled: 'Replicaton Package' should be 'Replication Package.'

Circularity Check

0 steps flagged

No demonstrated circularity; unspecified evaluation oracle is a reproducibility threat, not a circular reduction.

full rationale

ReLog's derivation chain does not reduce to its inputs. The framework iteratively refines logs against an LLM critic's rubric (traceability, state visibility, causal linkage), but the reported F1 and repair results are anchored to external Defects4J ground-truth defects and fixes, and the downstream debugging agent's outputs are not by construction identical to the critic's sufficiency judgment. The motivating observation that logging statements are revised iteratively (Figure 2) is a new measurement rather than a self-citation. The authors cite their own prior work (e.g., [15,16,17,47]) in related work and in the motivating-study methodology, but none of these citations is load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The main validity concern is that Section 4.1.1 defines a True Positive as "the generated defect description matches the ground truth" and a successful repair as "identical or semantically equivalent" to the ground-truth fix without specifying who or what performs the matching, whether the Defects4J tests are run, or how disagreements are resolved. If the matcher were the same LLM family (DeepSeek-V3) used in ReLog's critic/refiner loop, some gains could reflect LLM self-preference rather than objectively more useful logs. However, the paper does not state that the matcher is an LLM, and the external Defects4J anchors keep the central claim from being defined into existence. Because no specific equation or fitted parameter reduces the prediction to its input, no circular step is established under the required evidence standard.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claim is empirical rather than derived. The load-bearing assumptions are that LLM judgments (critic sufficiency, ground-truth matching) are reliable probes of log utility, and that Defects4J single-method snippets are a valid proxy for real debugging. These are domain assumptions, not standard mathematics.

free parameters (2)
  • Maximum refinement iterations = 5
    Stage 4 caps the outer loop at five iterations; no sensitivity analysis is reported, and the ablation shows refinement contributes substantially to the gains.
  • Critic rubric dimensions = 3
    The sufficiency critic uses three hand-selected dimensions (traceability, state visibility, causal linkage); changing the rubric would change when refinement stops and what feedback is produced.
axioms (3)
  • domain assumption The LLM-based critic's judgment that logs are 'sufficient' is a reliable proxy for downstream debugging usefulness.
    Stage 3 uses an LLM critic with a three-dimension rubric to terminate refinement; no validation is provided that critic decisions correlate with downstream debugging success.
  • domain assumption LLM-based matching of generated defect descriptions and patches to Defects4J ground truth is a valid oracle.
    Section 4.1 defines True Positive and successful repair by 'matches the ground truth' but never specifies the matching procedure or human verification.
  • domain assumption Defects4J single-fault method snippets retain enough context to measure log utility.
    The direct dataset extracts one faulty method per sample (Section 4.2), which may omit surrounding code and tests that real debugging relies on.

pith-pipeline@v1.3.0-alltime-deepseek · 18912 in / 12398 out tokens · 137412 ms · 2026-08-04T05:35:26.644652+00:00 · methodology

0 comments
read the original abstract

Logging statements are essential for software debugging and maintenance. However, existing approaches to automatic logging generation rely on static analysis and produce statements in a single pass without considering runtime behavior. They are also typically evaluated by similarity to developer-written logs, assuming these logs form an adequate gold standard. This assumption is increasingly limiting in the LLM era, where logs are consumed not only by developers but also by LLMs for downstream tasks. As a result, optimizing logs for human similarity does not necessarily reflect their practical utility. To address these limitations, we introduce ReLog, an iterative logging generation framework guided by runtime feedback. ReLog leverages LLMs to generate, execute, evaluate, and refine logging statements so that runtime logs better support downstream tasks. Instead of comparing against developer-written logs, we evaluate ReLog through downstream debugging tasks, including defect localization and repair. We construct a benchmark based on Defects4J under both direct and indirect debugging settings. Results show that ReLog consistently outperforms all baselines, achieving an F1 score of 0.520 and repairing 97 defects in the direct setting, and the best F1 score of 0.408 in the indirect setting where source code is unavailable. Additional experiments across multiple LLMs demonstrate the generality of the framework, while ablations confirm the importance of iterative refinement and compilation repair. Overall, our work reframes logging as a runtime-guided, task-oriented process and advocates evaluating logs by their downstream utility rather than textual similarity.

Figures

Figures reproduced from arXiv: 2603.29122 by Xiaoqian Jiao, Xin Wang, Yang Feng, Yang Zhang, Zhenhao Li, Zishuo Ding.

Figure 1
Figure 1. Figure 1: The iterative refinement process of a logging state [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of change frequencies for logging state [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The overall architecture of ReLog with a two-round example. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: A comparative case study of logging generation approaches on a representative defect. The figure illustrates the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs

    cs.SE 2026-04 unverdicted novelty 7.0

    MultiLogBench shows that LLM performance on automated logging varies substantially across programming languages, demonstrating that single-language evidence is insufficient for general claims about model behavior or t...

Reference graph

Works this paper leans on

63 extracted references · 7 canonical work pages · cited by 1 Pith paper · 5 internal anchors

  1. [1]

    [n. d.]. HBASE-24367. https://issues.apache.org/jira/browse/HBASE-24367

  2. [2]

    [n. d.]. Ollama. https://ollama.com

  3. [3]

    [n. d.]. Replicaton Package. https://github.com/klsc749/ReLog

  4. [4]

    Introducing GPT-5

    2026. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/

    2026

  5. [5]

    Titus Barik, Robert DeLine, Steven Drucker, and Danyel Fisher. 2016. The Bones of the System: A Case Study of Logging and Telemetry at Microsoft. In2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)

    2016

  6. [6]

    Ruisheng Cao, Mouxiang Chen, Jiawei Chen, Zeyu Cui, Yunlong Feng, Binyuan Hui, Yuheng Jing, Kaixin Li, Mingze Li, Junyang Lin, Zeyao Ma, Kashun Shum, Xuwu Wang, Jinxi Wei, Jiaxi Yang, Jiajun Zhang, Lei Zhang, Zongmeng Zhang, Wenting Zhao, and Fan Zhou. 2026. Qwen3-Coder-Next Technical Report. doi:10. 48550/arXiv.2603.00729

    Pith/arXiv arXiv doi:10.48550/arxiv.2603.00729 2026

  7. [7]

    An Ran Chen. 2019. An Empirical Study on Leveraging Logs for Debugging Production Failures. In2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). 126–128

    2019

  8. [8]

    An Ran Chen, Tse-Hsun Chen, and Shaowei Wang. 2022. Pathidea: Improving Information Retrieval-Based Bug Localization by Re-Constructing Execution Paths Using Logs.IEEE Transactions on Software Engineering48, 8 (Aug. 2022), 2905–2919. doi:10.1109/TSE.2021.3071473

    arXiv 2022

  9. [9]

    Boyuan Chen and Zhen Ming Jiang. 2017. Characterizing and Detecting Anti- Patterns in the Logging Code. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

    2017

  10. [10]

    Boyuan Chen and Zhen Ming Jiang. 2019. Extracting and studying the Logging- Code-Issue- Introducing changes in Java-based large-scale open source software systems.Empirical Softw. Engg.(2019)

    2019

  11. [11]

    Yi-Hung Chou, Yiyang Min, April Yi Wang, and James A. Jones. 2025. Learn- ing from Mistakes: Understanding Ad-hoc Logs through Analyzing Accidental Commits. doi:10.48550/arXiv.2501.09892

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.09892 2025

  12. [12]

    Yaxun Dai, Wenxuan Xie, Xialie Zhuang, Tianyu Yang, Yiying Yang, Haiqin Yang, Yuhang Zhao, Pingfu Chao, and Wenhao Jiang. 2025. ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQL. doi:10.48550/arXiv. 2505.12768

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.12768 2025

  13. [13]

    DeepSeek-AI, Daya Guo, Dejian Yang, and et al. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    2025

  14. [14]

    Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2024. TRACED: Execution-aware Pre-training for Source Code. InProceedings of the IEEE/ACM 46th International Conference on Software Engi- neering (ICSE ’24). Association for Computing Machinery, New York, NY, USA, 1–12. doi:10.1145/3597503.3608140

    arXiv 2024

  15. [15]

    Zishuo Ding, Heng Li, and Weiyi Shang. 2022. LoGenText: Automatically Gener- ating Logging Texts Using Neural Machine Translation. In2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)

    2022

  16. [16]

    Zishuo Ding, Yiming Tang, Xiaoyu Cheng, Heng Li, and Weiyi Shang. 2023. LoGenText-Plus: Improving Neural Machine Translation Based Logging Texts Generation with Syntactic Templates.ACM Trans. Softw. Eng. Methodol.(2023)

    2023

  17. [17]

    Zishuo Ding, Yiming Tang, Yang Li, Heng Li, and Weiyi Shang. 2023. On the Tem- poral Relations between Logging and Code. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

    2023

  18. [18]

    Patrick Loic Foalem, Foutse Khomh, and Heng Li. 2023. Studying Logging Practice in Machine Learning-based Applications. doi:10.48550/arXiv.2301.04234

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2301.04234 2023

  19. [19]

    Rui Guo, Aijia Sun, and Ying Xie. 2025. Execution-Aware Hierarchical Code Generation with Qwen-72B and Retrieval Augmentation. InProceedings of the 2025 International Symposium on Machine Learning and Social Computing (MLSC ’25). Association for Computing Machinery, New York, NY, USA, 417–422. doi:10. 1145/3778450.3778516

    arXiv 2025

  20. [20]

    Mirazul Haque, Petr Babkin, Farima Farmahinifarahani, and Manuela Veloso

  21. [21]

    Mehran Hassani, Weiyi Shang, Emad Shihab, and Nikolaos Tsantalis. 2018. Study- ing and detecting log-related issues.Empirical Softw. Engg.(2018)

    2018

  22. [22]

    Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2021. A Survey on Automated Log Analysis for Reliability Engineering. ACM Comput. Surv.(2021)

    2021

  23. [23]

    Yi Wen Heng, Zeyang Ma, Zhenhao Li, Dong Jae Kim, and Tse-Hsun Chen. 2025. Benchmarking Open-Source Large Language Models for Log Level Suggestion. In2025 IEEE Conference on Software Testing, Verification and Validation (ICST). 314–325

    2025

  24. [24]

    Cheng Ji and Huaiying Luo. 2025. Leveraging Large Language Model for In- telligent Log Processing and Autonomous Debugging in Cloud AI Platforms. https://arxiv.org/abs/2506.17900v1

    Pith/arXiv arXiv 2025

  25. [25]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? doi:10.48550/arXiv.2310.06770

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.06770 2024

  26. [26]

    René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437–

    2014

  27. [27]

    Jinhan Kim, Valeriy Savchenko, Kihyuck Shin, Konstantin Sorokin, Hyunseok Jeon, Georgiy Pankratenko, Sergey Markov, and Chul-Joo Kim. 2020. Automatic abnormal log detection by analyzing log history for providing debugging in- sight. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-...

    2020

  28. [28]

    Youngjoon Kim, Sunguk Shin, Hyoungshick Kim, and Jiwon Yoon. 2025. Logs in, Patches out: Automated Vulnerability Repair via Tree-of-Thought LLM Analysis. InProceedings of the 34th USENIX Conference on Security Symposium. Number

    2025

  29. [29]

    Boaz Lavon, Shahar Katz, and Lior Wolf. 2025. Execution Guided Line-by-Line Code Generation. doi:10.48550/arXiv.2506.10948

    doi:10.48550/arxiv.2506.10948 2025

  30. [30]

    Cuong Chi Le, Cuong Duc Van, Tung Duy Vu, Thai Minh Pham Vu, Hoang Nhat Phan, Huy Nhat Phan, and Tien N. Nguyen. 2026. TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models. doi:10.48550/arXiv.2508.01255

    work page doi:10.48550/arxiv.2508.01255 2026

  31. [31]

    Heng Li, Weiyi Shang, Bram Adams, Mohammed Sayagh, and Ahmed E. Hassan

  32. [32]

    Yichen Li, Yintong Huo, Zhihan Jiang, Renyi Zhong, Pinjia He, Yuxin Su, Lionel C Briand, and Michael R Lyu. 2024. Exploring the effectiveness of llms in automated logging statement generation: An empirical study.IEEE Transactions on Software Engineering(2024)

    2024

  33. [33]

    Yichen Li, Yintong Huo, Renyi Zhong, Zhihan Jiang, Jinyang Liu, Junjie Huang, Jiazhen Gu, Pinjia He, and Michael R. Lyu. 2024. Go Static: Contextualized Logging Statement Generation.Proc. ACM Softw. Eng.(2024)

    2024

  34. [34]

    Zhenhao Li, An Ran Chen, Xing Hu, Xin Xia, Tse-Hsun Chen, and Weiyi Shang

  35. [35]

    Zhenhao Li, Tse-Hsun Chen, and Weiyi Shang. 2020. Where Shall We Log? Study- ing and Suggesting Logging Locations in Code Blocks. In2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    2020

  36. [36]

    Zhenhao Li, Tse-Hsun Chen, Jinqiu Yang, and Weiyi Shang. 2019. DLFinder: Characterizing and Detecting Duplicate Logging Code Smells. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

    2019

  37. [37]

    Zhenhao Li, Heng Li, Tse-Hsun Chen, and Weiyi Shang. 2021. DeepLV: Suggest- ing Log Levels Using Ordinal Based Neural Networks. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

    2021

  38. [38]

    Zhenhao Li, Chuan Luo, Tse-Hsun Chen, Weiyi Shang, Shilin He, Qingwei Lin, and Dongmei Zhang. 2023. Did we miss something important? studying and Xin Wang, Feng Yang, Xiaoqian Jiao, Yang Zhang, Zhenhao Li, and Zishuo Ding exploring variable-aware log abstraction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 830–842

    2023

  39. [39]

    Hassan, and Shanping Li

    Zhongxin Liu, Xin Xia, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. 2021. Which Variables Should I Log?IEEE Transactions on Software Engineering47, 9 (Sept. 2021), 2012–2031. doi:10.1109/TSE.2019.2941943

    arXiv 2021

  40. [40]

    Antonio Mastropaolo, Valentina Ferrari, Luca Pascarella, and Gabriele Bavota

  41. [41]

    Antonio Mastropaolo, Luca Pascarella, and Gabriele Bavota. 2022. Using Deep Learning to Generate Complete Log Statements. doi:10.48550/arXiv.2201.04837

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2201.04837 2022

  42. [42]

    Federico Di Menna, Luca Traini, Gabriele Bavota, and Vittorio Cortellessa. 2025. Investigating Execution-Aware Language Models for Code Optimization. doi:10. 48550/arXiv.2503.08228

    work page doi:10.48550/arxiv.2503.08228 2025

  43. [43]

    Fredrik Milani and Fabrizio M. Maggi. 2018. A Comparative Evaluation of Log- Based Process Performance Analysis Techniques. InBusiness Information Systems (2018)

    2018

  44. [44]

    Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. NExT: Teaching Large Language Models to Reason about Code Execution. doi:10.48550/arXiv.2404.14662

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.14662 2024

  45. [45]

    Boyin Tan, Junjielong Xu, Zhouruixing Zhu, and Pinjia He. 2025. AL-Bench: A Benchmark for Automatic Logging. doi:10.48550/arXiv.2502.03160

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.03160 2025

  46. [46]

    Log Statements Generation via Deep Learning: Widening the Support Provided to Developers

    Log Statements Generation via Deep Learning: Widening the Support Provided to Developers. doi:10.48550/arXiv.2311.04587

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.04587

  47. [47]

    Xin Wang, Zhenhao Li, and Zishuo Ding. 2025. Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning. https://arxiv.org/abs/ 2508.11305v1

    Pith/arXiv arXiv 2025

  48. [48]

    Fengcai Wen, Csaba Nagy, Michele Lanza, and Gabriele Bavota. 2020. An Empir- ical Study of Quick Remedy Commits. InProceedings of the 28th International Conference on Program Comprehension (ICPC ’20). Association for Computing Machinery, New York, NY, USA, 60–71. doi:10.1145/3387904.3389266

    arXiv 2020

  49. [49]

    Xiaoyuan Xie, Zhipeng Cai, Songqiang Chen, and Jifeng Xuan. 2024. FastLog: An End-to-End Method to Efficiently Generate and Insert Logging Statements. doi:10.48550/arXiv.2311.02862

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.02862 2024

  50. [50]

    Junjielong Xu, Ziang Cui, Yuan Zhao, Xu Zhang, Shilin He, Pinjia He, Liqun Li, Yu Kang, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2024. UniLog: Automatic Logging via LLM and In-Context Learning. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machinery, New Y...

    arXiv 2024

  51. [51]

    Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qing- wei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. 2025. Open- RCA: Can Large Language Models Locate the Root Cause of Software Fail- ures?. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=M4qNIzQYpd

    2025

  52. [52]

    GLM-V. Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, Baoxu Wang, Bin Chen, Boyan Shi, Changyu Pang, Chenhui Zhang, Da Yin, Fan Yang, Guoqing Chen, Haochen Li, Jiale Zhu, Jiali Che...

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.01006 2026

  53. [53]

    Ding Yuan, Soyeon Park, and Yuanyuan Zhou. 2012. Characterizing logging prac- tices in open-source software. In2012 34th International Conference on Software Engineering (ICSE)

    2012

  54. [54]

    Renyi Zhong, Yintong Huo, Wenwei Gu, Yichen Li, and Michael R. Lyu. 2025. End-to-End Automated Logging via Multi-Agent Framework. doi:10.48550/arXiv. 2511.18528

    doi:10.48550/arxiv.2511.18528 2025

  55. [55]

    Renyi Zhong, Yichen Li, Jinxi Kuang, Wenwei Gu, Yintong Huo, and Michael R. Lyu. 2025. LogUpdater: Automated Detection and Repair of Specific Defects in Logging Statements.ACM Trans. Softw. Eng. Methodol.(2025)

    2025

  56. [56]

    Renyi Zhong, Yichen Li, Guangba Yu, Wenwei Gu, Jinxi Kuang, Yintong Huo, and Michael R Lyu. 2025. Beyond LLMs: An Exploration of Small Open-source Lan- guage Models in Logging Statement Generation.arXiv preprint arXiv:2505.16590 (2025)

    Pith/arXiv arXiv 2025

  57. [57]

    Lyu, and Dongmei Zhang

    Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R. Lyu, and Dongmei Zhang. 2015. Learning to Log: Helping Developers Make Informed Logging Decisions. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering

    2015

  58. [58]

    Ding Yuan, Haohui Mai, Weiwei Xiong, Lin Tan, Yuanyuan Zhou, and Shankar Pasupathy. 2010. SherLog: error diagnosis by connecting clues from run-time logs. InProceedings of the fifteenth International Conference on Architectural support for programming languages and operating systems

    2010

  59. [227]

    USENIX Association, USA, 4401–4419

  60. [440]

    doi:10.1145/2610384.2628055

    arXiv

  61. [2021]

    A Qualitative Study of the Benefits and Costs of Logging From Developers’ Perspectives.IEEE Transactions on Software Engineering(2021)

    2021

  62. [2023]

    Are They All Good? Studying Practitioners’ Expectations on the Readability of Log Messages

  63. [2025]

    https://arxiv.org/abs/2505.04441v1

    Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs. https://arxiv.org/abs/2505.04441v1

    Pith/arXiv arXiv