The reviewed record of science sign in
Pith

arxiv: 2607.05785 · v1 · pith:6A5XJ4WB · submitted 2026-07-07 · cs.SE

Can Large Language Models Generate Observability-Aware Code?

Yongliang Tao , Hongyu Zhang , Pengfei Gao , Minghua Ma , Zhiyu Fan , Yu Kang , Jue Zhang , Si Qin
show 3 more authors
Liqun Li Qingwei Lin Saravan Rajmohan
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-09 00:00 UTCglm-5.2pith:6A5XJ4WBrecord.jsonopen to challenge →

classification cs.SE
keywords observabilitycoding agentsfault signallog qualitymicroservicessoftware qualityLLM code generationdiagnostic semantics
0
0 comments X

The pith

AI-Generated Code Fails to Expose Failures: Only 5-14% of Faults Surface in Logs

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether coding agents can generate code that is not only functional but also observable—meaning it produces logs and other artifacts that expose what went wrong during failures. The authors find a persistent and substantial gap: while agents generate abundant logging, the logs rarely contain fault-specific diagnostic information. In 200 microservice systems deployed on Kubernetes with 1,615 injected fault instances, only 4.95% to 13.99% of failures produced explicit fault signals in runtime logs. The problem is not absence of logging but absence of meaningful failure semantics. At the source-code level, agents are better at placing observability statements in the right locations than at capturing the right diagnostic content (error codes, dependency states, timeout values). A lightweight observability skill derived from real engineering commits improved fault signal rates modestly (up to +8.67 percentage points for one model) but left overall rates low, suggesting the gap is structural rather than a simple prompting deficiency.

Core claim

The central finding is a semantic observability gap: coding agents can identify where to place logging and tracing statements but cannot reliably determine what diagnostic information those statements should capture. This gap propagates from source code to runtime—generated systems produce logs that are voluminous but semantically hollow regarding actual failures. Faults that inherently produce explicit errors (crashes, connection refusals) are sometimes caught; faults with implicit effects (CPU stress, clock skew, network corruption) are almost never caught (1-2% detection rates). Lightweight guidance narrows but does not close the gap, indicating that effective observability requires aform

What carries the argument

The paper introduces a two-level evaluation framework connecting source-level diagnostic semantics (what observability artifacts encode) to runtime fault signals (what actually surfaces during failures). The Fault Signal Rate (FSR) metric measures the fraction of injected faults that produce explicit, fault-specific log evidence. The KeyBag F1 metric measures lexical overlap of diagnostic content between agent-generated and human-written observability. Together these metrics separate placement accuracy from content quality, and static capability from runtime behavior.

If this is right

  • Coding agent benchmarks that evaluate only functional correctness are insufficient for production readiness; observability should be treated as a first-class evaluation dimension alongside correctness, security, and performance.
  • The knowledge debt problem identified by the authors—developers maintaining systems whose runtime behavior they do not understand—suggests that agent-generated code may require different operational practices than human-written code, potentially shifting diagnostic burden toward post-hoc tooling rather than built-in observability.
  • The finding that explicit-error faults are detectable but implicit-effect faults are not implies that observability gaps will be most severe for the hardest-to-diagnose failure classes, exactly where observability matters most.
  • The limited effectiveness of lightweight guidance suggests that closing the gap may require runtime feedback loops during generation—agents that can deploy, inject faults, observe their own logging behavior, and iterate—rather than static prompt engineering.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the FSR oracle underestimates actual observability by missing non-standard fault signal patterns, the true gap may be smaller than reported—but the consistency of the gap across models, fault types, and evaluation settings suggests the qualitative finding (agents produce semantically hollow logs) would persist even with a more generous oracle.
  • The quantity-over-quality tradeoff observed under explicit instructions mirrors a broader pattern in LLM code generation: prompting for more output tends to dilute precision. This suggests that observability-oriented fine-tuning or RL-based alignment may be more effective than prompt engineering for improving diagnostic content quality.
  • If observability is rarely an independent objective in training data (as the authors hypothesize), then models fine-tuned on post-incident commits—where logging was explicitly added to diagnose a specific failure—might close the gap more effectively than models trained on general repository code.

Load-bearing premise

The Fault Signal Rate metric relies on fault-specific log evidence oracles that define what counts as a valid fault signal using keyword and pattern matching (for example, requiring 'timeout' or 'latency' in logs for network-delay faults). If these oracles miss valid but non-standard fault signal patterns, the metric underestimates actual observability. The paper's conclusions about the observability gap rest on these oracles being a fair proxy for fault-detectability.

What would settle it

If a coding agent could be shown to generate systems where, under injected faults, a substantial majority (say >70%) of failures produce explicit fault-specific log evidence without human post-hoc instrumentation, the core claim of a persistent semantic observability gap would be weakened.

Figures

Figures reproduced from arXiv: 2607.05785 by Hongyu Zhang, Jue Zhang, Liqun Li, Minghua Ma, Pengfei Gao, Qingwei Lin, Saravan Rajmohan, Si Qin, Yongliang Tao, Yu Kang, Zhiyu Fan.

Figure 1
Figure 1. Figure 1: Overview of Study B. From Diagnostic Semantics to Fault Signals We study observability across two levels. At the source-code level, an observability artifact encodes diagnostic semantics: the failure-relevant context it is intended to capture, such as dependency states, error codes, request identifiers, state transitions, and variables that explain exceptional behavior. At the runtime level, observability … view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation of Observability in Agent-Generated Microservice Systems [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of Static Observability Restoration [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Quantity over Quality under observability-hinted prompt [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative success and failure cases under the same evaluation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
read the original abstract

Recent advances in coding agents have enabled the generation of increasingly complex software systems. While existing evaluations primarily focus on functional correctness, production systems must expose failure evidence to support observability. In this paper, we present a systematic study of observability in agent-generated systems. We examine whether agents can reconstruct source-level diagnostic semantics by restoring observability artifacts in 10 open-source and 8 industrial repositories. We also evaluate whether these artifacts translate into effective fault signals at runtime through 200 generated microservice systems deployed on Kubernetes with 13 injected faults. Our results reveal a consistent gap between diagnostic semantics at the source level and fault signals (i.e., explicit, fault-specific evidence) at runtime. At the source level, agents partially recover observability artifacts but struggle to capture key diagnostic semantics. At runtime, generated systems expose fault signals for only a small fraction of failures (up to 13.99\%), despite the presence of logging, suggesting that the generated observability artifacts may lack the failure-specific semantics needed to effectively expose faults. We further introduce an observability-oriented skill, which can serve as a guidance to improve both diagnostic semantics and fault-signal exposure, but the gains remain limited, indicating that the gap is not easily addressed. More broadly, our findings suggest that current evaluations focusing primarily on functional correctness may overlook observability as an important dimension of practical software quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 7 minor

Summary. This paper investigates whether coding agents can generate observability-aware code, addressing an important gap in existing evaluations that focus primarily on functional correctness. The study is structured around three research questions: (RQ1) whether agents can restore human-written observability artifacts at the source level, (RQ2) whether agent-generated microservice systems expose fault signals under injected failures at runtime, and (RQ3) whether lightweight observability-oriented guidance improves outcomes. The experimental design is systematic: 1,223 static restoration instances from 18 repositories, 200 generated microservice systems deployed on Kubernetes with 13 injected fault types, three frontier models, and five repetitions per experiment. The central finding is that coding agents exhibit a persistent 'semantic observability gap'—they are better at placing observability artifacts than capturing diagnostic semantics, and only 4.95%–13.99% of injected faults produce explicit fault signals in runtime logs. Lightweight guidance yields only marginal improvements.

Significance. The paper addresses a timely and underexplored dimension of code generation quality. The multi-level evaluation framework connecting source-level diagnostic semantics to runtime fault signals is a genuine contribution, as is the Fault Signal Rate (FSR) metric that measures whether generated systems actually expose failures rather than merely producing logging volume. The scale of the runtime evaluation (200 systems, 1,615 fault instances, Kubernetes deployment with Chaos Mesh) is commendable and provides falsifiable, reproducible evidence. The finding that observability should be treated as an independent generation-time objective rather than a by-product of functional code generation has clear implications for benchmark design and agent development.

major comments (4)
  1. §III.C, Table I: The FSR oracle uses keyword/pattern matching to define fault-specific log evidence (e.g., 'timeout/latency/deadline evidence' for F02, 'CPU/load/slow processing/timeout evidence' for F11). The paper acknowledges in §V.B that 'some logs may contain partial or indirect information that does not satisfy the oracle criteria and is therefore counted as no signal.' This is load-bearing because every RQ2 and RQ3 result flows from this oracle. The concern is not generic incompleteness but a specific systematic bias: LLM-generated code may produce logs with non-standard phrasing or indirect fault indicators that do not match oracle patterns derived from human-written systems. The paper should quantify the potential impact—for example, by manually auditing a sample of 'no signal' logs (e.g., 100 instances across fault types) to estimate how many contain valid but non-matchingfault
  2. §III.C, Table III: The FSR is reported as a single aggregate per model, but Table IV shows substantial variation across fault types (from 1.26% for time-skew to 27.45% for upstream-fail). The paper should report per-model FSR by fault type, at least for the baseline (no skill) condition. Without this, it is unclear whether the cross-model consistency claim (that all models produce low FSR) holds uniformly across fault types or is driven by particular fault categories. This is important because the paper's central claim about a 'systematic' gap depends on the pattern being consistent across fault types, not just in aggregate.
  3. §III.D, Table V: The observability skill is described at a high level ('due to space constraints, we present a high-level abstraction'), but it is the independent variable for RQ3. The paper lists five principle categories but does not provide the actual skill content, the number of principles, or examples of how they are structured as generation guidance. Since the RQ3 conclusion—that lightweight guidance is insufficient—depends entirely on what the skill contains, the paper should either include the full skill in an appendix or provide a representative excerpt sufficient for readers to assess whether the skill is a fair test of the hypothesis.
  4. §III.B, Table III: The 'Runnable Services' column shows that only 136–154 of 200 generated systems are runnable, yet the FSR denominator appears to be 1,615 fault instances for all models. The paper should clarify whether the 1,615 figure refers to intended fault instances (200 systems × 13 faults, minus failures) or successfully executed fault instances. The 'Subset FSR' metric addresses this partially, but the relationship between the denominator and runnable services needs explicit clarification, as it affects the interpretation of the absolute FSR values.
minor comments (7)
  1. §I: The phrase 'up to 13.99%' in the abstract and introduction is slightly misleading since 13.99% is the highest FSR, not a range ceiling. Consider rephrasing to 'ranging from 4.95% to 13.99%' for consistency with the body.
  2. Table II: The prompting strategy labels are inconsistent between the text and table. The text uses 'observability-blind,' 'observability-hinted,' and 'few-shot examples,' while the table uses 'observability-blind prompt,' 'observability-hinted prompt,' and 'few-shot examples prompt.' Standardize.
  3. Figure 4: The subfigure labels (a)–(d) are referenced in the text but the figure caption does not describe what each subfigure shows. Add descriptive captions.
  4. §III.E: The model versions (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash) are referenced but no access dates or API versions are specified beyond the general 'experiment time' note. Include specific dates for reproducibility.
  5. §III.B: The 200 architecture specifications are described as 'abstracted from real-world open-source and industrial systems' but no further detail is provided on how they were selected or validated. A brief description of the abstraction process would strengthen reproducibility.
  6. §V.A: The discussion offers two possible explanations for the observability gap (training data representation and lack of runtime feedback). These are speculative and not directly supported by the experimental data. Consider framing them more explicitly as hypotheses for future work.
  7. References [1]–[3]: The references to Copilot, Cursor, and Claude Code are cited as web pages with access year 2026. Consider citing more stable references (e.g., technical reports or documentation) where available.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee correctly identifies that the FSR oracle, the per-fault-type breakdown, the observability skill content, and the denominator clarification are all important points that warrant explicit attention in the manuscript. We address each comment below and commit to revisions for all four major comments.

read point-by-point responses

  1. Referee: §III.C, Table I: The FSR oracle uses keyword/pattern matching to define fault-specific log evidence... The concern is a specific systematic bias: LLM-generated code may produce logs with non-standard phrasing or indirect fault indicators that do not match oracle patterns derived from human-written systems. The paper should quantify the potential impact—for example, by manually auditing a sample of 'no signal' logs.

    Authors: The referee raises a valid and important concern. We agree that the FSR oracle, being derived from human-written observability patterns, could systematically miss valid but non-standard fault signals in LLM-generated logs. This is a genuine threat to the absolute FSR values we report. We will conduct a manual audit of 100 'no signal' log instances sampled across all 13 fault types and report the estimated false-negative rate of the oracle in a revised §V.B. We will also add an explicit discussion of this systematic bias risk. If the audit reveals a meaningful number of valid-but-non-matching signals, we will report adjusted FSR estimates. We note that even with upward adjustment, the relative comparisons across models and the overall finding of a substantial semantic observability gap are likely to hold, but we commit to quantifying this empirically rather than asserting it. revision: yes

  2. Referee: §III.C, Table III/IV: The FSR is reported as a single aggregate per model, but Table IV shows substantial variation across fault types. The paper should report per-model FSR by fault type, at least for the baseline (no skill) condition.

    Authors: We agree. Table IV currently aggregates across all models, which prevents readers from assessing whether the cross-model consistency claim holds uniformly across fault types. We will add a per-model, per-fault-type FSR breakdown for the baseline (no skill) condition, either as an expanded Table IV or as a supplementary table. This will allow direct inspection of whether, for example, all three models consistently produce low FSR for time-skew and CPU stress, or whether the aggregate pattern is driven by particular model-fault combinations. We agree this is necessary to support the 'systematic gap' claim. revision: yes

  3. Referee: §III.D, Table V: The observability skill is described at a high level... the paper should either include the full skill in an appendix or provide a representative excerpt sufficient for readers to assess whether the skill is a fair test of the hypothesis.

    Authors: We agree that the current high-level description is insufficient for readers to evaluate whether the skill constitutes a fair test of the hypothesis that lightweight guidance can improve observability. We will include the full skill content (or a representative excerpt that faithfully conveys its structure, principles, and concrete examples) in an appendix. We will also report the number of principles and provide at least one worked example showing how the skill is structured as generation guidance. This will allow readers to assess the skill's scope and limitations independently. revision: yes

  4. Referee: §III.B, Table III: The 'Runnable Services' column shows that only 136–154 of 200 generated systems are runnable, yet the FSR denominator appears to be 1,615 fault instances for all models. The paper should clarify whether the 1,615 figure refers to intended fault instances or successfully executed fault instances.

    Authors: The referee is correct that this relationship needs explicit clarification. The 1,615 figure refers to intended fault instances (200 systems × 13 faults, minus instances where the system was not runnable or deployment failed). The FSR in Table III uses this intended-fault denominator, while the Subset FSR restricts the denominator to successfully executed services. We will revise §III.C and Table III to state this explicitly, including the exact denominator breakdown per model (number of intended fault instances, number of successfully executed fault instances, and how each FSR variant is computed). This will remove any ambiguity about what the absolute FSR values represent. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical study with externally grounded metrics and independent data sources

full rationale

This is an empirical software engineering study with no theoretical derivation chain that could be circular. The three research questions each rest on independently grounded inputs: RQ1 uses 1,223 instances from 10 open-source and 8 industrial repositories with human-written ground truth; RQ2 uses 200 generated microservice systems, Chaos Mesh fault injection (an external tool), and fault-specific log evidence oracles (Table I) derived from expected failure manifestations; RQ3 derives its observability skill from ~200 real-world commits in an internal failure repository, not from the paper's own prior results. The FSR metric is defined as FSR = N_Signal / N_fault, where N_Signal is determined by keyword/pattern matching against the oracle criteria in Table I — this is a measurement procedure, not a derivation that reduces to its inputs. The oracle patterns (e.g., 'timeout/latency/deadline evidence' for F02) are derived from the expected manifestations of injected failures, not from the paper's own findings. Self-citations (refs [18], [20]) appear in a list of RCA-related citations and are not load-bearing for any derivation. The paper's central claim — that LLM-generated code has limited observability (FSR 4.95%–13.99%) — is an empirical measurement against external benchmarks, not a prediction derived from a fitted model or a self-referential definition. The threats-to-validity section honestly acknowledges that the oracle may miss valid signals, but this is a measurement-validity concern, not circularity. No step in the paper's chain reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no new theoretical entities, particles, or formal constructs. The KeyBag F1 and FSR metrics are evaluation instruments, not ontological commitments. The observability skill is a practical artifact, not a postulated entity. The free parameters are all design choices in the evaluation methodology.

free parameters (3)
  • Fault-specific log evidence oracle patterns = 13 fault types with keyword sets (Table I)
    The oracle patterns defining what constitutes a fault signal are manually curated per fault type. These are not derived from theory but from expected failure manifestations, introducing a design choice that affects all FSR measurements.
  • Observability skill principles = Derived from ~200 real-world commits
    The skill content used in RQ3 is abstracted from a specific set of commits. The selection criteria and abstraction process are not fully specified, making the skill a hand-crafted intervention.
  • KeyBag stop words and normalization rules = e.g., logger, span, event removed; lowercase, split on separators
    The token normalization and stop-word list for KeyBag F1 are manually defined, affecting the semantic overlap metric.
axioms (3)
  • domain assumption Human-written observability artifacts represent a meaningful reference standard for evaluating agent-generated observability.
    RQ1 uses human-written observability as ground truth. The paper acknowledges (§V.B) this is not optimal but argues it reflects practical engineering standards.
  • domain assumption Keyword-based log evidence oracles are a fair proxy for whether a fault signal is exposed.
    FSR depends entirely on the oracle patterns in Table I matching fault-specific log content. The paper acknowledges this may miss valid signals (§V.B).
  • domain assumption 200 microservice architecture specifications are representative of real-world agent-generated systems.
    RQ2 generalizes from 200 generated systems. The specifications are described as covering diverse domains but the sampling methodology is not detailed.

pith-pipeline@v1.1.0-glm · 19356 in / 2579 out tokens · 327395 ms · 2026-07-09T00:00:40.581503+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · 6 internal anchors

  1. [1]

    Microsoft copilot,

    Microsoft, “Microsoft copilot,” https://copilot.microsoft.com, 2026

    work page 2026

  2. [2]

    Cursor, “Cursor,” https://cursor.com, 2026

    work page 2026

  3. [3]

    Claude code,

    Anthropic, “Claude code,” https://www.anthropic.com/claude-code, 2026, accessed: 2026

    work page 2026

  4. [4]

    opentelemetry-demo,

    OpenTelemetry, “opentelemetry-demo,” gitHub repository. Available: https://github.com/open-telemetry/opentelemetry-demo. Accessed: July 2026

    work page 2026

  5. [5]

    microservices-demo,

    GoogleCloudPlatform, “microservices-demo,” gitHub repository. Avail- able: https://github.com/GoogleCloudPlatform/microservices-demo. Ac- cessed: July 2026

    work page 2026

  6. [6]

    Deathstarbench,

    Delimitrou Lab, “Deathstarbench,” gitHub repository. Available: https: //github.com/delimitrou/DeathStarBench. Accessed: July 2026

    work page 2026

  7. [7]

    Available: https://github.com/dotnet/ eShop

    dotnet, “eshop,” gitHub repository. Available: https://github.com/dotnet/ eShop. Accessed: July 2026

    work page 2026

  8. [8]

    Available: https://github.com/ golevelup/nestjs

    golevelup, “nestjs,” gitHub repository. Available: https://github.com/ golevelup/nestjs. Accessed: July 2026

    work page 2026

  9. [9]

    robusta,

    Robusta Dev, “robusta,” gitHub repository. Available: https://github.com/ robusta-dev/robusta. Accessed: July 2026

    work page 2026

  10. [10]

    microservices-demo,

    microservices-demo, “microservices-demo,” gitHub repository. Avail- able: https://github.com/microservices-demo/microservices-demo. Ac- cessed: July 2026

    work page 2026

  11. [11]

    train-ticket,

    FudanSELab, “train-ticket,” gitHub repository. Available: https://github. com/FudanSELab/train-ticket. Accessed: July 2026

    work page 2026

  12. [12]

    Available: https://github.com/strapi/ strapi

    Strapi, “strapi,” gitHub repository. Available: https://github.com/strapi/ strapi. Accessed: July 2026

    work page 2026

  13. [13]

    Available: https://github.com/ vectordotdev/vector

    Vector.dev, “vector,” gitHub repository. Available: https://github.com/ vectordotdev/vector. Accessed: July 2026

    work page 2026

  14. [14]

    Chaos mesh,

    C. M. Authors, “Chaos mesh,” https://github.com/chaos-mesh/ chaos-mesh, gitHub repository, accessed July 2026

    work page 2026

  15. [15]

    Logkg: Log failure diagnosis through knowledge graph,

    Y . Sui, Y . Zhang, J. Sun, T. Xu, S. Zhang, Z. Li, Y . Sun, F. Guo, J. Shen, Y . Zhanget al., “Logkg: Log failure diagnosis through knowledge graph,”IEEE Transactions on Services Computing (TSC), 2023

    work page 2023

  16. [16]

    Faultprofit: Hierarchical fault profiling of incident tickets in large-scale cloud systems,

    J. Huang, J. Liu, Z. Chen, Z. Jiang, Y . Li, J. Gu, C. Feng, Z. Yang, Y . Yang, and M. R. Lyu, “Faultprofit: Hierarchical fault profiling of incident tickets in large-scale cloud systems,” ser. ICSE-SEIP ’24. Association for Computing Machinery, 2024, p. 392–404

    work page 2024

  17. [17]

    Fault localization for microservice applications with system logs and monitoring metrics,

    Q. Zhang, T. Jia, Z. Wu, Q. Wu, L. Jia, D. Li, Y . Tao, and Y . Xiao, “Fault localization for microservice applications with system logs and monitoring metrics,” in2022 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), 2022, pp. 149–154

    work page 2022

  18. [18]

    Automatic root cause analysis via large language models for cloud incidents,

    Y . Chen, H. Xie, M. Ma, Y . Kang, X. Gao, L. Shi, Y . Cao, X. Gao, H. Fan, M. Wen, J. Zeng, S. Ghosh, X. Zhang, C. Zhang, Q. Lin, S. Rajmohan, D. Zhang, and T. Xu, “Automatic root cause analysis via large language models for cloud incidents,” ser. EuroSys ’24, New York, NY , USA, 2024, p. 674–688

    work page 2024

  19. [19]

    Microhecl: High-efficient root cause localization in large- scale microservice systems,

    D. Liu, C. He, X. Peng, F. Lin, C. Zhang, S. Gong, Z. Li, J. Ou, and Z. Wu, “Microhecl: High-efficient root cause localization in large- scale microservice systems,” in2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2021, pp. 338–347

    work page 2021

  20. [20]

    Latent error prediction and fault localization for microservice applications by learning from system trace logs,

    X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, D. Liu, Q. Xiang, and C. He, “Latent error prediction and fault localization for microservice applications by learning from system trace logs,” inProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the F oundations of Software Engineering, ser. ESEC/FSE 2019. ...

    work page 2019

  21. [21]

    Openai official website,

    OpenAI, “Openai official website,” https://openai.com, 2026

    work page 2026

  22. [22]

    Anthropic, “Claude,” https://www.anthropic.com/claude, 2026

    work page 2026

  23. [23]

    DeepMind, “Gemini,” https://deepmind.google/technologies/gemini/, 2026

    G. DeepMind, “Gemini,” https://deepmind.google/technologies/gemini/, 2026

    work page 2026

  24. [24]

    Copilot sdk,

    GitHub, “Copilot sdk,” https://github.com/github/copilot-sdk, gitHub repository, accessed July 2026

    work page 2026

  25. [25]

    LogLM: From Task-based to Instruction-based Automated Log Analysis

    Y . Liu, Y . Ji, S. Tao, M. He, W. Meng, S. Zhang, Y . Sun, Y . Xie, B. Chen, and H. Yang, “Loglm: From task-based to instruction-based automated log analysis,” 2025. [Online]. Available: https://arxiv.org/abs/2410.09352

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Truth-o-meter: Handling multiple inconsistent sources repairing llm hallucinations,

    B. Galitsky, A. Chernyavskiy, and D. Ilvovsky, “Truth-o-meter: Handling multiple inconsistent sources repairing llm hallucinations,” inProceed- ings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, p. 2817–2821

    work page 2024

  27. [27]

    Fuzz4all: Universal fuzzing with large language models,

    C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, “Fuzz4all: Universal fuzzing with large language models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    work page 2024

  28. [28]

    Livecodebench: Holistic and contamination free evaluation of large language models for code,

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,”

    work page

  29. [29]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    [Online]. Available: https://arxiv.org/abs/2403.07974

    work page internal anchor Pith review Pith/arXiv arXiv

  30. [30]

    Featbench: Towards more realistic evaluation of feature-level code generation

    H. Chen, C. Li, and J. Li, “Featbench: Towards more realistic evaluation of feature-level code generation.” [Online]. Available: https://arxiv.org/abs/2509.22237

    work page arXiv

  31. [31]

    Program Synthesis with Large Language Models

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  32. [32]

    DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

    J. Li, G. Li, Y . Zhao, Y . Li, H. Liu, H. Zhu, L. Wang, K. Liu, Z. Fang, L. Wang, J. Ding, X. Zhang, Y . Zhu, Y . Dong, Z. Jin, B. Li, F. Huang, and Y . Li, “Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories,” 2024. [Online]. Available: https://arxiv.org/abs/2405.19856

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [33]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , USA: Association for Computing Machinery,

    work page

  34. [34]

    Available: https://doi.org/10.1145/3597503.3623316

    [Online]. Available: https://doi.org/10.1145/3597503.3623316

    work page doi:10.1145/3597503.3623316

  35. [35]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Junet al., “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107. 03374

    work page 2021

  36. [36]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    M. A. Merrill, A. G. Shaw, N. Carliniet al., “Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces,” 2026. [Online]. Available: https://arxiv.org/abs/2601.11868

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  37. [37]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” 2024. [Online]. Available: https://arxiv.org/abs/2310. 06770

    work page 2024

  38. [38]

    Where do developers log? an empirical study on logging practices in industry,

    Q. Fu, J. Zhu, W. Hu, J.-G. Lou, R. Ding, Q. Lin, D. Zhang, and T. Xie, “Where do developers log? an empirical study on logging practices in industry,” pp. 24–33, 2014. [Online]. Available: https://doi.org/10.1145/2591062.2591175

    work page doi:10.1145/2591062.2591175 2014

  39. [39]

    Observability in microservices: An in-depth exploration of frameworks, challenges, and deployment paradigms,

    U. Faseeha, H. Jamil Syed, F. Samad, S. Zehra, and H. Ahmed, “Observability in microservices: An in-depth exploration of frameworks, challenges, and deployment paradigms,”IEEE Access, vol. 13, pp. 72 011–72 039, 2025

    work page 2025

  40. [40]

    Drain: An online log parsing approach with fixed depth tree,

    P. He, J. Zhu, Z. Zheng, and M. R. Lyu, “Drain: An online log parsing approach with fixed depth tree,” in2017 IEEE International Conference on Web Services (ICWS). IEEE, 2017, pp. 33–40

    work page 2017

  41. [41]

    A large-scale evaluation for log parsing techniques: How far are we?

    Z. Jiang, J. Liu, J. Huang, Y . Li, Y . Huo, J. Gu, Z. Chen, J. Zhu, and M. R. Lyu, “A large-scale evaluation for log parsing techniques: How far are we?” inProc. International Symposium on Software Testing and Analysis (ISSTA), 2024

    work page 2024

  42. [42]

    Llmparser: An exploratory study on using large language models for log parsing,

    Z. Ma, A. R. Chen, D. J. Kim, T.-H. Chen, and S. Wang, “Llmparser: An exploratory study on using large language models for log parsing,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering. ACM, Apr. 2024. [Online]. Available: http://dx.doi.org/10.1145/3597503.3639150

    work page doi:10.1145/3597503.3639150 2024

  43. [43]

    LogLLM: Log-based Anomaly Detection Using Large Language Models

    W. Guan, J. Cao, S. Qian, J. Gao, and C. Ouyang, “Logllm: Log-based anomaly detection using large language models.” [Online]. Available: https://arxiv.org/abs/2411.08561

    work page internal anchor Pith review Pith/arXiv arXiv

  44. [44]

    Logprompt: Prompt engineering towards zero-shot and interpretable log analysis,

    Y . Liu, S. Tao, W. Meng, F. Yao, X. Zhao, and H. Yang, “Logprompt: Prompt engineering towards zero-shot and interpretable log analysis,” in 2024 IEEE/ACM 46th International Conference on Software Engineer- ing: Companion Proceedings (ICSE-Companion), 2024, pp. 364–365

    work page 2024

  45. [45]

    Raglog: Log anomaly detection using retrieval augmented generation,

    J. Pan, W. S. Liang, and Y . Yidi, “Raglog: Log anomaly detection using retrieval augmented generation,” in2024 IEEE World F orum on Public Safety Technology (WFPST), 2024, pp. 169–174

    work page 2024

  46. [46]

    Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults,

    X. Li, P. Chen, L. Jing, Z. He, and G. Yu, “Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults,” in2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020, pp. 92–103

    work page 2020

  47. [47]

    Log clustering based problem identification for online service systems,

    Q. Lin, H. Zhang, J.-G. Lou, Y . Zhang, and X. Chen, “Log clustering based problem identification for online service systems,” inProceedings of the 38th International Conference on Software Engineering Compan- ion, ser. ICSE ’16, 2016, p. 102–111

    work page 2016

  48. [48]

    Evlog: Identifying anomalous logs over software evolution,

    Y . Huo, C. Lee, Y . Su, S. Shan, J. Liu, and M. R. Lyu, “Evlog: Identifying anomalous logs over software evolution,” in2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), 2023, pp. 391–402

    work page 2023

  49. [49]

    Mining invariants from console logs for system problem detection

    J.-G. Lou, Q. Fu, S. Yang, Y . Xu, and J. Li, “Mining invariants from console logs for system problem detection.” inUSENIX Annual Technical Conference, 2010, pp. 1–14

    work page 2010

  50. [50]

    Deeplog: Anomaly detection and diagnosis from system logs through deep learning,

    M. Du, F. Li, G. Zheng, and V . Srikumar, “Deeplog: Anomaly detection and diagnosis from system logs through deep learning,” inProceedings of the 2017 ACM SIGSAC Conference on Computer and Communica- tions Security, 2017, p. 1285–1298

    work page 2017

  51. [51]

    Anomaly detection in oper- ating system logs with deep learning-based sentiment analysis,

    H. Studiawan, F. Sohel, and C. Payne, “Anomaly detection in oper- ating system logs with deep learning-based sentiment analysis,”IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 5, pp. 2136–2148, 2021

    work page 2021

  52. [52]

    Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation,

    L. Yang, J. Chen, Z. Wang, W. Wang, J. Jiang, X. Dong, and W. Zhang, “Plelog: Semi-supervised log-based anomaly detection via probabilistic label estimation,” in2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2021, pp. 230–231

    work page 2021

  53. [53]

    Eadro: An end- to-end troubleshooting framework for microservices on multi-source data,

    C. Lee, T. Yang, Z. Chen, Y . Su, and M. R. Lyu, “Eadro: An end- to-end troubleshooting framework for microservices on multi-source data,” inProceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23, 2023, p. 1750–1762

    work page 2023

  54. [54]

    Deeptralog: Trace-log combined microservice anomaly de- tection through graph-based deep learning,

    C. Zhang, X. Peng, C. Sha, K. Zhang, Z. Fu, X. Wu, Q. Lin, and D. Zhang, “Deeptralog: Trace-log combined microservice anomaly de- tection through graph-based deep learning,” in2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 2022, pp. 623–634

    work page 2022

  55. [55]

    Localizing failure root causes in a microservice through causality inference,

    Y . Meng, S. Zhang, Y . Sun, R. Zhang, Z. Hu, Y . Zhang, C. Jia, Z. Wang, and D. Pei, “Localizing failure root causes in a microservice through causality inference,” in2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS), 2020, pp. 1–10

    work page 2020