{"id":"40c576b1-6df2-43ec-8626-ca1516acbfa4","arxiv_id":"2607.05785","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"AI coding agents produce microservice systems where only 4.95–13.99% of injected faults generate explicit fault signals in logs, revealing a semantic observability gap that lightweight guidance only partially addresses.","lead":"This paper finds that AI coding agents generate code with poor observability: logs exist but rarely capture fault-specific diagnostic information, with only 4.95–13.99% of injected faults producing detectable fault signals. A smart generalist should read this because it identifies a systematic blind spot in AI-generated software that affects production operability and debugging.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"FSR oracle's keyword-based matching may systematically undercount valid fault signals in LLM-generated logs, but the gap is large enough that the core conclusion likely holds.","rationale":"The reader correctly identified the most load-bearing concern: the FSR oracle's keyword-based matching is the single point through which all runtime conclusions flow. I agree with this assessment. The concern is genuine—LLM-generated logs may use non-standard phrasing that the oracle, derived from conventional logging patterns, would miss. However, several factors mitigate the risk: (1) the oracle patterns are reasonably broad, accepting multiple alternative keywords per fault type; (2) the gap is large enough (4.95%–13.99% FSR, even 20.62% on subset) that substantial oracle miss rates would be needed to overturn the conclusion; (3) the paper's qualitative examples (Figure 5) show logs that are genuinely semantically hollow (e.g., HTTP 201 status codes during a time-skew fault), supporting the claim that the issue is not just oracle mismatch but actual absence of diagnostic content; (4) the per-fault analysis (Table IV) shows a consistent pattern where faults with inherent error manifestations (pod kill, upstream fail) have higher FSR than faults requiring proactive instrumentation (time skew, CPU stress), which is exactly what one would expect from a real observability gap rather than an oracle artifact. Secondary concerns (KeyBag F1 as lexical proxy, no code/data release, high-level RQ3 skill description) are valid but do not independently threaten the central claim. The KeyBag F1 concern would, if anything, make RQ1 results look worse than they are (conservative bias). The lack of code/data release limits reproducibility but not correctness of the argument as presented. The RQ3 skill description is high-level, but the skill's effectiveness is a secondary claim; the primary finding is the observability gap itself. The CONDITIONAL verdict is appropriate: the core finding is defensible, but the precision of the FSR numbers and the inability to independently verify them warrant the qualifier. No verdict adjustment needed.","tokens_in":16615,"tokens_out":7177,"duration_ms":281789,"concrete_test":"Manually audit a stratified random sample of ~100 'no signal' fault instances (proportionally sampled across all 13 fault types and 3 models) from the RQ2 log data. For each, have two independent annotators determine whether the logs contain valid fault-specific evidence that the oracle missed. If fewer than 15% of 'no signal' cases contain missed valid evidence, the FSR would increase by at most a few percentage points and the core conclusion holds. If 30%+ contain missed valid evidence, the headline FSR numbers would need upward revision and the gap claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—FSR of 4.95%–13.99%—depends entirely on the fault-specific log evidence oracles defined in Table I. These oracles use keyword/pattern matching (e.g., 'timeout/latency/deadline evidence' for F02, 'CPU/load/slow processing/timeout evidence' for F11). The concern is not just generic oracle incompleteness, but a specific systematic bias: LLM-generated code may produce logs with non-standard phrasing, unconventional error messages, or indirect fault indicators that don't match the oracle's keyword lists. Human-written systems, from which these oracle patterns are derived, tend to follow conventional logging frameworks and terminology. Agent-generated systems may deviate from these conventions while still encoding valid diagnostic semantics. The paper acknowledges this in §V.B ('some logs may contain partial or indirect information that does not satisfy the oracle criteria') but does not quantify the impact. The concern is load-bearing because every RQ2 and RQ3 result flows from this oracle. However, the oracle patterns are fairly broad (e.g., F11 accepts any of 'CPU/load/slow/processing/timeout'), and the gap is large (even Subset FSR peaks at 20.62%), so it would require the oracle to miss a very large fraction of valid signals to overturn the conclusion. The concern is real but unlikely to be decisive.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper investigates whether coding agents can generate observability-aware code, addressing an important gap in existing evaluations that focus primarily on functional correctness. The study is structured around three research questions: (RQ1) whether agents can restore human-written observability artifacts at the source level, (RQ2) whether agent-generated microservice systems expose fault signals under injected failures at runtime, and (RQ3) whether lightweight observability-oriented guidance improves outcomes. The experimental design is systematic: 1,223 static restoration instances from 18 repositories, 200 generated microservice systems deployed on Kubernetes with 13 injected fault types, three frontier models, and five repetitions per experiment. The central finding is that coding agents exhibit a persistent 'semantic observability gap'—they are better at placing observability artifacts than capturing diagnostic semantics, and only 4.95%–13.99% of injected faults produce explicit fault signals in runtime logs. Lightweight guidance yields only marginal improvements.","tokens_in":16984,"tokens_out":1493,"duration_ms":423279,"significance":"The paper addresses a timely and underexplored dimension of code generation quality. The multi-level evaluation framework connecting source-level diagnostic semantics to runtime fault signals is a genuine contribution, as is the Fault Signal Rate (FSR) metric that measures whether generated systems actually expose failures rather than merely producing logging volume. The scale of the runtime evaluation (200 systems, 1,615 fault instances, Kubernetes deployment with Chaos Mesh) is commendable and provides falsifiable, reproducible evidence. The finding that observability should be treated as an independent generation-time objective rather than a by-product of functional code generation has clear implications for benchmark design and agent development.","major_comments":[{"comment":"§III.C, Table I: The FSR oracle uses keyword/pattern matching to define fault-specific log evidence (e.g., 'timeout/latency/deadline evidence' for F02, 'CPU/load/slow processing/timeout evidence' for F11). The paper acknowledges in §V.B that 'some logs may contain partial or indirect information that does not satisfy the oracle criteria and is therefore counted as no signal.' This is load-bearing because every RQ2 and RQ3 result flows from this oracle. The concern is not generic incompleteness but a specific systematic bias: LLM-generated code may produce logs with non-standard phrasing or indirect fault indicators that do not match oracle patterns derived from human-written systems. The paper should quantify the potential impact—for example, by manually auditing a sample of 'no signal' logs (e.g., 100 instances across fault types) to estimate how many contain valid but non-matchingfault","section":null},{"comment":"§III.C, Table III: The FSR is reported as a single aggregate per model, but Table IV shows substantial variation across fault types (from 1.26% for time-skew to 27.45% for upstream-fail). The paper should report per-model FSR by fault type, at least for the baseline (no skill) condition. Without this, it is unclear whether the cross-model consistency claim (that all models produce low FSR) holds uniformly across fault types or is driven by particular fault categories. This is important because the paper's central claim about a 'systematic' gap depends on the pattern being consistent across fault types, not just in aggregate.","section":null},{"comment":"§III.D, Table V: The observability skill is described at a high level ('due to space constraints, we present a high-level abstraction'), but it is the independent variable for RQ3. The paper lists five principle categories but does not provide the actual skill content, the number of principles, or examples of how they are structured as generation guidance. Since the RQ3 conclusion—that lightweight guidance is insufficient—depends entirely on what the skill contains, the paper should either include the full skill in an appendix or provide a representative excerpt sufficient for readers to assess whether the skill is a fair test of the hypothesis.","section":null},{"comment":"§III.B, Table III: The 'Runnable Services' column shows that only 136–154 of 200 generated systems are runnable, yet the FSR denominator appears to be 1,615 fault instances for all models. The paper should clarify whether the 1,615 figure refers to intended fault instances (200 systems × 13 faults, minus failures) or successfully executed fault instances. The 'Subset FSR' metric addresses this partially, but the relationship between the denominator and runnable services needs explicit clarification, as it affects the interpretation of the absolute FSR values.","section":null}],"minor_comments":[{"comment":"§I: The phrase 'up to 13.99%' in the abstract and introduction is slightly misleading since 13.99% is the highest FSR, not a range ceiling. Consider rephrasing to 'ranging from 4.95% to 13.99%' for consistency with the body.","section":null},{"comment":"Table II: The prompting strategy labels are inconsistent between the text and table. The text uses 'observability-blind,' 'observability-hinted,' and 'few-shot examples,' while the table uses 'observability-blind prompt,' 'observability-hinted prompt,' and 'few-shot examples prompt.' Standardize.","section":null},{"comment":"Figure 4: The subfigure labels (a)–(d) are referenced in the text but the figure caption does not describe what each subfigure shows. Add descriptive captions.","section":null},{"comment":"§III.E: The model versions (GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash) are referenced but no access dates or API versions are specified beyond the general 'experiment time' note. Include specific dates for reproducibility.","section":null},{"comment":"§III.B: The 200 architecture specifications are described as 'abstracted from real-world open-source and industrial systems' but no further detail is provided on how they were selected or validated. A brief description of the abstraction process would strengthen reproducibility.","section":null},{"comment":"§V.A: The discussion offers two possible explanations for the observability gap (training data representation and lack of runtime feedback). These are speculative and not directly supported by the experimental data. Consider framing them more explicitly as hypotheses for future work.","section":null},{"comment":"References [1]–[3]: The references to Copilot, Cursor, and Claude Code are cited as web pages with access year 2026. Consider citing more stable references (e.g., technical reports or documentation) where available.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is well-motivated and the experimental infrastructure is impressive. The primary concern is the FSR oracle validation: the keyword-based matching is the weakest link in an otherwise strong study. A manual audit of a sample of 'no signal' cases would substantially strengthen the paper's central claim. If the audit confirms that the oracle is not systematically missing valid signals, the paper's conclusions are well-supported. Without it, a reviewer cannot fully assess whether the reported FSR values are fair. The skill content disclosure issue is also important for RQ3 reproducibility but is more easily addressed. I would encourage the authors to address these points; the core contribution is valuable and the study design is sound."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee correctly identifies that the FSR oracle, the per-fault-type breakdown, the observability skill content, and the denominator clarification are all important points that warrant explicit attention in the manuscript. We address each comment below and commit to revisions for all four major comments.","responses":[{"response":"The referee raises a valid and important concern. We agree that the FSR oracle, being derived from human-written observability patterns, could systematically miss valid but non-standard fault signals in LLM-generated logs. This is a genuine threat to the absolute FSR values we report. We will conduct a manual audit of 100 'no signal' log instances sampled across all 13 fault types and report the estimated false-negative rate of the oracle in a revised §V.B. We will also add an explicit discussion of this systematic bias risk. If the audit reveals a meaningful number of valid-but-non-matching signals, we will report adjusted FSR estimates. We note that even with upward adjustment, the relative comparisons across models and the overall finding of a substantial semantic observability gap are likely to hold, but we commit to quantifying this empirically rather than asserting it.","revision_made":"yes","referee_comment":"§III.C, Table I: The FSR oracle uses keyword/pattern matching to define fault-specific log evidence... The concern is a specific systematic bias: LLM-generated code may produce logs with non-standard phrasing or indirect fault indicators that do not match oracle patterns derived from human-written systems. The paper should quantify the potential impact—for example, by manually auditing a sample of 'no signal' logs."},{"response":"We agree. Table IV currently aggregates across all models, which prevents readers from assessing whether the cross-model consistency claim holds uniformly across fault types. We will add a per-model, per-fault-type FSR breakdown for the baseline (no skill) condition, either as an expanded Table IV or as a supplementary table. This will allow direct inspection of whether, for example, all three models consistently produce low FSR for time-skew and CPU stress, or whether the aggregate pattern is driven by particular model-fault combinations. We agree this is necessary to support the 'systematic gap' claim.","revision_made":"yes","referee_comment":"§III.C, Table III/IV: The FSR is reported as a single aggregate per model, but Table IV shows substantial variation across fault types. The paper should report per-model FSR by fault type, at least for the baseline (no skill) condition."},{"response":"We agree that the current high-level description is insufficient for readers to evaluate whether the skill constitutes a fair test of the hypothesis that lightweight guidance can improve observability. We will include the full skill content (or a representative excerpt that faithfully conveys its structure, principles, and concrete examples) in an appendix. We will also report the number of principles and provide at least one worked example showing how the skill is structured as generation guidance. This will allow readers to assess the skill's scope and limitations independently.","revision_made":"yes","referee_comment":"§III.D, Table V: The observability skill is described at a high level... the paper should either include the full skill in an appendix or provide a representative excerpt sufficient for readers to assess whether the skill is a fair test of the hypothesis."},{"response":"The referee is correct that this relationship needs explicit clarification. The 1,615 figure refers to intended fault instances (200 systems × 13 faults, minus instances where the system was not runnable or deployment failed). The FSR in Table III uses this intended-fault denominator, while the Subset FSR restricts the denominator to successfully executed services. We will revise §III.C and Table III to state this explicitly, including the exact denominator breakdown per model (number of intended fault instances, number of successfully executed fault instances, and how each FSR variant is computed). This will remove any ambiguity about what the absolute FSR values represent.","revision_made":"yes","referee_comment":"§III.B, Table III: The 'Runnable Services' column shows that only 136–154 of 200 generated systems are runnable, yet the FSR denominator appears to be 1,615 fault instances for all models. The paper should clarify whether the 1,615 figure refers to intended fault instances or successfully executed fault instances."}],"tokens_in":16701,"tokens_out":900,"duration_ms":123215,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The paper asks a genuinely new question: can coding agents generate code that is not just runnable but observable under failure? The answer is a clear no. Across 200 agent-generated microservice systems deployed on Kubernetes with 1,615 injected fault instances, only 4.95–13.99% of faults produce explicit fault signals in runtime logs. That is the headline you should remember. The multi-level evaluation framework — static restoration (RQ1) plus runtime fault injection (RQ2) — is a solid contribution. The KeyBag F1 and FSR metrics are purpose-built and reasonable. The finding that agents are better at placing logging statements than at capturing diagnostic semantics (Position F1 consistently exceeds KeyBag F1) is a useful decomposition. The RQ3 result — that lightweight observability guidance yields only marginal gains — is honest and important. The experimental scale is respectable: 1,223 static instances, 200 microservice systems, 13 fault types, three frontier models, five repetitions each. The per-fault breakdown in Table IV is informative, showing that faults with explicit error manifestations (upstream fail, pod kill) are detected at much higher rates than implicit ones (time skew, CPU stress). Now the soft spots. The FSR oracle uses keyword-based pattern matching (Table I) to decide whether a fault signal exists in logs. The stress-test concern about this is valid in principle: LLM-generated logs may use non-standard phrasing that the oracle misses, systematically undercounting valid signals. But having read the paper, I think this concern is unlikely to be decisive. The oracle patterns are fairly broad (e.g., F11 accepts any of 'CPU/load/slow/processing/timeout'), and the gap is large enough that even doubling FSR would still leave most faults undetected. The paper also acknowledges this limitation directly in §V.B. A more substantive concern is reproducibility: no code or data is released. The RQ3 observability skill is described only at a high level — the five bullet points in §III.D are principles, not a reproducible artifact. The KeyBag F1 metric is a lexical proxy, which the authors acknowledge. These are real limitations but they constrain precision of claims, not the core finding. The central argument holds: agent-generated code has a systematic observability gap that lightweight guidance does not close. This paper is for researchers and practitioners working on coding agents, software observability, and LLM-based code generation evaluation. It deserves a serious referee who can push on the oracle design and demand artifact release.","headline":"Agent-generated code has a real observability gap; the FSR oracle is keyword-based but the gap is large enough that the core finding survives.","tokens_in":17573,"tokens_out":593,"would_cite":true,"duration_ms":131369,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"AI-Generated Code Fails to Expose Failures: Only 5-14% of Faults Surface in Logs","keywords":["observability","coding agents","fault signal","log quality","microservices","software quality","LLM code generation","diagnostic semantics"],"falsifier":"If a coding agent could be shown to generate systems where, under injected faults, a substantial majority (say >70%) of failures produce explicit fault-specific log evidence without human post-hoc instrumentation, the core claim of a persistent semantic observability gap would be weakened.","tokens_in":16708,"feed_emoji":"","tokens_out":1096,"duration_ms":187201,"temperature":0.7,"pith_summary":"The paper investigates whether coding agents can generate code that is not only functional but also observable—meaning it produces logs and other artifacts that expose what went wrong during failures. The authors find a persistent and substantial gap: while agents generate abundant logging, the logs rarely contain fault-specific diagnostic information. In 200 microservice systems deployed on Kubernetes with 1,615 injected fault instances, only 4.95% to 13.99% of failures produced explicit fault signals in runtime logs. The problem is not absence of logging but absence of meaningful failure semantics. At the source-code level, agents are better at placing observability statements in the right locations than at capturing the right diagnostic content (error codes, dependency states, timeout values). A lightweight observability skill derived from real engineering commits improved fault signal rates modestly (up to +8.67 percentage points for one model) but left overall rates low, suggesting the gap is structural rather than a simple prompting deficiency.","feed_headline":"AI-Generated Code Barely Exposes Failures: Only 5-14% of Faults Surface","feed_subtitle":"200 microservice systems tested with 1,615 injected faults show that coding agents produce abundant but semantically hollow logs, and light","key_machinery":"The paper introduces a two-level evaluation framework connecting source-level diagnostic semantics (what observability artifacts encode) to runtime fault signals (what actually surfaces during failures). The Fault Signal Rate (FSR) metric measures the fraction of injected faults that produce explicit, fault-specific log evidence. The KeyBag F1 metric measures lexical overlap of diagnostic content between agent-generated and human-written observability. Together these metrics separate placement accuracy from content quality, and static capability from runtime behavior.","core_discovery":"The central finding is a semantic observability gap: coding agents can identify where to place logging and tracing statements but cannot reliably determine what diagnostic information those statements should capture. This gap propagates from source code to runtime—generated systems produce logs that are voluminous but semantically hollow regarding actual failures. Faults that inherently produce explicit errors (crashes, connection refusals) are sometimes caught; faults with implicit effects (CPU stress, clock skew, network corruption) are almost never caught (1-2% detection rates). Lightweight guidance narrows but does not close the gap, indicating that effective observability requires aform","pith_inferences":["If the FSR oracle underestimates actual observability by missing non-standard fault signal patterns, the true gap may be smaller than reported—but the consistency of the gap across models, fault types, and evaluation settings suggests the qualitative finding (agents produce semantically hollow logs) would persist even with a more generous oracle.","The quantity-over-quality tradeoff observed under explicit instructions mirrors a broader pattern in LLM code generation: prompting for more output tends to dilute precision. This suggests that observability-oriented fine-tuning or RL-based alignment may be more effective than prompt engineering for improving diagnostic content quality.","If observability is rarely an independent objective in training data (as the authors hypothesize), then models fine-tuned on post-incident commits—where logging was explicitly added to diagnose a specific failure—might close the gap more effectively than models trained on general repository code."],"forward_implications":["Coding agent benchmarks that evaluate only functional correctness are insufficient for production readiness; observability should be treated as a first-class evaluation dimension alongside correctness, security, and performance.","The knowledge debt problem identified by the authors—developers maintaining systems whose runtime behavior they do not understand—suggests that agent-generated code may require different operational practices than human-written code, potentially shifting diagnostic burden toward post-hoc tooling rather than built-in observability.","The finding that explicit-error faults are detectable but implicit-effect faults are not implies that observability gaps will be most severe for the hardest-to-diagnose failure classes, exactly where observability matters most.","The limited effectiveness of lightweight guidance suggests that closing the gap may require runtime feedback loops during generation—agents that can deploy, inject faults, observe their own logging behavior, and iterate—rather than static prompt engineering."],"fun_headline_variants":["Agent-Generated Code Logs Failures Without Useful Diagnostics","LLMs Add Logging but Miss the Diagnostic Semantics for Real Faults","AI-Generated Microservices Surface Under 14% of Runtime Faults","Coding Agents Produce Logs That Lack Failure-Specific Semantics","Observability Gap in LLM Code: Voluminous Logs, Hollow Diagnostics"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The Fault Signal Rate metric relies on fault-specific log evidence oracles that define what counts as a valid fault signal using keyword and pattern matching (for example, requiring 'timeout' or 'latency' in logs for network-delay faults). If these oracles miss valid but non-standard fault signal patterns, the metric underestimates actual observability. The paper's conclusions about the observability gap rest on these oracles being a fair proxy for fault-detectability.","fun_headline_variants_meta":{"raw":{"variants":["Agent-Generated Code Logs Failures Without Useful Diagnostics","LLMs Add Logging but Miss the Diagnostic Semantics for Real Faults","AI-Generated Microservices Surface Under 14% of Runtime Faults","Coding Agents Produce Logs That Lack Failure-Specific Semantics","Observability Gap in LLM Code: Voluminous Logs, Hollow Diagnostics"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":677,"prompt_tokens":602,"completion_tokens":75,"prompt_tokens_details":null},"tokens_in":602,"tokens_out":75,"duration_ms":24240,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T00:00:40.581503+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a coding agent could be shown to generate systems where, under injected faults, a substantial majority (say >70%) of failures produce explicit fault-specific log evidence without human post-hoc instrumentation, the core claim of a persistent semantic observability gap would be weakened.","supporting_citations":[],"review_version":1}