{"id":"33f73a5a-a6ee-44d0-b40d-ae0c305de1e9","arxiv_id":"2607.21911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Structured retrieval of resolved incident triples, instead of raw documentation chunks, yields large gains in LLM-based bug diagnosis accuracy in a Dataverse evaluation.","lead":"The paper presents OM-RAG, a retrieval system that stores past software bugs as structured symptom, root cause, and fix records, and feeds the most similar ones to an LLM when diagnosing new failures. In a 1,172-issue Dataverse benchmark it reports diagnosis accuracy of 0.931, far above chunk-based retrieval and documentation graphs, plus a six-month production deployment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth and retrieval index both come from the same gpt-4o-mini extraction pipeline, so OM-RAG's DA/FC advantage may reflect target/evidence format self-consistency rather than diagnostic value; the 123-issue manual check does not verify the extracted ground truth.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern, and I agree with it: the benchmark's ground truth and the OM-RAG knowledge base are both outputs of the same LLM extraction pipeline, with the quality filter assigned by the extracting model. This is the weakest point because every headline number in Table III flows through it. The paper has genuine strengths: a controlled four-configuration ablation with the same inference LLM, tight bootstrap intervals, a consistent ordering across dimensions and domains, model-free retrieval metrics, a manual 123-issue subset, and an unusually candid limitations section. None of those strengths, however, validates the extracted ground truth against the original issue threads at scale; the manual pass validates judge decisions, not extraction correctness. A second, more subtle consequence is that the 13 domain categories are also assigned by the same extractor, so the across-domain consistency of the ranking may inherit label noise or extractor bias, although the authors acknowledge this. The absence of a released benchmark, dataset, or code link is a secondary but real obstacle: the paper claims a public benchmark in the introduction, yet no artifact is provided, making the circularity untestable by readers. Given the large effect sizes and the independent retrieval metrics, rejection is not warranted; the honest outcome is to keep the conditional verdict and require (1) independent human re-annotation of a statistically adequate sample, (2) artifact release, and ideally (3) a second-system replication before the comparative strength of the claim is accepted. For those reasons I would leave the reader's verdict unchanged.","tokens_in":8953,"tokens_out":6143,"duration_ms":57688,"concrete_test":"Re-annotate a random sample of at least 200 of the 1,172 benchmark issues: two domain-expert annotators, blind to the gpt-4o-mini triples and to OM-RAG outputs, independently read the original GitHub threads and write symptom/root-cause/resolution triples; adjudicate disagreements. Then recompute Table III's DA and FC for all four configurations against this independently sourced ground truth, keeping retrieval, inference, and judging unchanged. If the OM-RAG vs C-chunk DA gap (approximately 0.606 in Table III) falls materially, e.g., by more than 0.10, or if the ordering C-zero < C-chunk < C-doc-kg < OM-RAG no longer holds on the independent truth, the shared-extraction artifact is confirmed to be load-bearing. If the gap remains large and monotone, the circularity concern is substantially mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim rests on a circularity located in Sections III-B and IV-B. The ground-truth triples used to score Diagnosis Accuracy and Fix Correctness are produced by the same gpt-4o-mini extraction step that builds the OM-RAG knowledge base, and the quality_score >= 3 filter that selects the 1,172-issue benchmark is assigned by that same extractor. Thus the scoring target and the retrieved evidence share a schema, a vocabulary, and whatever systematic transformations the extractor imposes on GitHub issue threads. OM-RAG presents the inference LLM with an already-normalized symptom-root-cause string as a precedent; C-chunk presents raw thread chunks; C-doc-kg presents documentation concepts. When a judge compares the generated answer against an LLM-extracted ground-truth root cause, OM-RAG can be rewarded for reproducing the extractor's canonical phrasing rather than for correctly identifying the actual fault. The claim that ground truth is 'derived from developer-authored resolution discussions' does not remove this, because the extraction itself is unverified at full scale. The 123-issue manual verification checks judge decisions against the extracted root cause, not the extraction against the original issue threads, so it cannot break the loop. The Limitations section concedes reliance on an LLM judge but understates the earlier problem: the target itself is model-generated. If the extractor systematically paraphrases or selects a recognizable root-cause style, OM-RAG's retrieval can surface the answer format directly, and the reported 0.931 DA may partly measure self-consistency of one LLM pipeline rather than structured retrieval quality. The model-free metrics are real but are absolute retrieval-side measures, not baselined comparisons, so they do not establish that the DA/FC gaps are diagnostic rather than format-alignment artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces OM-RAG, a retrieval-augmented generation system that indexes resolved GitHub issues as structured symptom–root-cause–resolution triples and retrieves top-k precedents via single-hop embedding to help an LLM diagnose software bugs. The authors evaluate OM-RAG against three baselines (no retrieval, chunk-based retrieval from the same issues, and a documentation-anchored concept graph) on 1,172 issues from the IQSS/dataverse repository. Using claude-sonnet-4-6 as an LLM judge, they report that OM-RAG achieves Diagnosis Accuracy 0.931 and Fix Correctness 0.809, with the ordering C-zero < C-chunk < C-doc-kg < OM-RAG across all four rubric dimensions and all domain categories. Additional analyses include bootstrap CIs, a 23-issue cross-judge/positional-bias study, a 123-issue manual review, and model-free retrieval metrics.","tokens_in":9180,"tokens_out":8492,"duration_ms":68210,"significance":"If the reported effect is real, the paper makes a useful contribution: it demonstrates a concrete instantiation of the 'operational memory' idea, with a production deployment and a public benchmark. The design of the ablation (C-chunk vs OM-RAG isolating structure, C-doc-kg vs OM-RAG isolating source) is clean, and the bootstrap CIs and cross-judge agreement lend quantitative support. However, the central claim is weakened by the fact that the ground-truth triples and the OM-RAG knowledge base are generated by the same LLM extraction pipeline, a concern that the manual verification does not address. The model-free metrics, as reported, do not corroborate the ranking because they are only computed for OM-RAG.","major_comments":[{"comment":"The ground-truth triples used to compute Diagnosis Accuracy and Fix Correctness and the OM-RAG knowledge base are both produced by the same gpt-4o-mini extraction step, with the quality_score >= 3 filter assigned by the extracting model itself. This links the scoring target to the retrieved evidence in vocabulary, schema, and systematic extraction bias. Because the judge (claude-sonnet-4-6) is asked to compare the generated root cause against the extracted ground-truth root_cause, OM-RAG can be rewarded for reproducing the extractor's canonical phrasing of a root cause, while C-chunk and C-doc-kg are disadvantaged by presenting information in a less normalized form. The manual verification in Section IV-F checks judge decisions against the extracted root cause, not the extraction against the original issue threads, so it does not break this loop. To support the central ranking claim, the authors should either (a) have the ground-truth triples validated by human annotation on a sample of issues before scoring, or (b) construct an independent ground truth from developer commits/PRs that are referenced in the resolution discussions, and show that the OM-RAG advantage persists.","section":"Sections III-B and IV-B"},{"comment":"The abstract and Section IV-G state that model-free retrieval metrics (mean similarity 0.880) 'independently corroborate the ranking.' However, flat_rc_score and gt_keyword_hit_rate are reported only for OM-RAG; there is no model-free comparison for C-chunk or C-doc-kg. Thus these metrics cannot corroborate a ranking across configurations. Additionally, flat_rc_score measures embedding similarity between the query and the retrieved record, which is expected to be high for a KB built from the same extraction pipeline, and does not measure diagnostic correctness. The claim of independent corroboration is therefore overstated and should be removed or supported with analogous metrics for all configurations.","section":"Section IV-G and Abstract"},{"comment":"The paper states that ground truth triples 'represent actual human-verified resolutions.' This is not supported by the described methodology: the 1,172 triples are LLM extractions, and the only manual step is the 123-issue verification of judge decisions against the extracted root causes. The sentence should be qualified or replaced by a description of the actual validation performed, and the Limitations section should acknowledge that the evaluation target itself is model-generated, not just the judge.","section":"Section IV-B"}],"minor_comments":[{"comment":"The construction of C-chunk chunks (chunk size, overlap) and C-doc-kg (node types, edges, traversal) is not described in enough detail to permit replication; please add these details.","section":"Section IV-C"},{"comment":"The relationship between the 1,457 triples from ~1,396 issues and the 1,172-issue benchmark is unclear for issues that yield multiple triples; specify how ground truth is assigned and how the inference-time exclusion handles such issues.","section":"Section IV-B"},{"comment":"The paper does not report a quantitative judge-error rate from the 123-issue subset; 'all decisions were confirmed as reasonable' is not the same as a measured accuracy, and the effect-size robustness argument would be strengthened by reporting the number of judge errors out of 123.","section":"Section IV-F"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the ground-truth/KB circularity; if the authors can develop evidence against it, the paper would make a useful contribution to AIOps and RAG. As written, the absolute numbers should be interpreted with caution, and the claimed independent corroboration is weaker than presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper's real contribution is the retrieval target: structured symptom–root-cause–resolution triples from resolved incidents, framed as episodic memory, plus a 1,172-issue benchmark and a six-month production deployment. That's worth paying attention to. The writing is clear, the ablation design is sensible (four configurations isolating source and structure), and the ordering C-zero < C-chunk < C-doc-kg < OM-RAG holds across all four metrics and 13 domains. The bootstrap intervals are tight and the effect sizes are large. The authors also state limitations honestly.\n\nThe weak spot is structural, not cosmetic. Sections III-B and IV-B build both the knowledge base and the ground-truth triples with the same gpt-4o-mini extraction pipeline, including the quality_score >= 3 filter that selects the benchmark. So the scoring target and the retrieved evidence share the extractor's vocabulary and normalization. When the claude-sonnet judge compares OM-RAG's answer—anchored on a retrieved triple in that same style—against a ground-truth triple in that same style, the high DA may partly measure format self-consistency, not diagnostic truth. The 123-issue manual check verifies judge decisions against the extracted root cause, not the extraction against the original thread, so it doesn't break the loop. The model-free retrieval metrics are real but absolute; they don't baseline C-chunk or C-doc-kg. This is a genuine circularity burden, though not a fatal one. The +186% and +77% gaps are so large that a pure judge artifact seems unlikely, but the paper needs to close the loop with human-verified extractions, released artifacts, and ideally a second system.\n\nThe other issue: the abstract promises a public benchmark, but the full text gives no repo or Dataverse link. For an empirical paper that claims an open testbed, that's a missing deliverable.\n\nOverall: yes, send it to review. A serious referee should ask for the artifacts, a human-verified extraction sample, and a comparative model-free metric. The core idea—operational memory as a retrieval target—is solid and worth testing. I wouldn't cite it until the benchmark is actually available.","headline":"The retrieval-target idea is genuinely new and the benchmark is useful, but the ground-truth extraction loop and missing artifacts keep me from trusting the headline numbers.","tokens_in":9833,"tokens_out":2324,"would_cite":false,"duration_ms":21014,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM bug diagnosis fails when it retrieves design documentation rather than structured records of past failures; OM-RAG stores resolved incidents as symptom–root cause–resolution triples and reaches 0.931 diagnosis accuracy on 1,172 real…","keywords":["retrieval-augmented generation","operational memory","bug diagnosis","root cause analysis","LLM agents","episodic memory","case-based reasoning","self-adaptive systems"],"falsifier":"A direct test is to rebuild the knowledge base with triples written by human maintainers from the same 1,172 resolved issues and rerun the four-configuration ablation: if OM-RAG's diagnosis accuracy falls toward the chunk baseline when the index and ground truth no longer share the same LLM extraction, the 0.931 advantage is largely extraction self-consistency. A cheaper model-free check is to take a random sample of retrieved top-1 triples and have a human maintainer confirm that the root cause matches the actual resolution discussion, independent of any LLM judge.","tokens_in":8691,"feed_emoji":"🐛","tokens_out":11963,"duration_ms":93965,"temperature":0.7,"pith_summary":"This paper sets out to establish that LLM-based software bug diagnosis fails for two independent reasons, both rooted in what the retrieval system brings back. The first is a source mismatch: typical RAG systems retrieve design documentation, which describes how software is supposed to work, rather than records of how it has actually failed. The second is a structure mismatch: even when resolved incidents are retrieved, text chunking breaks the causal chain connecting symptom, root cause, and resolution. The authors propose Operational Memory RAG (OM-RAG), which stores each resolved incident as a structured triple and retrieves the most similar precedent with a single embedding lookup. On 1,172 real issues from the issue tracker of a large open-source research-data platform, OM-RAG achieves 0.931 diagnosis accuracy and 0.809 fix correctness, outperforming chunk-based retrieval by 186% and documentation-anchored graph retrieval by 77%; the paper presents this as evidence that operational memory, not more documentation, is the missing ingredient for diagnostic RAG.","feed_headline":"Past failures, not documentation, lift LLM diagnosis to 93%","feed_subtitle":"Retrieving symptom-cause-fix triples beats chunked text and documentation graphs in a 1,172-issue test.","key_machinery":"Operational Memory RAG (OM-RAG) is a four-phase pipeline whose load-bearing object is the structured diagnostic triple: symptom, root cause, and resolution, extracted from each closed bug report and kept in a flat embedding index. Phase 1 harvests closed bug reports and their comment threads from the system's issue tracker; Phase 2 prompts an LLM to distill each issue into a triple plus a self-assigned quality score, keeping only triples scored at least 3; Phase 3 concatenates symptom and root cause into one string, embeds it with a dense encoder, and stores the records in a flat database; Phase 4 embeds the new issue, retrieves the top-3 most similar triples by cosine similarity, and prompts the LLM to reason explicitly from those precedents. The index deliberately has no graph layer: the paper argues that bug diagnosis is a single-hop lookup problem, so cross-incident graph structure adds path-traversal noise without improving precision, and the ablation's documentation-graph configuration supports that claim by underperforming the flat incident index.","core_discovery":"The paper's central claim is that the right retrieval target for bug diagnosis is a structured operational case, not a documentation fragment, and that shifting this target alone produces large, consistent, and measurement-stable gains. The demonstration is a four-configuration ablation on 1,172 real bug reports: no retrieval, chunk-based retrieval from the same resolved issues, a documentation-anchored concept graph, and OM-RAG's structured incident triples. OM-RAG dominates the other configurations on every evaluation dimension—diagnosis accuracy, fix correctness, groundedness, and specificity—and within every domain category, with non-overlapping bootstrap confidence intervals; the observed ordering C-zero < C-chunk < C-doc-kg < OM-RAG is the principal result. The two baselines each display one of the two diagnosed failure modes: chunk retrieval is highly grounded but cannot reconstruct the causal chain, and documentation-graph retrieval routes through conceptual detours that dilute operational specificity. The authors conclude that only structured operational triples address both mismatches simultaneously, and they corroborate the LLM-judged scores with cross-judge checks, positional-bias tests, manual review of a 123-issue subset, and model-free retrieval similarity of 0.880.","pith_inferences":["If the result generalizes, the most valuable data for coding and operations assistants is the trail of past failures that organizations already keep but rarely structure; the paper's single-system design leaves that generality unproven.","A natural testable extension is to apply the same triple-extraction and flat-index recipe to cloud incident postmortems or issue trackers of other open-source systems and check whether the ordering C-zero < C-chunk < C-doc-kg < OM-RAG reproduces.","Because extraction and evaluation share one LLM, a stronger design would decouple them—human-authored root causes for a held-out subset and a different model family for index construction—to isolate the retrieval-structure contribution from extraction bias.","The 'retrieved but wrong' framing suggests a broader design principle for RAG: knowledge stores for diagnostic tasks should be episodic and case-shaped rather than semantic and rule-shaped, which may transfer to technical-support and clinical RAG where past cases are authoritative."],"forward_implications":["If the paper is right, engineering effort for diagnostic agents should go into structuring and indexing resolved incident history, not into building larger document corpora or richer knowledge graphs.","A flat single-hop index is sufficient for bug diagnosis on this task; graph-based retrieval adds cost and conceptual indirection without a precision gain.","Feeding resolved incidents back into the index turns the Knowledge component of a self-adaptation loop into an accumulating episodic memory, so a deployed agent improves as it operates.","The decomposed four-dimension scoring rubric provides a reusable template for evaluating diagnostic RAG systems beyond this benchmark.","Operational-memory RAG is economically viable at production scale: the full 1,172-issue evaluation cost about $65, or $0.056 per issue, including inference and judging."],"supporting_citations":[{"why":"Anchor for the 'retrieved but wrong' failure mode; the source/structure mismatch is framed as a failure point of RAG systems.","marker":"[1]"},{"why":"Supplies the episodic-versus-semantic memory distinction that motivates treating resolved incidents as episodic operational memory.","marker":"[2]"},{"why":"Defines the self-adaptation loop whose Knowledge component OM-RAG operationally fills.","marker":"[4]"},{"why":"Documents the open challenge that Knowledge components are populated with static models, the gap OM-RAG targets.","marker":"[5]"},{"why":"Provides the case-based reasoning principle that new problems be solved by retrieving structured prior cases.","marker":"[6]"},{"why":"Source of the decomposed RAG evaluation framework the paper adapts into its four-dimension scoring rubric.","marker":"[9]"},{"why":"Evidence that graphs add limited value when answers reside in single records, supporting the flat single-hop index design.","marker":"[16]"},{"why":"Context for the LLM-as-judge evaluation approach and its reliability checks.","marker":"[23]"}],"fun_headline_variants":["Structured past failures beat doc chunks for bug diagnosis","Operational triples, not docs, boost LLM diagnosis to 93%","LLM bug fixing jumps 186% with structured incident memory","OM-RAG: structured incident history lifts diagnosis to 93%","Resolved incidents as triples beat chunk and doc RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ground-truth triples used for scoring and the knowledge-base triples used for retrieval are both generated by the same LLM extraction pipeline, with the quality filter assigned by the extracting model itself, so the reported accuracy could partly reflect how well retrieval matches the extractor's own representation of root causes rather than how well it matches independently verifiable reality.","fun_headline_variants_meta":{"raw":{"variants":["Structured past failures beat doc chunks for bug diagnosis","Operational triples, not docs, boost LLM diagnosis to 93%","LLM bug fixing jumps 186% with structured incident memory","OM-RAG: structured incident history lifts diagnosis to 93%","Resolved incidents as triples beat chunk and doc RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000659,"raw_usage":{"total_tokens":3046,"prompt_tokens":1008,"completion_tokens":2038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1949}},"tokens_in":624,"tokens_out":2038,"duration_ms":13408,"temperature":1.0,"reasoning_tokens":1949,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:29:30.676560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to rebuild the knowledge base with triples written by human maintainers from the same 1,172 resolved issues and rerun the four-configuration ablation: if OM-RAG's diagnosis accuracy falls toward the chunk baseline when the index and ground truth no longer share the same LLM extraction, the 0.931 advantage is largely extraction self-consistency. A cheaper model-free check is to take a random sample of retrieved top-1 triples and have a human maintainer confirm that the root cause matches the actual resolution discussion, independent of any LLM judge.","supporting_citations":[{"cited_title":"Seven failure points when engineering a retrieval augmented generation system,","cited_arxiv_id":null,"evidence_quote":"Anchor for the 'retrieved but wrong' failure mode; the source/structure mismatch is framed as a failure point of RAG systems."},{"cited_title":"Episodic and semantic memory,","cited_arxiv_id":null,"evidence_quote":"Supplies the episodic-versus-semantic memory distinction that motivates treating resolved incidents as episodic operational memory."},{"cited_title":"Generative AI for self-adaptive systems: State of the art and research roadmap,","cited_arxiv_id":null,"evidence_quote":"Documents the open challenge that Knowledge components are populated with static models, the gap OM-RAG targets."},{"cited_title":"Case-based reasoning: Foundational issues, methodological variations, and system approaches,","cited_arxiv_id":null,"evidence_quote":"Provides the case-based reasoning principle that new problems be solved by retrieving structured prior cases."}],"review_version":1}