{"id":"c6829351-5d8f-488e-9cf6-a779ba6d7d0c","arxiv_id":"2608.10613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"By combining static test slicing with dynamic execution-trace slicing, CausalRepair builds compact causal contexts that let DeepSeek-V3 fix 313 Defects4J bugs with a 25-patch budget.","lead":"CausalRepair is a new automated program repair framework that uses program slicing to give large language models only the code needed to explain a test failure, instead of noisy full contexts. On the Defects4J benchmark it reports 313 correct fixes, more than prior state-of-the-art methods, at about $0.029 per bug.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The prompt template in Figure 3 leaks the exact buggy line to the LLM; the 313-bug headline may overstate the method's advantage over baselines that do not receive this hint.","rationale":"The reader's weakest assumption concerned the precision and completeness of dynamic slicing (Slicer4J, timeouts, failure not covered by the slice), which is a real threat to the mechanism's reliability. The reader's rationale also mentioned the buggy-line hint as an addressable fairness concern, but did not treat it as the primary load-bearing issue. I disagree with that ordering: the prompt template in Figure 3 explicitly tells the LLM which line is buggy, and this appears to be done outside the stated algorithm and without any ablation. If the marker comes from Defects4J metadata, the comparison against baselines is not merely noisy or incomplete; it is potentially confounded by ground-truth fault location leakage. Thus the most load-bearing concern is not whether Slicer4J is precise, but whether the evaluation isolates the contribution of dual-slicing from the contribution of an oracle-provided buggy-line hint. The proposed concrete test would settle this by rerunning with the marker removed. If the margin collapses, the paper's central claim needs substantial qualification; if the margin holds, the dynamic-slicing concerns raised by the reader remain the appropriate caveats. Because the available evidence does not yet resolve this, the existing CONDITIONAL verdict remains appropriate; I do not propose changing it, but I would tighten the condition to require the marker-free rerun before the headline number is used for comparison.","tokens_in":29201,"tokens_out":6191,"duration_ms":63400,"concrete_test":"In the released artifact, create a variant 'CausalRepair-NoBuggyMarker' that generates the same prompts except that the '// Buggy Line' comment is removed or replaced with a neutral marker such as '// relevant line'. Run this variant on Defects4J V1.2 and V2.0 with the same DeepSeek-V3 backbone, repair budget, and validation procedure, and compare correct fixes against Table 2. If the number of correct fixes drops by a substantial fraction of the 77-bug margin over ReinFix, the reported superiority is not established; if it remains near 313, the hint is not the driver of the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Figure 3 shows the actual repair prompt containing '// Buggy Line' markers on the root-cause statement and instructs the model that 'Buggy lines are marked ... The actual fix may involve changes around these lines or adding new statements if necessary.' This marker is absent from Algorithm 1's stated inputs (only F_unc and the test suite T are passed), is not listed among the position-aware markers in Section 3.2.3, and is not removed in any ablation. If this marker is derived from Defects4J's ground-truth fix location, then CausalRepair is receiving exact buggy-line information under the 'perfect fault localization' setting of Section 4.5, while the reproduced baselines receive only their default, unmarked contexts. The headline gap of 313 versus 236 correct fixes for ReinFix could then be partly or largely due to leaked location information rather than to dual-slicing. This is more load-bearing than dynamic-slice imprecision: slice incompleteness would cost individual fixes, but a ground-truth buggy-line hint threatens the fairness of the entire comparative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CausalRepair, a test-driven LLM-based automated program repair framework that constructs a 'minimal causal context' by applying static slicing to the failing test and dynamic slicing to the execution trace of the buggy program, and then feeds this context into a conversation-driven repair loop with patch augmentation. The evaluation is conducted on Defects4J V1.2/V2.0, Defects4J-Trans, RWB, and GitBug-Java. With a DeepSeek-V3 backbone, the authors report 313 correct fixes on Defects4J, outperforming the reproduced ReinFix (236) and TSAPR (224) baselines, and an average cost of $0.029 per fixed bug.","tokens_in":29372,"tokens_out":7846,"duration_ms":80587,"significance":"If the reported effect is real, CausalRepair is a substantial advance in LLM-based APR: it directly targets the recognized context-noise problem, provides a concrete mechanism (dual-slicing) for constructing failure-specific contexts, and reports large gains over strong baselines on a standard benchmark. The study also includes ablations, a contamination-reduced benchmark, a realistic-fault-localization robustness check, and releases an artifact, all of which are strengths. However, the central comparative claim currently rests on a prompt that appears to mark the exact buggy line, on hyperparameters selected using the same benchmark that is later reported as the headline result, and on a single stochastic run. These issues must be resolved before the claimed superiority can be accepted.","major_comments":[{"comment":"The prompt template shown in Figure 3 marks the exact defective statement with '// Buggy Line' and instructs the model that 'Buggy lines are marked ... The actual fix may involve changes around these lines or adding new statements if necessary.' This annotation is not described in Algorithm 1's inputs (which are only F_unc and the test suite), is not listed among the position-aware markers in Section 3.2.3, and is not removed in any ablation. If '// Buggy Line' is derived from the known defect location in Defects4J, then CausalRepair receives exact buggy-line information under the 'perfect fault localization' setting, while the reproduced baselines receive unmarked contexts; the headline gap of 313 vs. 236 correct fixes (Table 2) could then be attributable to leaked location information rather than to dual-slicing. Please clarify the source of this marker, add an ablation that removes it, and, if it is ground-truth-derived, rerun the comparison with the same information provided to the baselines.","section":"Figure 3 and Section 3.2.3"},{"comment":"The values m=5 and n=3 are selected by observing performance saturation on Defects4J in Figure 7, and the same Defects4J numbers are then reported as the headline results in Table 2. This is a form of test-set hyperparameter tuning: the reported 313 correct fixes is the best configuration found on the evaluation benchmark, not an unbiased estimate of the method's performance. The authors should tune hyperparameters on a separate validation set (e.g., one Defects4J version while reporting the other, or a held-out subset) and then report the performance of the chosen configuration, or at least report how the headline numbers change across the full grid rather than only the selected point.","section":"Section 5.3, Figure 7"},{"comment":"The repair process is stochastic because the sampling temperature is set to 1.0, yet all reported correct-fix counts come from a single run, with no variance or confidence intervals. Because the headline claim is a 77-bug advantage over ReinFix (313 vs. 236), and the reproduced baselines are also stochastic, a single run does not establish that the ordering is reliable. Please report multiple seeds (at least 3-5) with the range of correct fixes, or use deterministic decoding if the exact counts are meant to be reproducible.","section":"Section 4.5 and Section 5.1"},{"comment":"The paper's core premise is that the backward dynamic slice of the failing assertion contains the causally responsible statements, but this premise is never directly validated. No statistics are reported on slice size, precision, or completeness, and Section 4.5 notes that a 5-minute timeout causes the dynamic context to be omitted without saying how often this occurs. The ablation in Table 4 shows that removing dynamic slicing hurts performance, but it does not measure whether the slices actually contain the root cause when repairs succeed or fail. The authors should report the success rate of dynamic slicing, average slice size, and the fraction of correct repairs whose slice contains the ground-truth fix location, so that the 'causal context' claim can be assessed.","section":"Section 3.2.2 and Section 4.5"}],"minor_comments":[{"comment":"The function IdentifyFail(T, F_unc) is described only as 'locate the failing test function'; the paper should specify the procedure used when multiple tests fail or when the failing test is not uniquely determined.","section":"Algorithm 1, line 1"},{"comment":"The text lists only File Path Headers, Intent Preservation, and Position-Aware Markers as semantic anchors, but Figure 3 also shows external method/field lists and the 'Buggy Line' annotation. Please either align the figure with the described prompt construction or describe all prompt elements in the text.","section":"Section 3.2.3 and Figure 3"},{"comment":"The realistic fault-localization experiment reports 232 correct fixes with GZoltar/Ochiai top-50 candidates, but no baseline is evaluated under the same setting, so the result is not a comparative robustness check. Adding a ReinFix or TSAPR run under the same GZoltar setting would make this experiment interpretable.","section":"Section 6.2"},{"comment":"The cost column is labeled 'Money/Bug' but it is not stated whether the denominator is the number of fixed bugs or the number of all attempted bugs; please state the exact formula so the cost comparison is reproducible.","section":"Section 5.5, Table 6"},{"comment":"The term 'minimal causal context' is used as a technical claim, but no operational definition of minimality is given. I suggest defining it in terms of a measurable quantity, such as the size of the resulting context relative to the full function and test, or using a weaker term like 'reduced causal context'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is the '// Buggy Line' annotation in Figure 3. If this annotation is derived from the ground-truth defect location, the paper's comparative evaluation is not fair and the reported 313-bug result could be substantially inflated. I recommend asking the authors to provide the exact prompt template in the artifact and to run a no-marker ablation before making a final decision. The hyperparameter section and single-run evaluation are also fixable but require re-running experiments. The core idea is novel enough for ISSTA, but the current evidence does not yet support the claimed superiority."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a solid engineering contribution with a serious evaluation, but the 313-bug headline carries a load-bearing caveat. The prompt template in Figure 3 contains \"// Buggy Line\" markers on the root-cause statement, and the paper never explains where those markers come from. If they come from Defects4J ground truth, that is exact location information given to CausalRepair but not to the reproduced baselines, and it could explain a large share of the 77-bug gap over ReinFix. That needs to be resolved before the comparative claim is credible. What the paper does well: dual-slicing is a genuinely new combination for LLM-based APR. Test-side static slicing plus source-side dynamic slicing is a clean idea with a sensible motivation, and the ablations mostly support it. The evaluation is broad: Defects4J V1.2/V2.0, Defects4J-Trans, RWB, and GitBug-Java, plus a realistic-FL robustness check. Reproducing all LLM baselines under one backbone is the right call, and the cost reporting is clear. The open-sourced artifact is also a plus. Beyond the marker, the soft spots are secondary but real: the evaluation is a single stochastic run with no variance or confidence interval; m=5 and n=3 were selected on Defects4J and then reported on Defects4J; and correct-patch labeling relies on manual review, though the dual-review process helps. The hyperparameter tuning on the test benchmark is a genuine circularity concern but not damning, since the ablation in Figure 7 shows saturation rather than cherry-picking. Bottom line: this paper deserves a serious referee. The core mechanism is plausible, the experiments are extensive, and the concerns are addressable. I would ask the authors to clarify whether the \"// Buggy Line\" hint is derived from ground-truth fix locations, add an ablation without that marker, and report variance or repeat runs. If the marker is ground truth, the current headline overstates the advantage of dual-slicing. If it is not, the paper is in good shape. I would bring it to reading group either way, because the slicing recipe and the evaluation structure are worth discussing.","headline":"Genuinely new context-construction idea, but the prompt template appears to leak the exact buggy line; the 313-bug headline needs that resolved before it is credible.","tokens_in":605,"tokens_out":675,"would_cite":true,"duration_ms":30217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A minimal causal context—a purified failing test plus the executed statements behind the failed assertion—fixes 313 Defects4J bugs on DeepSeek-V3.","keywords":["Automated Program Repair","Large Language Models","Program Slicing","Dynamic Slicing","Static Slicing","Causal Context","Defects4J","Conversation-Driven Repair"],"falsifier":"Construct or select a bug whose root cause is a missing action, such as a deleted null-check or an uninitialized field, so the failure is caused by an absent statement rather than an executed one, and check whether CausalRepair fixes it while a baseline given the full method can; a failure here would show the dynamic slice omits the causal fact. A more direct check is to instrument the slicer and record, for each of the 313 claimed fixes, whether the developer patch's changed lines fall inside the dynamic slice; any fixed bug whose changed lines lie outside the slice would contradict the claim that the slice carries the causal context.","tokens_in":28959,"feed_emoji":"🐛","tokens_out":10494,"duration_ms":94395,"temperature":0.7,"pith_summary":"The paper argues that large language models fail at automated program repair not because they lack context but because the context is the wrong kind: full tests bury the failing assertion in unrelated scenarios, and static source retrieval includes code that never runs. It proposes a minimal causal context, defined as the essential set of dependencies required to explain a specific failure, and builds it by two slicing passes: a static backward slice of the failing test that keeps only the assertion and its setup, and a dynamic backward slice of the executed trace that keeps only the statements that influenced the failed assertion. On Defects4J V1.2 and V2.0 with DeepSeek-V3, this approach correctly fixes 313 bugs, 77 more than the best reproduced baseline, at an average cost of $0.029 per fixed bug. The paper's point is that causal relevance, not context size, is what lets an LLM repair code.","feed_headline":"Dual-slicing context fixes 313 Java bugs at $0.029 each","feed_subtitle":"Giving LLMs only the code that caused a test failure fixes 77 more bugs than the previous best method.","key_machinery":"The load-bearing object is the minimal causal context, instantiated by dual-slicing. On the test side, context-aware static slicing builds a System Dependence Graph and takes the transitive closure of data and control dependence from the failing assertion ($\\kappa = \\langle s_{fail}, V_{fail}\\rangle$) to obtain $S_{intra}$, then resolves inter-procedural dependencies for called methods and referenced fields. On the source side, execution-trace-based dynamic slicing maps the failing line to its last runtime event and computes backward dependence over a Dynamic Dependence Graph of the execution trace, yielding $S_{dyn}$. An AST-based reconstruction step re-embeds the fragmented statements into class and method skeletons and injects semantic anchors, so the final prompt is structurally coherent rather than a list of lines. This machinery does the filtering work: it removes unexecuted code and unrelated assertions while preserving the chain from a state-mutating test action to the failed assertion.","core_discovery":"The central claim is that a repair prompt built from the intersection of test semantics and actual runtime causality lets an LLM identify root causes it otherwise misses. CausalRepair starts from the failing test, computes a static backward slice from the failing assertion to remove unrelated test scenarios while preserving state-mutating calls and setup, then executes the program and computes a dynamic backward slice from the last execution event of that assertion, keeping only executed statements that influenced its outcome. These slices are merged into a structured prompt with file paths, Javadoc, and relative line numbers. In the paper's evaluation, this context produces 313 correct fixes on Defects4J (162 in V1.2, 151 in V2.0), 289 fixes on the contamination-reduced Defects4J-Trans benchmark, and 232 fixes under realistic spectrum-based fault localization, and removing the source-side dynamic slice alone lowers correct fixes by 26.5%.","pith_inferences":["A natural boundary of the approach is bugs whose root cause is a missing statement, such as a deleted null check or an uninitialized field: a backward dynamic slice can only contain executed statements, so the absent action never appears in the context.","The five-minute slicing budget and the Java-specific dynamic slicer imply that transferring the recipe to other languages depends on the availability of a comparable dynamic slicer; a static fallback would likely narrow the reported advantage.","The Defects4J-Trans result (289 fixes) supports robustness against memorization, but the stronger test would be applying CausalRepair to newly collected bugs with no public fix commits, where training-data overlap cannot help.","Because cost is driven by token count, the minimal causal context should also reduce latency in interactive repair deployments, not just API expenditure."],"forward_implications":["If the central claim holds, LLM-based APR should be built around execution-trace slices rather than retrieved static context; the paper reports 313 versus 236 correct Defects4J fixes for the strongest baseline under the same DeepSeek-V3 backbone.","Test context quality is as important as source context: reducing the test to only the failing line costs 38.3% of fixes, while using the full test costs 10.0%.","The slicing must be paired with iterative validation and patch augmentation: disabling either lowers correct fixes by 18.5% and 16.9%, respectively.","The gain transfers across models: with GPT-4o, CausalRepair still fixes 297 Defects4J bugs, so the improvement is attributable to the context rather than one backbone.","Under imperfect fault localization with a top-50 candidate list, CausalRepair fixes 232 Defects4J bugs, indicating the causal context compensates for noisy localization."],"supporting_citations":[{"why":"Supplies the dynamic slicer that records execution traces for the source-side causal context.","marker":"[6]"},{"why":"Provides the Defects4J benchmark (V1.2 and V2.0) on which the main repair results are measured.","marker":"[27]"},{"why":"Establishes the conversation-driven generate-validate-feedback loop that CausalRepair adapts in Phase 2.","marker":"[67]"},{"why":"ReinFix is the strongest reproduced baseline, and the retrieval-augmented approach that CausalRepair surpasses by 77 fixes.","marker":"[82]"},{"why":"TSAPR is the tree-search baseline that CausalRepair surpasses by 89 fixes under the same backbone.","marker":"[22]"},{"why":"ContrastRepair provides the contrastive test-case conversation baseline used in comparison and overlap analysis.","marker":"[28]"},{"why":"Defects4J-Trans supplies the contamination-reduced benchmark used to evaluate data-leakage robustness.","marker":"[35]"},{"why":"Provides the dependency-analysis formulation that inspires the intra-procedural test-side slicing.","marker":"[76]"},{"why":"Defines program slicing, the foundational technique that the dual-slicing strategy builds on.","marker":"[61]"},{"why":"Supplies the spectrum-based fault localization framework used in the realistic-FL evaluation.","marker":"[10]"}],"fun_headline_variants":["Dual-slicing context fixes 313 bugs at $0.029 each","CausalRepair: LLMs need only the code that caused the failure","Slicing test semantics and runtime traces boosts LLM repair by 77 bugs","CausalRepair cuts average repair cost to $0.029 per bug"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach stands on the assumption that the executed statements that influenced the failed assertion really contain the bug's cause, and that Slicer4J captures those statements within the five-minute trace budget; if the trace is truncated or imprecise, the LLM receives a context that cannot explain the failure.","fun_headline_variants_meta":{"raw":{"variants":["Dual-slicing context fixes 313 bugs at $0.029 each","CausalRepair: LLMs need only the code that caused the failure","Slicing test semantics and runtime traces boosts LLM repair by 77 bugs","CausalRepair cuts average repair cost to $0.029 per bug"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4218,"prompt_tokens":959,"completion_tokens":3259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":3174}},"tokens_in":575,"tokens_out":3259,"duration_ms":25488,"temperature":1.0,"reasoning_tokens":3174,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:50:40.691608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or select a bug whose root cause is a missing action, such as a deleted null-check or an uninitialized field, so the failure is caused by an absent statement rather than an executed one, and check whether CausalRepair fixes it while a baseline given the full method can; a failure here would show the dynamic slice omits the causal fact. A more direct check is to instrument the slicer and record, for each of the 313 claimed fixes, whether the developer patch's changed lines fall inside the dynamic slice; any fixed bug whose changed lines lie outside the slice would contradict the claim that the slice carries the causal context.","supporting_citations":[],"review_version":1}