{"id":"fa79f78e-c56f-4290-80e1-5b072ec9c23c","arxiv_id":"2602.02138","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CAM ranks how much each intermediate output of a multi-agent code-generation system causally matters for final correctness, then uses that ranking to repair failures and prune features.","lead":"This paper introduces CAM, a framework that classifies intermediate outputs of multi-agent code-generation systems into features and uses counterfactual error injection to rank which features matter for final code correctness. It reports that those rankings agree with human judgment and can guide failure repair and token-cost pruning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's influence-set pruning (lines 19–20) assumes an unproven error-containment property; if intervening on S does not fail, it prunes all length-ℓ combinations from E(S), but direct intervention on downstream features can fail even when indirect perturbation through S does not.","rationale":"The reader's weakest assumption identifies the most load-bearing concern: the soundness of Algorithm 1's pruning, which directly affects the correctness of the FR rankings that the central claims depend on. The central claim of CAM is that it systematically quantifies intermediate-feature contributions to MACGS correctness; this quantification is only as good as the search that discovers important feature combinations. If the error-containment property is false, the algorithm can systematically miss combinations that would be discovered by direct intervention, biasing the aggregated rankings and all downstream results (repair success, pruning trade-offs, hybrid architecture conclusions). The paper provides no proof of this property, and the intuitive justification is insufficient because direct and indirect perturbations are semantically different. The human-agreement pilot provides some empirical support but only validates the aggregate ranking over 12 features, not the completeness of the search; a missed combination that does not affect the top-level ranking would go undetected. Other issues, such as the inconsistent statistics between the abstract and body and the loose connection to formal actual causality, are real but less fundamental to the empirical validity. The concern is addressable via the proposed ablation test, and the reader's CONDITIONAL verdict remains appropriate: not an outright rejection, but conditional on resolving this algorithmic assumption and releasing artifacts. Therefore the verdict should remain UNCHANGED.","tokens_in":26165,"tokens_out":6987,"duration_ms":69580,"concrete_test":"On a random sample of, say, 20–30 problems per setting, run CAM twice: (a) with the current Alg. 1 including lines 19–20 pruning, and (b) with that pruning disabled while keeping the same query budget N=100 and early stopping (or, if feasible, exhaustive length-2 and length-3 search). Compare the discovered important-feature sets S_p and the resulting FR rankings. If (b) finds combinations that (a) misses, or if the FR ranking changes by more than a small margin (e.g., Kendall < 0.9 between (a) and (b)), the pruning property is unsound and the reported rankings/applications need re-evaluation. Ideally also instrument the check to log all pruned combinations and test a random subset of them directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 4.3, Alg. 1 lines 19–20 prune every length-ℓ combination drawn from E(S) whenever intervening on S (with |S|=ℓ) does not cause a failure. The text justifies this by claiming that intervening on S already perturbs all features in E(S) transitively, so any combination from E(S) is a 'less comprehensive perturbation.' This is not entailed by the definitions: E(S) is simply the set of features whose semantic similarity to their original values falls below θ (Eq. 2). Indirect perturbation via S produces particular corrupted values, while a direct intervention on S'⊆E(S) uses independently LLM-generated errors. There is no monotonicity guarantee that if the indirect, possibly self-corrected, changes are non-fatal, then direct, different corruptions of the same features are also non-fatal. Two individually weak errors can interact into a failure (indeed the paper's context-dependent-feature finding depends on such interactions). If the property fails, the algorithm silently omits important feature combinations, biasing the FR aggregation (Eq. 4) and all downstream rankings, repair, and pruning results. The human-agreement pilot (Table 4) is an aggregate Kendall correlation over 12 features and cannot detect omitted combinations that happen not to change the top-level ranking.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAM, a framework that categorizes intermediate outputs of multi-agent code generation systems (MACGS) into 12 features across four categories, constructs a causal graph, and uses LLM-generated counterfactual interventions to find minimal feature combinations whose simultaneous corruption leads to failure. It then aggregates a feature responsibility (FR) score via Eq. (4) and validates the resulting ranking against human annotations (Kendall 0.76–0.91). The paper reports several downstream findings: context-dependent features, hybrid-backend Pass@1 gains up to 7.2%, causality-guided failure repair at 73.3% success, and up to 66.8% token reduction via feature pruning.","tokens_in":26576,"tokens_out":5433,"duration_ms":57204,"significance":"If the causal ranking is trustworthy, CAM is a useful and timely contribution: intermediate-output analysis in MACGS is currently manual or coarse-grained, and the paper provides a concrete feature taxonomy, an intervention-generation method, and two practical applications with quantitative benefits. The human-agreement study is a genuine strength, and the hybrid-backend and pruning results are actionable. However, the central claim of formal causality is not actually instantiated, and the search algorithm relies on an unproven pruning property. These issues affect the correctness of the rankings and therefore of the empirical findings, so the paper cannot be accepted in its current form; the framework and data collection are substantial enough that a major revision is warranted.","major_comments":[{"comment":"The paper invokes Halpern actual causality, but the method does not implement Definitions 2–3. Interventions are applied only to problems that originally succeed, so AC1 (actuality) is never satisfied for a failure outcome; the discovered combinations are failure-inducing counterfactuals, not actual causes of observed failures. Moreover, FR in Eq. (4) is a squared-inverse-length heuristic, not the responsibility 1/(1+|W|) from Eq. (1), and no contingency set W is computed. This is a substantive mismatch with the title and abstract claims. The authors should either instantiate the formal definitions or explicitly reframe CAM as intervention-based importance analysis rather than actual-causality analysis.","section":"Sec. 2.2 and Sec. 4.3 (Def. 3, Eq. 4)"},{"comment":"The pruning step assumes an error-containment property: if intervening on S does not induce failure, then no length-ℓ combination drawn from E(S) can induce failure. This is not entailed by Eq. (2), because E(S) only records that downstream features have changed semantically after the S intervention; the specific corrupted values are generated by that intervention. A direct intervention on S' subset of E(S) uses independently generated errors, and there is no monotonicity guarantee that two individually weak errors cannot interact into a failure. In fact, the context-dependence finding in Sec. 7.1 presupposes such interactions. If the property fails, important combinations are silently discarded, biasing S_p, Eq. (4), Table 6, and all downstream applications. The pilot study cannot detect this because Table 4 validates only aggregate 12-feature rankings. Please provide a proof under stat","section":"Sec. 4.3, Alg. 1 lines 19–20"},{"comment":"The human-agreement study validates only the aggregate FR ranking over 12 features. The central new finding—context-dependent features, e.g., 78.8% of Program_Lang appearances occurring in combinations—is an interaction-level claim derived from the same search procedure that defines interactions. Because the search is incomplete (budget N, patience k, and the pruning in Alg. 1), the interaction distribution is not independently confirmed. The feature vocabulary is also partly constitutive: if importance is defined through minimal failure-inducing combinations, interaction effects are guaranteed by construction; the empirical content lies in which features interact and how often. This needs external validation, e.g., human annotation of a sample of discovered minimal combinations or a held-out comparison.","section":"Sec. 6, Table 4 and Sec. 7.1, Table 6"},{"comment":"FR rankings and all downstream results depend on several thresholds and design choices: θ in Eq. (2), L_max, k, N, and the weighting exponent in Eq. (4). Only k and L_max receive sensitivity analysis. In particular, θ determines the influence set and therefore the search and pruning behavior, and the exponent determines the FR aggregation; the stability of Tables 5–10 with respect to these choices should be reported. Without this, the reader cannot tell whether the main findings are robust or artifacts of parameter settings.","section":"Sec. 5 Parameters and Sec. 9.2"}],"minor_comments":[{"comment":"The abstract reports 73.6% repair success, 33.6% token reduction, and 7.3% Pass@1 improvement, while the introduction and tables report 73.3%, 66.8%, and 7.2%. Please harmonize these numbers.","section":"Abstract and Section 7/8"},{"comment":"The generalizability claim for PairCoder and MapCoder is prose-only; a table mapping each system's intermediate outputs to the four categories would make the extensibility argument concrete.","section":"Sec. 9.1"},{"comment":"The collective influence set uses union over previously computed subsets; if some subsets were pruned or never executed, its definition should state how missing E(S') values are handled (e.g., treated as empty).","section":"Eq. (3)"},{"comment":"The manual inspection of 100 counterfactual interventions confirms semantic coherence, but not that the interventions represent realistic errors rather than merely plausible paraphrases; a small set of examples in the text would help the reader assess this.","section":"Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software engineering venue and the empirical footprint is impressive. My main concern is that the formal causality framing and the search algorithm's pruning property are load-bearing, and both need substantial strengthening. If the authors can reframe the contribution more modestly or prove/validate the pruning property, the paper could become a solid contribution. I do not see grounds for rejection at this stage, but the revision must address the conceptual and algorithmic issues, not just presentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading if you work on LLM-based multi-agent code generation. It tackles a real problem—which intermediate outputs matter for correctness—and offers a concrete way to find out. The function-oriented taxonomy is sensible and portable across MACGS, the LLM-based counterfactual interventions are a clever way to generate realistic errors, and the empirical findings are interesting: the context-dependence of Program_Lang (only 21.2% of its importance shows up alone) is a genuinely useful observation, and the hybrid-backend experiment is a nice payoff. The human-agreement study (Kendall 0.76–0.91) gives some confidence that the rankings track something real. The repair and pruning applications show practical value, and the token-reduction numbers are eye-catching.\n\nThe soft spots are not fatal but they are real. The connection to actual causality is overstated. What they compute in Eq. 4 is a squared-inverse weighting of combination lengths, not the responsibility defined in Eq. 1. That is fine as a heuristic, but the paper should not lean so hard on the formal machinery. More importantly, the search algorithm's pruning (Alg. 1, lines 19–20) assumes a monotonicity property that is unproven: if intervening on S does not break the code, they prune all length-ℓ combinations drawn from S's influence set. The justification is that S already perturbs those features transitively, but that conflates indirect, possibly self-corrected changes with direct interventions that introduce independent errors. The paper's own finding about context-dependent features shows that two weak errors can combine into a failure, so the assumption can fail. If it does, the search silently omits important combinations and the FR rankings are biased. The human-agreement pilot checks the final rankings, not the search's completeness, so it cannot catch this.\n\nThere are also smaller annoyances: the abstract's key numbers don't match the body (73.6% vs 73.3%; 33.6% vs 66.8%; 7.3% vs 7.2%), and no code or data are released, which makes the results hard to reproduce.\n\nWho this is for: researchers in LLM-agent software engineering who care about intermediate-output analysis, optimization, or hybrid architectures. A serious referee should engage with it, but the most important thing to examine is the pruning assumption—either prove it under stated conditions or relax the pruning and show the cost. With that fixed, plus a harmonized abstract and released artifacts, this would be a solid contribution. Send it to peer review, but flag the monotonicity issue clearly.","headline":"A useful empirical framework for ranking intermediate features in multi-agent code generation, but the search pruning rests on an unproven monotonicity assumption and the causality framing is looser than claimed.","tokens_in":705,"tokens_out":1048,"would_cite":false,"duration_ms":36939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A causality-based analysis framework quantifies the causal contribution of each intermediate output to multi-agent code generation correctness, and its feature-responsibility rankings enable targeted failure repair and token-cost pruning.","keywords":["multi-agent systems","code generation","causal analysis","counterfactual intervention","feature importance","LLM agents","software testing","feature responsibility"],"falsifier":"Run Algorithm 1 on a sample of problems with the influence-set pruning disabled, exhaustively testing all length-ℓ feature combinations up to the same execution budget, and compare the discovered important feature sets. If the unpruned search finds any important combination that the pruned search missed — particularly a combination whose features all lie in E(S) for some non-failing set S — the monotonicity assumption is false and the FR rankings are systematically biased.","tokens_in":26077,"feed_emoji":"🎯","tokens_out":4452,"duration_ms":41488,"temperature":0.7,"pith_summary":"Multi-agent code generation systems produce many intermediate outputs, but which of those outputs actually determine whether the final code is correct has been opaque. The paper proposes CAM, a causality-based framework that decomposes intermediate outputs into a small set of semantic features, simulates realistic errors on each feature, and measures the causal effect of these errors on final correctness. The resulting feature-responsibility rankings agree strongly with human annotation (Kendall tau 0.76–0.91) and reveal that some features matter mainly in combination with others. These rankings support two practical applications: repairing failures by improving the top-ranked features achieves a 73.3% success rate, and pruning low-ranked features saves up to 33.6% of intermediate tokens without performance loss. If valid, the framework turns an otherwise opaque multi-agent pipeline into a measurable, optimization-ready account of what truly matters.","feed_headline":"Counterfactual tests reveal which agent outputs decide code success","feed_subtitle":"A causality-based framework ranks intermediate outputs, fixing failures at 73% and cutting tokens by a third.","key_machinery":"The central machinery is a three-part causal-measurement pipeline. First, a function-oriented categorization maps intermediate outputs to twelve feature variables arranged in a directed acyclic graph, using each feature's final value to avoid cycles from iterative refinement. Second, the influence set E(S) — the set of downstream features whose semantics change when a feature set S is intervened — and its collective approximation are used to greedily prioritize feature-combination search and to prune unpromising combinations. Third, Feature Responsibility FR(f_i) = Σ over important sets S containing f_i of (1/|S|)^2 aggregates the importance of each feature, weighting shorter (more decisive)","core_discovery":"The paper establishes that intermediate outputs of a MACGS can be systematically mapped to twelve semantic features organized into four functional categories, and that the causal importance of each feature can be quantified through counterfactual interventions. Using an LLM to generate realistic errors on features, the method identifies minimal feature combinations whose simultaneous intervention causes failure, and aggregates these into a Feature Responsibility (FR) ranking. The ranking matches human judgment across four datasets and three backend LLMs, uncovers context-dependent features (e.g., Program_Lang is important in combination with other features in 78.8% of cases), and demonstrate","pith_inferences":["The entire ranking inherits the fidelity of the LLM used for counterfactual intervention: errors that the intervention model cannot render as plausible will be invisible to the analysis, so the method's output is bounded by the intervention engine's realism.","The FR rankings are computed on a fixed problem set; transferring them to a new domain would require re-running the analysis, but the method itself is designed to be re-instantiated.","The context-dependent feature finding suggests a testable extension: automatically checking pairwise consistency of high-FR features (e.g., Program_Lang × Data_Struct) could serve as an inexpensive pre-submission filter for MACGS.","The pruning application hints at a cost-geometry trade-off: the fact that pruning a few features sometimes helps suggests some low-FR features add noise rather than information, a hypothesis that could be probed on other architectures."],"forward_implications":["Quality assurance for MACGS should incorporate cross-feature consistency checks, because many failures stem from interactions between individually plausible features rather than from any single flaw.","Hybrid multi-backend architectures — assigning different LLMs to different stages according to relative feature strengths — can yield measurable Pass@1 gains, especially on design-heavy tasks.","Failure repair can be prioritized by improving the top-3 FR-ranked features rather than exhaustively inspecting all intermediate outputs, raising the repair success rate from 27–41% to 73.3%.","Pruning 2–4 low-FR features in production can reduce intermediate token consumption by up to 33.6% with no performance loss (or even improvement), while aggressive pruning degrades performance.","Feature importance is not static: it shifts with dataset difficulty and domain, so resource allocation and optimization should be adapted to the target problem distribution."],"fun_headline_variants":["Causal test ranks agent outputs: 73% of failures fixed","Which agent outputs matter? Causality says prune 33% tokens","Prune by causal importance: 33% fewer tokens, same code quality","Hybrid agents get 7.3% boost via causal output ranking","Causal framework: top features fix 73% failures, save 33% tokens"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The search prunes combinations under an unproven monotonicity: if intervening on a feature set S does not cause failure, then no length-ℓ combination drawn from S's influence set E(S) can cause failure either (Sec. 4.3, Algorithm 1, lines 19–20); if individually weak errors interact into a failure, this pruning biases the entire feature-responsibility ranking.","fun_headline_variants_meta":{"raw":{"variants":["Causal test ranks agent outputs: 73% of failures fixed","Which agent outputs matter? Causality says prune 33% tokens","Prune by causal importance: 33% fewer tokens, same code quality","Hybrid agents get 7.3% boost via causal output ranking","Causal framework: top features fix 73% failures, save 33% tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001312,"raw_usage":{"total_tokens":5209,"prompt_tokens":796,"completion_tokens":4413,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":4315}},"tokens_in":540,"tokens_out":4413,"duration_ms":27601,"temperature":1.0,"reasoning_tokens":4315,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:26:08.159260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 on a sample of problems with the influence-set pruning disabled, exhaustively testing all length-ℓ feature combinations up to the same execution budget, and compare the discovered important feature sets. If the unpruned search finds any important combination that the pruned search missed — particularly a combination whose features all lie in E(S) for some non-failing set S — the monotonicity assumption is false and the FR rankings are systematically biased.","supporting_citations":[],"review_version":1}