{"id":"9d608c3c-8746-4495-8934-403cd2d549f5","arxiv_id":"2506.00172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Breakpoint generates code-repair benchmarks by corrupting real GitHub functions and shows that frontier AI coding agents solve easy repairs but fail completely on tasks requiring coordinated, system-wide changes.","lead":"The paper introduces Breakpoint, a system that automatically creates hard debugging tasks by injecting subtle bugs into real open-source software. On 930 such tasks, state-of-the-art AI coding agents fall from 55% success on easy tasks to 0% on the hardest, revealing a large gap in system-level reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'system-level reasoning' axis rests on a harmonic-centrality metric defined two incompatible ways (incoming in Eq. 3.5, outgoing in App. C.2) and never validated against runtime behavior; the difficulty decomposition may be an artifact of an unstated direction choice.","rationale":"The reader's conditional verdict is right, and I want to sharpen the reason. The system-level reasoning axis is the paper's distinctive contribution, but the metric defining it is not uniquely specified. The main text and appendix give opposite directionalities for harmonic centrality on a directed call graph. This is not a cosmetic typo: incoming vs. outgoing centrality selects different functions (low-level utilities vs. high-level orchestrators), so the logistic-regression coefficients, the hard-set selection, and the inference-time scaling results could all change if the direction is flipped. The appendix's internal note that PageRank correlates poorly with harmonic centrality also contradicts the main text's claim that harmonic centrality is correlated with PageRank, further suggesting the implementation may use a different formula than the one displayed. Because no code or task list is linked, the ambiguity cannot be resolved from the paper. I do not think this warrants rejection: the benchmark pipeline, 930-task scale, and test-based evaluation are solid, and the discrepancy is fixable by releasing code and re-running the analyses with the correct metric. But until then the central claim should be conditional on removing the ambiguity and validating the chosen metric (e.g., against dynamic call graphs or independent expert ratings).","tokens_in":11808,"tokens_out":9570,"duration_ms":100869,"concrete_test":"Release the task-generation code and centrality implementation; then recompute the Section 4.1.2 REMOVE-mode logistic regression twice, once with the Section 3.5 incoming harmonic centrality and once with the Appendix C.2 outgoing variant, on the same 930-task set. If the centrality coefficient changes sign, loses significance, or materially changes the hard-set composition, the paper must state which definition was actually used and rerun the scaling analyses; otherwise the difficulty decomposition is not robust to a definitional ambiguity that the text itself contains.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Breakpoint's scalable-difficulty and system-level-reasoning claims depend on Section 3.5's harmonic-centrality proxy. The definition is internally inconsistent: Section 3.5 defines H(f) = sum over f' of 1/dist(f', f) (shortest path from f' to f, i.e. incoming/dependency), while Appendix C.2 defines HC(f) = sum over g of 1/d(f, g) (shortest path from f to g, i.e. outgoing/control) and calls it an orchestrator metric. Since a call-graph edge (fa, fb) means fa calls fb, these are not equivalent: the first ranks low-level utilities that many callers reach; the second ranks high-level orchestrators that reach many callees. The main-text intuition ('significantly influence many other parts of the system') matches the outgoing formula, not the displayed one. The paper also claims harmonic centrality correlates with PageRank (Section 3.5), but Appendix C.3 reports PageRank has low correlation with harmonic centrality. No release code is linked, so the reader cannot tell which metric generated the logistic-regression results in Section 4.1.2, the hard-set selection, or the inference-time scaling conclusion. The static-call-graph extraction is also unspecified, so the metric cannot be independently recomputed. If the wrong (or an arbitrary) direction was used, the system-level difficulty axis and every conclusion built on it are not reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Breakpoint, a benchmark construction methodology that automatically generates code-repair tasks by corrupting functions in real-world Python repositories. Task difficulty is controlled along two claimed axes: local reasoning (measured by code-line count / cyclomatic complexity) and system-level reasoning (measured by call-graph harmonic centrality and the number of simultaneously corrupted interdependent functions). The authors evaluate frontier LLM agents on 930 tasks, reporting success rates from 55% on the easiest down to 0% on the hardest, and use logistic regression, Mann-Whitney tests, and agent telemetry to argue that difficulty decomposes into local and system-level components, that inference-time compute disproportionately helps system-level reasoning, and that different models exhibit different information-gathering strategies. A small human evaluation (eight tasks solved by the authors) is offered as calibration.","tokens_in":12014,"tokens_out":4400,"duration_ms":46566,"significance":"If the central claims hold, Breakpoint would be a valuable alternative to manually curated long-horizon benchmarks: it is automatic, uses existing test suites for objective evaluation, and produces a difficulty gradient that is tunable via static code metrics. The paper's strengths include the scale of the evaluation (930 tasks), the use of logistic regression with reported AIC and McFadden R^2, per-model difficulty curves with bootstrapped confidence intervals, Mann-Whitney analyses of solved-task distributions, and agent tool-use telemetry. However, the reproducibility and internal consistency of the system-level difficulty axis are currently problematic: the harmonic centrality definition is stated in two incompatible ways, the claimed correlation with PageRank is contradicted by the paper's own appendix, and the code/toolkit is not linked. These issues are load-bearing because centrality is used for hard-set construction, difficulty decomposition, and the inference-time scaling conclusions. The empirical patterns are plausible, but the manuscript cannot be accepted without resolving these points.","major_comments":[{"comment":"","section":"3.5 vs. C.2"},{"comment":"","section":"3.5 vs. C.3"},{"comment":"","section":"Abstract, 4.1.1, Fig. 3"},{"comment":"","section":"1, 3.6, 5, Fig. 3b"}],"minor_comments":[{"comment":"","section":"3.5"},{"comment":"","section":"4.1.1"},{"comment":"","section":"4.1.2"},{"comment":"","section":"4.4"},{"comment":"","section":"D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of cs.LG and the benchmark is potentially useful to the community. My recommendation is major revision rather than rejection because the central empirical pattern is plausible and the technical inconsistencies appear fixable: the authors need to commit to one definition of harmonic centrality, reconcile the PageRank correlation claim, report confidence intervals for the headline numbers, and release code/artifacts. If the code is not released or the centrality direction cannot be resolved, the paper's contribution would be substantially weakened. I would not hold the human evaluation size against the paper beyond a minor revision, as it is presented as preliminary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is good: take working GitHub repos, corrupt functions with an LLM, use failing tests as ground truth, and you get an auto-generated benchmark that can be scaled. Breakpoint adds genuinely useful pieces on top of R2E: multi-function corruption of interdependent functions, a remove/discovery task split, and a clean separation between local and system-level difficulty axes. The empirical gradient is credible at a high level: success falls off with complexity, centrality, and number of corruptions, and the logistic regressions on 930 tasks are a reasonable first pass.\n\nThe soft spot is load-bearing. The system-level difficulty axis rests on harmonic centrality, and the paper defines it two incompatible ways. Section 3.5 writes H(f) = sum_{f'} 1/dist(f', f) — distances from other functions to f, i.e. incoming dependency. Appendix C.2 writes HC(f) = sum_g 1/d(f, g) — distances from f to other functions, i.e. outgoing control. The main-text intuition (\"influence many other parts of the system\") matches the outgoing version, not the displayed formula. C.3 then says PageRank has low correlation with harmonic and distance discount centrality because of \"opposite directionality\" — which directly contradicts Section 3.5's claim that harmonic centrality is correlated with PageRank. The reader cannot tell which formula generated the regression results, the hard-set selection, or the scaling conclusions. Since the code and the 930-task list are promised but not linked, this is not independently checkable. That's not a minor typo; it calls into question the main measurement.\n\nOther concerns are more ordinary: the 55% and 0% headline numbers have no confidence intervals, the human evaluation is eight tasks solved by the authors, and \"scale to arbitrary difficulty\" oversells what was shown (up to 4 corruptions, 90th percentile selection). These are fixable with a release, repeated runs, and modest wording changes.\n\nIf the authors pick one centrality direction, re-run the regressions, and ship the code, this is a solid contribution the community will use. As is, I would not trust the quantitative difficulty decomposition without those fixes. The benchmark idea and behavioral analysis (tool usage, iteration curves) are worth engaging with.\n\nSerious editor should send to review — major revision, not desk reject. The central conflict is resolvable and the benchmark is useful.","headline":"Good benchmark idea, but the central difficulty metric is defined two incompatible ways, and that undercuts the main quantitative claims.","tokens_in":12634,"tokens_out":3531,"would_cite":false,"duration_ms":32604,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Breakpoint automatically turns real codebases into code-repair tasks whose difficulty is controlled by code complexity and call-graph centrality, driving frontier model success from 55% to 0%.","keywords":["LLM code agents","benchmark methodology","system-level reasoning","code-repair tasks","call-graph centrality","difficulty scaling","inverse problems","software engineering evaluation"],"falsifier":"Take a sample of Breakpoint tasks, ask independent programmers to fix them and rate the degree of cross-codebase coordination each repair required, and compare those ratings to harmonic centrality; if low-centrality functions turn out to be just as hard or high-centrality functions turn out to be trivial, the claimed system-level difficulty axis is not measuring what it claims.","tokens_in":11519,"feed_emoji":"🐛","tokens_out":8041,"duration_ms":85021,"temperature":0.7,"pith_summary":"Breakpoint is a benchmarking methodology that automatically converts working GitHub repositories into code-repair tasks by adversarially corrupting or deleting functions and requiring an agent to restore the original tests to passing. The paper's central claim is that this inverse-problem construction scales difficulty in a controlled way: across more than 900 tasks, state-of-the-art models solve up to 55% of the easiest tasks and 0% of the hardest, with difficulty set by code complexity and call-graph centrality. This matters because existing long-horizon coding benchmarks rely on manually curated issues, which is expensive to expand and leads to quick saturation. A scalable automatic benchmark would let researchers track whether agents understand the interconnected systems they modify, not just whether they can fix isolated bugs.","feed_headline":"Auto-built code-repair tests take LLMs from 55% to 0% success","feed_subtitle":"Breakpoint corrupts real repositories into repair tasks, with difficulty set by code complexity and call-graph centrality.","key_machinery":"The central object is the inverse repair task built from a repository's call graph and test suite. A repository is represented as functions, tests, and a directed call graph; a task corrupts one or several functions and asks the agent to produce a patch that makes the original tests pass. Difficulty is controlled by two static measures: code-line or cyclomatic complexity for local reasoning, and harmonic centrality $H(f) = \\sum_{f' \\neq f} 1/\\mathrm{dist}(f', f)$ over the call graph for system-level reasoning. Harmonic centrality is the load-bearing metric: it ranks functions by how close the rest of the system is, so repairing central functions supposedly requires understanding the wider system. The paper demonstrates difficulty control through a joint logistic regression and through engineered hard sets that select tasks above the 90th percentile in both complexity and centrality.","core_discovery":"On the paper's own terms, the discovery is that difficulty in code-repair can be decomposed into two independent, static axes and then scaled by construction. Local reasoning difficulty is controlled by code complexity metrics such as cyclomatic complexity; system-level reasoning difficulty is controlled by harmonic centrality in the repository's call graph and by the number of simultaneously corrupted functions that are close in that graph. The empirical core is the resulting gradient: success rates fall monotonically with both axes, four simultaneous corruptions defeat every tested model, and a 'hard' set at the 90th percentile of both complexity and centrality cuts success by roughly 40% per model. The same construction also enables behavioral analysis, showing that models differ in information-gathering and in whether added compute improves local or system-level reasoning.","pith_inferences":["Editorial inference: the same inverse-problem construction could be applied to non-Python repositories or to other executable artifacts with a dependency graph and a test suite, since the method only requires a known-good system and objective success checks.","Editorial inference: harmonic centrality on a static call graph will undercount dynamic dispatch, reflection, and vendored code; a stronger test of the proxy would compare centrality rankings against runtime call tracing or independent human judgments of cross-system impact.","Editorial inference: because every corrupted task has known ground truth and failing tests provide a reward signal, Breakpoint could be repurposed from pure evaluation into a training environment for reinforcement learning on system-level repair."],"forward_implications":["Difficulty can be scaled automatically: selecting functions above the 90th percentile in both complexity and centrality reduces per-model performance by roughly 40%, and four simultaneous corruptions drop success to 0%.","The two difficulty axes are complementary: a joint logistic model on code-lines and harmonic centrality fits success better than either alone, with both predictors significant at $p < 10^{-4}$.","Improvements from stronger reasoning and from extra inference-time compute differ: the tested reasoning model shifts solved tasks toward higher code complexity but not centrality, while increasing tool and test iterations shifts solved tasks toward both axes, with a larger effect on centrality.","Most of the benefit of test feedback occurs on the first revision; later submissions add little, especially in the mode where the corrupted function is unknown."],"supporting_citations":[{"why":"Supplies SWE-bench, the manually curated real-issue benchmark whose curation bottleneck and saturation Breakpoint aims to overcome.","marker":"Jimenez et al., 2024"},{"why":"Provides R2E, the prior inverse-problem environment that generates programming tasks from repositories without labeled data, which Breakpoint extends to deeply embedded functions.","marker":"Jain et al., 2024"},{"why":"Supplies SWE-Lancer, a manually curated long-horizon benchmark showing the difficulty of scaling human-labeled tasks.","marker":"Miserendino et al., 2025"},{"why":"Provides the METR long-task framing that motivates measuring agents on long-horizon, system-level work.","marker":"Kwa et al., 2025"},{"why":"Supplies AgentBench as a related interactive benchmark context for long-horizon agent evaluation.","marker":"Liu et al., 2024"}],"fun_headline_variants":["Auto-corrupt code repos to scale LLM repair tests from 55% to 0%","Complexity and call-graph knobs drop LLM code-repair success to 0%","Two axes—code complexity and call-graph—tune LLM repair difficulty","Breakpoint: corrupting repo functions scales LLM code-repair difficulty","From 55% to 0%: auto-generated code-repair tasks scale LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that harmonic centrality computed from a static call graph tells you how much system-level understanding a repair actually needs; if what is central on paper is peripheral at runtime, the difficulty decomposition and hard-task construction inherit that error.","fun_headline_variants_meta":{"raw":{"variants":["Auto-corrupt code repos to scale LLM repair tests from 55% to 0%","Complexity and call-graph knobs drop LLM code-repair success to 0%","Two axes—code complexity and call-graph—tune LLM repair difficulty","Breakpoint: corrupting repo functions scales LLM code-repair difficulty","From 55% to 0%: auto-generated code-repair tasks scale LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001749,"raw_usage":{"total_tokens":6884,"prompt_tokens":902,"completion_tokens":5982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":5869}},"tokens_in":518,"tokens_out":5982,"duration_ms":47982,"temperature":1.0,"reasoning_tokens":5869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:10:59.060868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of Breakpoint tasks, ask independent programmers to fix them and rate the degree of cross-codebase coordination each repair required, and compare those ratings to harmonic centrality; if low-centrality functions turn out to be just as hard or high-centrality functions turn out to be trivial, the claimed system-level difficulty axis is not measuring what it claims.","supporting_citations":[],"review_version":1}