{"id":"8c2a81a3-a44c-4f3e-a40a-64bc4fd37f4b","arxiv_id":"2607.25996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.","lead":"A new benchmark, RepoReasoner, tests whether large language models can reason across multiple files in real code repositories by predicting function outputs and reconstructing call chains from noisy context. Initial results show that even with perfect context, the best model answers only 69.1% of output-prediction tasks correctly, exposing a persistent cross-file reasoning gap.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth depends on an unvalidated dynamic tracer; if hunter misses subprocess/C-extension/dynamic-import calls, RQ1 oracle contexts and RQ2 call-chain labels are incomplete, making the measured ceiling artificially low.","rationale":"The reader’s weakest assumption identifies the same load-bearing point: the oracle context and call-chain ground truth both depend on the dynamic tracer being complete. That is the right concern because the central quantitative claims — 69.1% Pass@1 under oracle context and the high-precision/low-recall pattern — would both be distorted if file-level transitions are missing. The paper otherwise has real strengths: ground truth is derived from actual execution rather than static heuristics, the two tasks are cleanly separated, and the results are reported across diverse models. The concern is not an external disagreement with consensus; it is an internal dependency of the benchmark’s validity that the paper itself acknowledges but does not yet resolve. Since the reader already made the verdict CONDITIONAL and this concern is the main condition, no verdict adjustment is needed; the concrete test above would settle whether the concern actually lands.","tokens_in":22176,"tokens_out":5473,"duration_ms":60544,"concrete_test":"Re-run the Stage II tracer on the 169 tests that define call-chain ground truth, but install hunter in every spawned worker and thread (e.g., via multiprocessing initializer / sitecustomize and threading.settrace) and also enable Python audit hooks to catch eval/exec and dynamic imports. Diff the resulting file-transition multisets against the released ground-truth call chains. If any repository file is added or removed for tests using joblib, multiprocessing, subprocess, or C extensions, the RQ1 oracle contexts and RQ2 recall labels are incomplete and the headline ceiling must be recomputed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark’s ground truth rests on the dynamic tracer described in Section 3.2.2: a custom pytest plugin plus hunter that supposedly “records every function call during the test runs, capturing its origin file, destination file, and the precise sequence of file-level transitions.” Both the oracle context used in RQ1 and the ground-truth call chains used in RQ2 are derived directly from these traces. The paper reports no completeness check: no comparison against static call graphs, no instrumentation of subprocesses, no audit of C-extension, eval/exec, or dynamic-import paths. This is not a hypothetical concern: the selected scientific repositories are exactly the kind of code that uses multiprocessing/joblib — the paper’s own Fig. 1 example is a joblib Parallel test with n_jobs=2, and joblib’s default loky backend executes in worker processes. A Python-level tracer installed only in the parent process will miss calls made in those workers. If any repository file is omitted, then the “oracle” context is not actually complete, so the 69.1% Pass@1 ceiling in RQ1 is an under-estimate of model capability, and the low recall in RQ2 is partly label error rather than model error. Section 6.1 concedes that the pipeline “could introduce errors” and only promises future validation; that admitted gap sits exactly on the load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RepoReasoner, a benchmark for repository-level code reasoning with two tasks: Output Prediction, where a model fills a masked assertion by simulating cross-file execution, and Call Chain Prediction, where a model identifies the set/sequence of files invoked by a test from a noisy file pool. Ground truth is produced by a custom pytest plugin plus the hunter tracing library inside Dockerized repositories, augmented by LLM-based I/O rewriting intended to reduce memorization. Seven LLMs are evaluated under oracle and retrieval contexts. The central findings are that even with oracle context the best model reaches only 69.1% Pass@1 on Output Prediction, and that models show high precision but low recall on Call Chain Prediction, with universal performance drops on rewritten data and mixed effects from longer contexts.","tokens_in":22437,"tokens_out":6728,"duration_ms":65778,"significance":"If the benchmark is valid, it addresses a real gap: existing code-reasoning benchmarks largely operate at function level, while repository-level reasoning is both practically important and under-evaluated. The design has genuine strengths: ground truth comes from real execution rather than static heuristics; the oracle-context setup cleanly isolates reasoning from retrieval; and the I/O rewriting is a sensible first attempt to detect memorization. The error analysis (RQ5) is also useful. However, the benchmark's load-bearing assumption is that the dynamic tracer captures every relevant file-level call, and the paper does not validate this. Several additional issues — oracle context truncation, a confounded memorization comparison, and missing statistical uncertainty — affect the strength of the headline claims. These are correctable, so the contribution is potentially solid but not yet established as presented.","major_comments":[{"comment":"The completeness of the dynamic tracer is load-bearing for both tasks. The paper states the tracer 'records every function call' and that the oracle context is 'complete', but no validation is reported. The Fig. 1 example uses joblib Parallel with n_jobs=2; joblib's loky backend runs in child processes, and a Python-level tracer installed only in the parent process would miss those calls. C extensions, eval/exec, and dynamic imports are also plausible blind spots. If any file is missed, the RQ1 oracle context is not perfect (lowering the measured ceiling) and the RQ2 ground-truth call chains are incomplete (lowering recall). Section 6.1 concedes that pipeline bugs 'could introduce errors' and defers 'more thorough validation' to future work. Please add a concrete completeness audit — e.g., compare tracer output against static call graphs, run known tests with subprocesses, or manually tr","section":"§3.2.2, Fig. 1, §6.1"},{"comment":"The oracle context is described as containing 'exactly the ground-truth files' and 'minimal yet complete', but RQ1 uses '10k tokens containing exactly the ground-truth files'. The paper does not state what happens when the ground-truth call-chain files exceed 10k tokens. If the context is truncated, the oracle is not complete and the 69.1% Pass@1 ceiling is an underestimate of model capability. If instances with oversized contexts are excluded, the benchmark may be biased toward simpler tests. Please specify the handling (truncation, exclusion, or per-instance token budget) and report how many instances are affected.","section":"§4.3.3, §5.1"},{"comment":"The memorization conclusion is confounded with task difficulty. The rewriting pipeline changes input values and expected outputs while preserving code logic, so the rewritten instances may simply be harder for any reasoner — e.g., more unusual constants, edge-case types, or less familiar value distributions — rather than exposing memorization. The paper provides no control for inherent difficulty (such as measuring a non-memorizing baseline or matching value distributions between original and rewritten sets). Without such a control, the universal performance drop in Table 3 does not isolate memorization. Please add a difficulty-matching analysis or a control condition that separates reasoning difficulty from training-data familiarity.","section":"§3.2.3, Table 3, RQ3"},{"comment":"Pass@1 is computed as the correctness of the first of five stochastic samples (Eq. (1) with k=1), and the paper reports no confidence intervals or significance tests. The headline ceiling of 69.1% is thus a single-draw point estimate with variance, and model comparisons such as 'R1-Distill-Qwen-14B performs worse than its base model' could easily be noise. Since the paper makes strong claims about ceilings and rankings, please report confidence intervals (e.g., bootstrap) or use the standard unbiased Pass@k estimator over the five samples, and indicate whether Pass@1 is actually the first-sample criterion.","section":"§4.2, Eq. (1), Table 1"}],"minor_comments":[{"comment":"Figure 3's call-chain section lists 'initial.py, domain.py, filed.py' — 'filed.py' appears to be a typo for 'field.py'. Please check filenames.","section":"§3.3"},{"comment":"The definition of EM for Call Chain Prediction says 'identical to the ground-truth sequence', but the task description says 'list of files'. Clarify whether the order of files matters in the ground truth and in the EM metric.","section":"§4.3.2"},{"comment":"Table 3's I/O-Rewritten columns do not report whether the rewritten instances were also evaluated under the same oracle context length and whether the same token truncation rules applied.","section":"§5.3"},{"comment":"The sentence 'we rigorously tested these tools across diverse codebases' is vague. If a validation was performed, it should be described in detail or moved to an appendix; otherwise the claim is unverifiable.","section":"§6.1"},{"comment":"The phrase 'scientific repositories' and 'scientific domains' in §3.2.1 is used but the benchmark could benefit from a breakdown of the 14 included repositories and their domains.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a good idea, but the validity hinges on the unvalidated dynamic tracer. The authors should be encouraged to provide a tracer validation artifact, ideally comparing against static call graphs and subprocess scenarios, and to clarify the oracle truncation policy. The memorization analysis also needs a difficulty control. These are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the news: this is a solid benchmark paper that deserves a real referee. The oracle-context setup is the right way to separate retrieval from reasoning, and the two tasks—output prediction and call-chain prediction—are complementary. The headline result, 69.1% Pass@1 for DeepSeek-R1 even when given exactly the files the test uses, is a believable statement that cross-file execution reasoning is still hard.\n\nWhat is new: bringing CRUXEval-style output prediction to multi-file repositories, plus the call-chain reconstruction task with noisy file pools. The construction pipeline uses real pytest execution, not static analysis, for ground truth, and the LLM-based I/O rewriting is a reasonable first attempt at reducing memorization. The manual check of 100 instances shows they care that tasks actually need repository-level knowledge. They also link data and code, which is the right move.\n\nThe soft spot is the one the stress-test identified, and I think it lands. Both the oracle context and the call-chain labels come from a custom pytest plugin plus hunter, and the paper gives no evidence that this tracer sees every file-level call. Scientific Python uses subprocesses, joblib worker processes, C extensions, and eval/exec; hunter will not trace inside a worker or a C library. The paper's own Figure 1 is a joblib test. If any file is missing from the trace, the \"perfect\" context is not perfect, and the 69.1% ceiling and low recall are underestimates. Section 6.1 admits the pipeline could introduce errors and only promises future validation. That is the load-bearing gap.\n\nSmaller problems: Pass@1 is scored from the first of five stochastic samples, so the differences between models in Table 1 are within noise but no intervals are reported. And RQ3's memorization comparison confounds difficulty with rewriting—the rewritten instances are not shown to be equally hard, so the universal performance drop supports memorization concerns but does not prove them. Both can be fixed with better reporting and a referee's push.\n\nWho should read it: anyone building or using repository-level code benchmarks. The conclusions directionally match what I would expect, and the benchmark is reusable. I would send it to peer review with a request for tracer validation, confidence intervals, and a difficulty-controlled memorization check.","headline":"Useful benchmark with a clean oracle design; the absolute ceiling numbers rest on a dynamic tracer the paper does not validate, so treat them as conditional.","tokens_in":22987,"tokens_out":4309,"would_cite":true,"duration_ms":36802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repository-level code reasoning is a distinct, largely unsolved ability for LLMs: even with oracle context—the exact files a test touches—the best evaluated model scores only 69.1% on output prediction.","keywords":["repository-level code reasoning","large language models","benchmark","output prediction","call chain prediction","dynamic tracing","memorization vs reasoning","long-context"],"falsifier":"Run a sample of RepoReasoner test suites under a second, independent tracing method—for example, a sys.settrace-based profiler or static call-graph analysis—and compare the resulting file-level call chains. If the two tracers disagree on a nontrivial fraction of instances, or if model recall rises sharply when the oracle context is replaced by a broader static-analysis-reachable file set, the reported performance ceiling and recall figures are artifacts of the tracer rather than genuine model limits.","tokens_in":22023,"feed_emoji":"🧩","tokens_out":8508,"duration_ms":80053,"temperature":0.7,"pith_summary":"RepoReasoner asks whether large language models can reason about code the way developers do: across file boundaries, tracking state through a whole repository, not just inside one function. The paper builds a benchmark from real Python projects by dynamically tracing pytest runs to capture ground-truth file-level call chains, then challenges models to predict a masked assertion's value (Output Prediction) and to identify which files a test actually invokes from a noisy pool (Call Chain Prediction). Even with perfect oracle context, the best evaluated model answers only 69.1% of Output Prediction tasks correctly, and no model reconstructs full call chains reliably—precision is high, recall low. Models also lose accuracy when the same logic is rewritten with different inputs and outputs, revealing partial reliance on memorization, and longer contexts sometimes hurt more than they help. If these findings hold, cross-file inference is the bottleneck, not retrieval or context length.","feed_headline":"Scores just 69.1%: best LLM still can't trace code across files","feed_subtitle":"A new repo-level benchmark finds even the best models miss multi-hop file dependencies and lean on memorization.","key_machinery":"The load-bearing machinery is the benchmark construction pipeline. A custom pytest plugin built on the hunter tracing library records every function call during test runs, capturing each call's origin file, destination file, and the sequence of file-level transitions; this dynamic trace provides both the ground-truth call chains and the oracle context for evaluation. An LLM-assisted rewriting stage alters input values and expected outputs while preserving logic, producing instances less likely to be memorized, and assertion masking turns real tests into Output Prediction tasks while file pools mix signal files with distractors for Call Chain Prediction. Evaluation then measures Pass@k for ou","core_discovery":"The paper's central claim is that contemporary LLMs cannot reliably perform repository-level code reasoning even when the exact files a test touches are supplied. Under oracle context with 10k tokens of ground-truth source code, the best evaluated model reaches only 69.1% Pass@1 on Output Prediction, and on Call Chain Prediction the oracle setting's best F1 is 66.3% with 21.3% exact match. The high-precision/low-recall pattern is interpreted as evidence that models recognize direct imports but lose multi-hop, runtime-dependent dependencies; the universal drop on I/O-rewritten data is interpreted as evidence of partial memorization; and the mixed effect of longer contexts indicates that noise","pith_inferences":["The tracer's completeness is the key unverified link: calls made through C extensions, dynamic imports, eval/exec, or pytest fixture machinery may be invisible to the hunter-based trace. If so, oracle contexts would omit files models actually need, making the 69.1% ceiling and low recall artificially pessimistic.","The high-precision/low-recall signature is exactly what a model that reasons over import statements rather than runtime dispatch would produce. A natural next experiment is to compare RepoReasoner results against a pure static import-graph baseline to separate 'tracing architecture' from 'recalling imports.'","Since the rewriting stage changes only inputs and expected outputs, it tests robustness to surface variation, not to algorithmic novelty. Refactoring function structure or reordering call chains—an extension the paper explicitly leaves for future work—would likely widen the memorization gap.","Because the tracing approach is Python-specific (pytest and hunter), extension to other languages will require a different dynamic-analysis backend, so cross-language generalization is a real constraint on the benchmark's reach."],"forward_implications":["A perfect-context ceiling near 69% on Output Prediction implies that repository-level reasoning is a separate capability from retrieval, and current models have not mastered it.","High precision but low recall on Call Chain Prediction means models tend to confirm direct dependencies but miss the deeper structural paths that define real architectures, so architectural understanding remains incomplete.","The consistent drop on I/O-rewritten data implies benchmarks that use only original tests overstate genuine reasoning; test-driven rewriting should become a standard part of code-reasoning evaluation.","Increasing context from 10k to 30k tokens did not reliably improve results and sometimes lowered them, so future long-context systems need noise-robust attention, not just larger windows.","The automated pipeline—Docker environment setup, dynamic tracing, rewriting, and ground-truth packaging—is designed to extend to larger repositories and other languages, making the benchmark scaffold reusable."],"fun_headline_variants":["Oracle context still caps LLM code tracing at 69.1%","Cross-file code reasoning: best LLM only 69.1% Pass@1","LLMs struggle with repo-level code flow, even with full context","New repo benchmark: LLMs recall code but don't reason across files","Even with oracle files, LLMs hit 69.1% on repo tracing"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Section 3.2.2 asserts that the tracer 'records every function call during the test runs,' but no completeness check is reported; if calls through C extensions, dynamic imports, eval/exec, or pytest fixtures are missed, the oracle context omits needed files and the call-chain labels are wrong, making both the 69.1% ceiling and the low recall numbers artificially low.","fun_headline_variants_meta":{"raw":{"variants":["Oracle context still caps LLM code tracing at 69.1%","Cross-file code reasoning: best LLM only 69.1% Pass@1","LLMs struggle with repo-level code flow, even with full context","New repo benchmark: LLMs recall code but don't reason across files","Even with oracle files, LLMs hit 69.1% on repo tracing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1275,"prompt_tokens":772,"completion_tokens":503,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":516,"tokens_out":503,"duration_ms":4955,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:52:35.493218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a sample of RepoReasoner test suites under a second, independent tracing method—for example, a sys.settrace-based profiler or static call-graph analysis—and compare the resulting file-level call chains. If the two tracers disagree on a nontrivial fraction of instances, or if model recall rises sharply when the oracle context is replaced by a broader static-analysis-reachable file set, the reported performance ceiling and recall figures are artifacts of the tracer rather than genuine model limits.","supporting_citations":[],"review_version":1}