{"id":"0c615035-0fc1-4126-921c-513637fad431","arxiv_id":"2501.13699","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new large-scale benchmark shows that current large language models pass under half of repository execution tests when asked to infer dependencies.","lead":"DI-BENCH is a new benchmark of 581 real GitHub repositories with dependency information hidden, used to test whether large language models can infer the packages a project needs to run. It gives AI developers a standardized way to measure a practical bottleneck in automated software generation: even the best tested model passed only around 43% of execution tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CI-as-oracle fidelity is the load-bearing risk: the act-based test-job oracle and the LLM-selected test job are unvalidated, so the 42.9% executability claim could be an artifact of the evaluation harness.","rationale":"The reader's weakest-assumption diagnosis is correct: the CI-as-oracle premise is the most load-bearing piece of the paper's argument. If act is systematically lenient or strict, the curation filter changes and every executability number shifts, directly undermining the central empirical conclusion rather than merely adjusting a headline. The paper reports no reliability check for the oracle, and the LLM-assisted test-job locator adds a second unvalidated component: job mis-selection would make the 'test suite' something the project never ran. The internal discrepancy between 42.9% (Python) and 43.2% (JavaScript) in Table 3 is real but secondary; it changes the exact best number, not the qualitative finding that all models remain far below passing. The proposed sample-based validation is feasible because the dataset and workflows are public; it would either confirm the oracle or quantify the needed corrections. Until such validation is provided, conditional acceptance remains the appropriate verdict.","tokens_in":16085,"tokens_out":3356,"duration_ms":31525,"concrete_test":"Sample 80-100 DI-BENCH repositories stratified by language and by GPT-4o pass/fail outcome. For each, run the original test workflow on GitHub Actions (or a manually built environment from the repo's documented setup) after masking dependencies and inserting ground-truth dependencies, and compare the binary pass/fail result with the act oracle; also have two human annotators verify that the LLM-selected job is indeed the project's intended test job. Report disagreement rates separately for job selection and pass/fail. If combined disagreement exceeds roughly 5%, recompute all executability tables with the corrected oracle.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DI-BENCH's executability metric is computed by re-running each repository's CI test job under act after dependency masking (Section 4.2). The central claim that dependency inference is a major bottleneck assumes this oracle faithfully measures whether a dependency specification is correct. That assumption has three unverified links: (1) the test job itself is selected by an LLM-assisted procedure with no reported accuracy; if the wrong job is chosen, both curation and evaluation test the wrong thing; (2) act does not emulate all GitHub Actions features (services, OS-specific actions, caching), so a workflow that passes under act may fail on GitHub or vice versa; (3) the paper's own Limitation 3 concedes test coverage may not be exhaustive. Because the same oracle is used to filter repositories and to score models, any systematic discrepancy biases both the dataset and the headline 42.9% executability. No independent validation against manually constructed environments or real CI runs is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DI-BENCH is a new benchmark and evaluation framework for LLM dependency inference at the repository level. It collects 581 real-world GitHub repositories across Python, C#, Rust, and JavaScript, masks dependency-related sections in build configuration files, and asks an LLM to regenerate them. Evaluation combines textual precision/recall and fake-rate metrics with an execution-based metric: each repository's original CI test job is run locally under act after installing the model-predicted dependencies, and a repository is counted as executable only if all tests pass. Experiments with several LLMs and three prompting strategies (All-In-One, File-Iterate, Imports-Only) report that the best method achieves only a 42.9% execution pass rate, with significantly lower rates for compiled languages and large repositories. The paper also analyzes failure categories, the effect of dependency metadata, hallucinated dependencies, and model/repository size trends.","tokens_in":16309,"tokens_out":5711,"duration_ms":52987,"significance":"If the execution oracle is trustworthy, DI-BENCH fills a real gap: it is larger than existing execution-based repository benchmarks and is the first to focus specifically on dependency inference. The automated CI-reuse pipeline is a plausible route to scalable, continuously updatable repository-level evaluation, and the empirical finding that state-of-the-art LLMs pass tests on fewer than half of the repositories is a useful, falsifiable result. I credit the authors for releasing code and data, for specifying exact model versions and greedy decoding, and for making the dataset construction pipeline fully automated. The main value of the benchmark depends, however, on the fidelity of the CI-based oracle, which is currently unvalidated; the missing comparison against non-LLM dependency tools and the absence of statistical confidence also weaken the central empirical claim.","major_comments":[{"comment":"The act-based CI oracle is load-bearing but unvalidated. The paper reuses each repository's CI test job as the environment for both curation filtering and final executability evaluation, yet it reports no comparison between act pass/fail outcomes and real GitHub Actions runs or manually configured environments. act is known not to emulate all GitHub Actions features (services, OS-specific actions, caching), and the manuscript itself concedes in Limitation 3 that test coverage may not be exhaustive. Because the same oracle is used to construct the dataset and to score models, any systematic discrepancy is inherited by both the 581-repository dataset and the headline 42.9% executability. I ask for a validation study on a sample of repositories: run the original unmasked workflows on GitHub Actions, compare pass/fail with act, and build a small manually verified environment for each language, reporting the discrepancy rate and its effect on the executability metric.","section":"§4.2 and §5"},{"comment":"The LLM-assisted procedure that selects which CI job is the testing job has no reported accuracy, and this choice determines what is executed during both curation and evaluation. If the wrong job is selected, a repository may be admitted or rejected for the wrong reason, and a model's reconstructed dependencies may be tested against a linting or publishing job instead of the test job. The paper should report the accuracy of job selection on a labeled sample, describe the LLM prompt and parsing used for this step, and state what fallback or manual verification is applied when the selector is uncertain.","section":"§4.2 (Test Job Locating)"},{"comment":"The abstract states that the current best-performing model achieves only a 42.9% execution pass rate, but Table 3 and Table 4 report a 43.2% executability rate for JavaScript with the All-In-One method on GPT-4o (Regular subset). The headline number should be corrected to refer explicitly to the global maximum, to a specific language/setting, or to an average; as written, the paper's central empirical claim is internally inconsistent.","section":"Abstract vs. Table 3/Table 4"},{"comment":"The evaluation includes no existing dependency inference tool as a baseline. Since DI-BENCH is positioned as a benchmark for dependency inference, the absence of non-LLM baselines (e.g., pigar/pipreqs for Python, cargo-based dependency resolution for Rust, npm/yarn resolution for JavaScript) makes it hard to interpret the 42.9% result: is this a property of LLMs, or of the task when measured with an execution oracle? Adding at least one traditional/static tool per language would calibrate the difficulty of the benchmark and would directly support the claim that current LLMs are the bottleneck.","section":"§5 (Baseline Methods) and §6"},{"comment":"All results are single-run point estimates with no confidence intervals, significance tests, or variance information. Given that several reported differences are small (e.g., Python executability 17.3 vs. 22.4 for DeepSeek-Coder vs. Qwen in Table 4; Rust 1.0 vs. 2.0 in Table 4), sampling or nondeterminism could change model or method rankings. The paper should report bootstrap confidence intervals or repeat runs for a subset of configurations, or otherwise justify that the reported gaps are meaningful.","section":"§6 and Tables 3, 4, 7"}],"minor_comments":[{"comment":"In the last paragraph, 'ike Rust' should be 'like Rust'.","section":"Appendix B.2"},{"comment":"The caption contains the typo 'exeutability rate'; it should be 'executability rate'.","section":"Table 6 caption"},{"comment":"The benchmark name appears as 'DI-B ENCH' with an unwanted space in many places; please normalize it to 'DI-BENCH'.","section":"Throughout"},{"comment":"The reference 'damnever, 2024' is a GitHub tool page rather than an archival publication; if it is kept, it should be formatted as a software/tool reference with a URL and access date.","section":"Section 2"},{"comment":"The pipeline is described as 'fully automated,' but the Test Job Locating step is LLM-assisted; the text should reconcile these statements, for example by saying 'fully automated with an LLM-based selector.'","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a plausible fit for the venue and the core dataset contribution is substantial. The main risk is the unvalidated CI oracle, which I have made a major comment; this is fixable with a validation study and appropriate caveats. The abstract inconsistency and missing non-LLM baselines also need to be addressed before publication. I do not see grounds for rejection, assuming the oracle validation is supplied."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful benchmark paper that should go to review, but the headline 42.9% needs to be read as \"under act with an LLM-chosen test job,\" not as an oracle measurement.\n\nWhat's actually new: a 581-repo, four-language dependency-inference benchmark with execution-based evaluation through reused CI workflows. That's a real step up from previous repo benchmarks — the largest prior test-execution benchmark had 25 repos. The CI-as-harness idea is clever and pragmatic, and the paper reports both textual and execution metrics, plus failure analysis and ablations (oracle metadata, hallucination impact). Those ablations are informative.\n\nSoft spots, in order of importance:\n\n1. The CI oracle is the main caveat. The paper uses act to run workflows locally, and the test job is selected by an LLM-assisted procedure with no reported accuracy. act cannot emulate every GitHub Actions feature. The paper honestly notes in Limitation 3 that test coverage may not be exhaustive, but it doesn't validate the oracle against a manual environment or real CI runs. That doesn't sink the paper — the low pass rates are consistent across models and languages, and using the same oracle for curation and evaluation makes it self-consistent — but the absolute numbers are \"pass under act with a particular job selection,\" not \"correct dependencies in the wild.\" I'd want to see that validation before citing the 42.9% as a hard fact.\n\n2. No non-LLM baseline. Existing tools like pigar or the ICSE 2022 method are cited but not run. For a benchmark paper, that's a missed opportunity, not a fatal flaw.\n\n3. Reporting slips. The abstract says \"42.9% best\" while Table 3 shows JavaScript at 43.2. No confidence intervals or significance tests. Dataset and code are announced but not pinned to a specific commit. All minor, but a referee should ask for cleanup.\n\nI disagree with the stress-test framing that the 42.9% could be an artifact of the harness. That's too strong. The oracle could go either way, and the relative rankings across models are still meaningful. The risk is precisely that the absolute executability numbers are environment-dependent, which the paper already concedes in its limitations.\n\nWho it's for: anyone building or evaluating LLM coding agents, and benchmark builders who want a template for CI-driven evaluation. I'd bring it to a reading group and would cite it if I were working on repository-level code generation. It deserves serious peer review, with requests for oracle validation, tool baselines, and cleaned-up numbers.","headline":"A credible, useful benchmark for dependency inference held back by an unvalidated CI oracle and a few reporting slips, but still worth reviewing seriously.","tokens_in":16818,"tokens_out":2692,"would_cite":true,"duration_ms":23611,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DI-BENCH masks repository dependencies and re-runs each project's real CI tests; the best model passes only 42.9% of the 581-repository set.","keywords":["dependency inference","large language models","repository-level code generation","execution-based evaluation","continuous integration","build configuration","benchmark","hallucination"],"falsifier":"Take a random sample of DI-BENCH repositories, run the same inferred-dependency outputs on the actual hosted CI service (or in hand-constructed environments checked by a human), and compare pass/fail outcomes with the act-based local run; if a substantial fraction flip status, the executability metric is not a stable oracle. A complementary check would look for repositories whose tests pass with an obviously wrong dependency set, which would reveal coverage gaps.","tokens_in":15830,"feed_emoji":"📦","tokens_out":6600,"duration_ms":52348,"temperature":0.7,"pith_summary":"This paper argues that dependency inference—figuring out which internal modules and external packages a code repository needs in order to build and run—is a separate, under-evaluated capability that currently blocks end-to-end LLM code generation. To test that capability, it introduces DI-BENCH, a benchmark of 581 real-world repositories across Python, Rust, C#, and JavaScript with dependency sections removed from build files. Each repository's own continuous-integration test job is reused as the grading oracle, so evaluation runs the project's real tests instead of only comparing text. Under this execution-based measure, the best model tested passes only 42.9% of regular-sized repositories, and roughly 10% of Rust and C# repositories. The paper concludes that dependency inference, especially dependency metadata such as version constraints, is a major open bottleneck.","feed_headline":"Best LLM passes only 42.9% of dependency inference tests","feed_subtitle":"A 581-repo benchmark runs each project's own CI tests to see whether AI-inferred dependencies actually work.","key_machinery":"The engine of the benchmark is the masked build-file task: the model receives the full source tree plus build files with dependency sections removed, and must emit complete build files; the ground truth is the repository's original configuration. For evaluation, DI-BENCH reuses each repository's own CI test job, executed locally through a GitHub-Actions-compatible runner after the inferred dependencies are installed, scoring 1 only if all tests pass. This CI-as-oracle design makes both curation and evaluation fully automated, letting the authors scale to 581 verified repositories, 23 times the size of the previous largest execution-backed repository benchmark. Textual precision, recall, and F1 on the inferred dependency list sit alongside the executability score and a fake rate that counts generated packages or versions that cannot be found in the ecosystem.","core_discovery":"Its central finding is that contemporary LLMs cannot yet reliably reconstruct the dependency list that makes a repository executable, and that textual agreement with ground truth overstates their competence. On the regular subset, the best-performing model achieves a 42.9% execution pass rate on Python and 43.2% on JavaScript, but only 11.2% on Rust and 13.5% on C#. Replacing the model's predicted dependency metadata with the true versions and extras raises Python executability to 55.1% and Rust to 38.8%, showing that version and metadata errors—not just missing package names—are a major failure cause. Removing hallucinated dependencies that do not exist in package registries or the repository also improves executability. The decline in pass rate with dependency count and repository size indicates that long-context and many-dependency settings remain unsolved.","pith_inferences":["If the local CI runner's behavior diverges from the real hosted CI service, the executability numbers could shift; a validation study comparing the two on a sample would tighten the benchmark's claim.","Because the oracle inherits whatever test coverage a repository's contributors wrote, executability should be read as 'passes the project's own tests,' not 'is fully correct'; blind spots in tests would make bad dependencies look acceptable.","The same masked-build-file task could be extended to transitive dependency resolution and lockfile correctness, where the version-pinning failure mode is likely even harsher.","The reported 42.9% ceiling suggests that agentic methods that install dependencies and read execution errors, rather than a single static prompt, are a natural next system to benchmark."],"forward_implications":["Repository-level code generation evaluations that skip execution likely overstate LLM capability, since textual accuracy can be high while executability is low.","Dependency metadata—versions, extras, feature flags—deserves study as its own bottleneck, not just package-name recall.","CI reuse offers a path to continuously updatable, execution-based benchmarks at scales that manual environment setup cannot reach.","The performance gap between scripting languages (Python, JavaScript) and compiled languages (Rust, C#) suggests separate challenges in each ecosystem's build model.","Generated dependencies that do not exist at all, measured by fake rate, are a concrete failure mode to optimize against."],"supporting_citations":[{"why":"Supplies the motivating evidence that over 50% of runtime errors in generated repositories come from dependency-related issues.","marker":"(Qian et al., 2024)"},{"why":"DevBench baseline whose reported dependency-related failures motivate a dedicated benchmark.","marker":"(Li et al., 2024a)"},{"why":"MetaGPT evidence that missing or wrongly generated dependencies are a major hallucination type in full-project generation.","marker":"(Hong et al., 2024)"},{"why":"Prior repository benchmark that the paper contrasts with: only about 1% of its patches touch build configurations.","marker":"(Jimenez et al., 2023)"},{"why":"EvoCodeBench was the largest prior execution-backed repository benchmark at 25 repositories, establishing the scale gap DI-BENCH fills.","marker":"(Li et al., 2024b)"},{"why":"Supplies the local GitHub Actions runner ('act') used to execute test jobs during curation and evaluation.","marker":"(nektos, 2024)"},{"why":"Provides the parsing library used to extract import/use statements for the Imports-Only baseline.","marker":"(tree-sitter, 2024)"}],"fun_headline_variants":["LLMs fail 57% of dependency inference tests","DI-BENCH: best LLM scores 42.9% on 581 repos","Dependency inference remains unsolved: 42.9% top pass","New benchmark reveals LLM dependency skills gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's scores stand or fall on the premise that re-running a repository's own CI test job with the locally installed inferred dependencies is a faithful, complete measure of whether those dependencies are right; if the local runner behaves differently from the real CI service, or if the repository's tests miss important code paths, the executability rate will misstate model capability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail 57% of dependency inference tests","DI-BENCH: best LLM scores 42.9% on 581 repos","Dependency inference remains unsolved: 42.9% top pass","New benchmark reveals LLM dependency skills gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1248,"prompt_tokens":867,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":483,"tokens_out":381,"duration_ms":4050,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:40:46.816416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of DI-BENCH repositories, run the same inferred-dependency outputs on the actual hosted CI service (or in hand-constructed environments checked by a human), and compare pass/fail outcomes with the act-based local run; if a substantial fraction flip status, the executability metric is not a stable oracle. A complementary check would look for repositories whose tests pass with an obviously wrong dependency set, which would reveal coverage gaps.","supporting_citations":[],"review_version":1}