Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DI-BENCH masks repository dependencies and re-runs each project's real CI tests; the best model passes only 42.9% of the 581-repository set.

desk verdict A credible, useful benchmark for dependency inference held back by an unvalidated CI oracle and a few reporting slips, but still worth reviewing seriously. read the letter →

arxiv 2501.13699 v1 pith:S547DYWH submitted 2025-01-23 cs.CL cs.SE

Linghao Zhang , Junhao Wang , Shilin He , Chaoyun Zhang , Yu Kang , Bowen Li , Jiaheng Wen , Chengxing Xie
show 8 more authors
Maoquan Wang Yufan Huang Elsie Nallipogu Qingwei Lin Yingnong Dang Saravan Rajmohan Dongmei Zhang Qi Zhang
This is my paper · ORCID
classification cs.CLcs.SE
keywords dependencyinferencelargelanguagemodelsrepository-levelcodegenerationexecution-basedevaluationcontinuousintegrationbuildconfigurationbenchmarkhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that dependency inference—figuring out which internal modules and external packages a code repository needs in order to build and run—is a separate, under-evaluated capability that currently blocks end-to-end LLM code generation. To test that capability, it introduces DI-BENCH, a benchmark of 581 real-world repositories across Python, Rust, C#, and JavaScript with dependency sections removed from build files. Each repository's own continuous-integration test job is reused as the grading oracle, so evaluation runs the project's real tests instead of only comparing text. Under this execution-based measure, the best model tested passes only 42.9% of regular-sized repositories, and roughly 10% of Rust and C# repositories. The paper concludes that dependency inference, especially dependency metadata such as version constraints, is a major open bottleneck.

What carries the argument

The engine of the benchmark is the masked build-file task: the model receives the full source tree plus build files with dependency sections removed, and must emit complete build files; the ground truth is the repository's original configuration. For evaluation, DI-BENCH reuses each repository's own CI test job, executed locally through a GitHub-Actions-compatible runner after the inferred dependencies are installed, scoring 1 only if all tests pass. This CI-as-oracle design makes both curation and evaluation fully automated, letting the authors scale to 581 verified repositories, 23 times the size of the previous largest execution-backed repository benchmark. Textual precision, recall, and F1 on the inferred dependency list sit alongside the executability score and a fake rate that counts generated packages or versions that cannot be found in the ecosystem.

What would settle it

Take a random sample of DI-BENCH repositories, run the same inferred-dependency outputs on the actual hosted CI service (or in hand-constructed environments checked by a human), and compare pass/fail outcomes with the act-based local run; if a substantial fraction flip status, the executability metric is not a stable oracle. A complementary check would look for repositories whose tests pass with an obviously wrong dependency set, which would reveal coverage gaps.

Watch

Extended reading notes

Core claim

Its central finding is that contemporary LLMs cannot yet reliably reconstruct the dependency list that makes a repository executable, and that textual agreement with ground truth overstates their competence. On the regular subset, the best-performing model achieves a 42.9% execution pass rate on Python and 43.2% on JavaScript, but only 11.2% on Rust and 13.5% on C#. Replacing the model's predicted dependency metadata with the true versions and extras raises Python executability to 55.1% and Rust to 38.8%, showing that version and metadata errors—not just missing package names—are a major failure cause. Removing hallucinated dependencies that do not exist in package registries or the repository also improves executability. The decline in pass rate with dependency count and repository size indicates that long-context and many-dependency settings remain unsolved.

Load-bearing premise

The benchmark's scores stand or fall on the premise that re-running a repository's own CI test job with the locally installed inferred dependencies is a faithful, complete measure of whether those dependencies are right; if the local runner behaves differently from the real CI service, or if the repository's tests miss important code paths, the executability rate will misstate model capability.

Editorial extensions

If this is right

  • Repository-level code generation evaluations that skip execution likely overstate LLM capability, since textual accuracy can be high while executability is low.
  • Dependency metadata—versions, extras, feature flags—deserves study as its own bottleneck, not just package-name recall.
  • CI reuse offers a path to continuously updatable, execution-based benchmarks at scales that manual environment setup cannot reach.
  • The performance gap between scripting languages (Python, JavaScript) and compiled languages (Rust, C#) suggests separate challenges in each ecosystem's build model.
  • Generated dependencies that do not exist at all, measured by fake rate, are a concrete failure mode to optimize against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the local CI runner's behavior diverges from the real hosted CI service, the executability numbers could shift; a validation study comparing the two on a sample would tighten the benchmark's claim.
  • Because the oracle inherits whatever test coverage a repository's contributors wrote, executability should be read as 'passes the project's own tests,' not 'is fully correct'; blind spots in tests would make bad dependencies look acceptable.
  • The same masked-build-file task could be extended to transitive dependency resolution and lockfile correctness, where the version-pinning failure mode is likely even harsher.
  • The reported 42.9% ceiling suggests that agentic methods that install dependencies and read execution errors, rather than a single static prompt, are a natural next system to benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. DI-BENCH is a new benchmark and evaluation framework for LLM dependency inference at the repository level. It collects 581 real-world GitHub repositories across Python, C#, Rust, and JavaScript, masks dependency-related sections in build configuration files, and asks an LLM to regenerate them. Evaluation combines textual precision/recall and fake-rate metrics with an execution-based metric: each repository's original CI test job is run locally under act after installing the model-predicted dependencies, and a repository is counted as executable only if all tests pass. Experiments with several LLMs and three prompting strategies (All-In-One, File-Iterate, Imports-Only) report that the best method achieves only a 42.9% execution pass rate, with significantly lower rates for compiled languages and large repositories. The paper also analyzes failure categories, the effect of dependency metadata, hallucinated dependencies, and model/repository size trends.

Significance. If the execution oracle is trustworthy, DI-BENCH fills a real gap: it is larger than existing execution-based repository benchmarks and is the first to focus specifically on dependency inference. The automated CI-reuse pipeline is a plausible route to scalable, continuously updatable repository-level evaluation, and the empirical finding that state-of-the-art LLMs pass tests on fewer than half of the repositories is a useful, falsifiable result. I credit the authors for releasing code and data, for specifying exact model versions and greedy decoding, and for making the dataset construction pipeline fully automated. The main value of the benchmark depends, however, on the fidelity of the CI-based oracle, which is currently unvalidated; the missing comparison against non-LLM dependency tools and the absence of statistical confidence also weaken the central empirical claim.

major comments (5)
  1. [§4.2 and §5] The act-based CI oracle is load-bearing but unvalidated. The paper reuses each repository's CI test job as the environment for both curation filtering and final executability evaluation, yet it reports no comparison between act pass/fail outcomes and real GitHub Actions runs or manually configured environments. act is known not to emulate all GitHub Actions features (services, OS-specific actions, caching), and the manuscript itself concedes in Limitation 3 that test coverage may not be exhaustive. Because the same oracle is used to construct the dataset and to score models, any systematic discrepancy is inherited by both the 581-repository dataset and the headline 42.9% executability. I ask for a validation study on a sample of repositories: run the original unmasked workflows on GitHub Actions, compare pass/fail with act, and build a small manually verified environment for each language, reporting the discrepancy rate and its effect on the executability metric.
  2. [§4.2 (Test Job Locating)] The LLM-assisted procedure that selects which CI job is the testing job has no reported accuracy, and this choice determines what is executed during both curation and evaluation. If the wrong job is selected, a repository may be admitted or rejected for the wrong reason, and a model's reconstructed dependencies may be tested against a linting or publishing job instead of the test job. The paper should report the accuracy of job selection on a labeled sample, describe the LLM prompt and parsing used for this step, and state what fallback or manual verification is applied when the selector is uncertain.
  3. [Abstract vs. Table 3/Table 4] The abstract states that the current best-performing model achieves only a 42.9% execution pass rate, but Table 3 and Table 4 report a 43.2% executability rate for JavaScript with the All-In-One method on GPT-4o (Regular subset). The headline number should be corrected to refer explicitly to the global maximum, to a specific language/setting, or to an average; as written, the paper's central empirical claim is internally inconsistent.
  4. [§5 (Baseline Methods) and §6] The evaluation includes no existing dependency inference tool as a baseline. Since DI-BENCH is positioned as a benchmark for dependency inference, the absence of non-LLM baselines (e.g., pigar/pipreqs for Python, cargo-based dependency resolution for Rust, npm/yarn resolution for JavaScript) makes it hard to interpret the 42.9% result: is this a property of LLMs, or of the task when measured with an execution oracle? Adding at least one traditional/static tool per language would calibrate the difficulty of the benchmark and would directly support the claim that current LLMs are the bottleneck.
  5. [§6 and Tables 3, 4, 7] All results are single-run point estimates with no confidence intervals, significance tests, or variance information. Given that several reported differences are small (e.g., Python executability 17.3 vs. 22.4 for DeepSeek-Coder vs. Qwen in Table 4; Rust 1.0 vs. 2.0 in Table 4), sampling or nondeterminism could change model or method rankings. The paper should report bootstrap confidence intervals or repeat runs for a subset of configurations, or otherwise justify that the reported gaps are meaningful.
minor comments (5)
  1. [Appendix B.2] In the last paragraph, 'ike Rust' should be 'like Rust'.
  2. [Table 6 caption] The caption contains the typo 'exeutability rate'; it should be 'executability rate'.
  3. [Throughout] The benchmark name appears as 'DI-B ENCH' with an unwanted space in many places; please normalize it to 'DI-BENCH'.
  4. [Section 2] The reference 'damnever, 2024' is a GitHub tool page rather than an archival publication; if it is kept, it should be formatted as a software/tool reference with a URL and access date.
  5. [Section 4.2] The pipeline is described as 'fully automated,' but the Test Job Locating step is LLM-assisted; the text should reconcile these statements, for example by saying 'fully automated with an LLM-based selector.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DI-BENCH's ground truth and execution oracle are external to model outputs; CI reuse for curation and evaluation is a design choice, not a logical reduction.

full rationale

The paper's construction and evaluation chain is self-contained and non-circular. Ground-truth dependencies are taken from the repositories' original build configuration files; the task masks dependency sections and asks the model to reconstruct them. Textual metrics compare generated dependencies against those original configurations, and the execution metric runs each repository's own CI test job under act on the model-completed configuration. No parameter is fitted to the evaluation data, no prediction is defined in terms of its own outcome, and no load-bearing claim rests on a self-citation chain. The reuse of the same CI pipeline for dataset curation and for final evaluation is a design choice rather than a circular reduction: a repository must pass its own tests before masking, and after masking the same tests serve as an external, repository-authored oracle for whether the inferred dependencies are sufficient. The paper's acknowledged weaknesses, such as the unvalidated LLM-assisted test-job locating step, act's incomplete emulation of GitHub Actions, and the Limitations item conceding that test coverage may not be exhaustive, are empirical validity risks for the 42.9% executability claim, not circularity in the derivation. Because the central claim is an empirical benchmark result evaluated against externally grounded tests, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

DI-BENCH is an empirical benchmark; it introduces no fitted constants, no new physical entities, and no derivation. The assumptions listed are the load-bearing choices about dataset representativeness, CI fidelity, test-suite sufficiency, and task inferability.

assumptions (4)
  • domain assumption Selected repositories with >100 stars, GitHub Actions enabled, size under 10MB, and passing CI validation represent the real-world population of interest.
    Dataset curation criteria in Section 4.2 restrict the sample; generalizing DI-BENCH results to all repositories relies on this assumption.
  • domain assumption The LLM-located test job executed with act faithfully reproduces the repository's actual CI environment and test outcome.
    Section 4.2 'Execution Validating' uses act as a local GitHub Actions runner; any divergence between act and real CI would corrupt both curation and executability evaluation.
  • domain assumption Passing the repository's own test suite is a sufficient binary oracle for correct dependency inference.
    The Executability Rate in Section 5 assigns 1 only if all tests pass; incomplete test coverage, acknowledged in the Limitations section, could let wrong dependencies pass.
  • domain assumption The source code and masked build files contain enough information to recover the ground-truth dependency list.
    The task definition in Section 3 assumes dependencies are inferable from code and configuration structure; otherwise the benchmark is underspecified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale." pith.science (2026). https://pith.science/paper/S547DYWH

@misc{pith2026250113699,
  author       = {Pith},
  title        = {Pith review of: DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S547DYWH}},
  note         = {Machine review of arXiv:2501.13699}
}
read the original abstract

Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40\% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs' capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 42.9% execution pass rate, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.

Figures

Figures reproduced from arXiv: 2501.13699 by the authors.

Figure 1
Figure 1. An example of Python project dependencies. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of incorrectly identifying dependencies used in code. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. CI-based curation pipeline for DI-BENCH. ing as a quality filter criterion. 3) Repository size is less than 10MB to avoid extremely large repositories and maintain a manageable dataset size. 4) Most importantly, the repository must have GitHub Actions enabled, indicated by the presence of the .github/workflows folder. Repositories that meet these criteria proceed as candidate repos￾itories into subsequent phases. Te… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Distribution of failure categories (GPT-4o, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Execution pass rate w.r.t dependency count. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Model performance across programming lan [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Execution pass rate w.r.t repository size [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: An example of pyproject.toml in Python [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 11
Figure 11. Figure 11: An example of example.csproj in C# [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 10
Figure 10. Figure 10: An example of Cargo.toml in Rust C# ( [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 13
Figure 13. Figure 13: Prompt template used to generate build file. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Prompt used to merge build file edits [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner

    cs.CL 2025-06 conditional novelty 7.0 of 10

    SWE-Flow synthesizes incremental, test-driven development tasks from real GitHub projects and shows that fine-tuning Qwen2.5-Coder-32B-Instruct on them improves performance on the resulting SWE-Flow-Bench benchmark.

  2. SWE-bench Goes Live!

    cs.SE 2025-05 conditional novelty 7.0 of 10

    SWE-bench-Live provides a live, automatically curated, Docker-backed benchmark of 1,319 fresh GitHub issue-fixing tasks, on which leading agents score around 19%, well below their SWE-bench Verified results.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  4. [4]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . Preprint, arXiv:2308.14508

    arXiv 2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  6. [6]

    crates.io . 2024. The Rust community’s crate registry . https://crates.io/

    work page 2024

  7. [7]

    damnever . 2024. A tool to generate requirements.txt for Python project, and more than that . https://github.com/damnever/pigar

    work page 2024

  8. [8]

    Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. https://arxiv.org/abs/2310.11248 Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion . Preprint, arXiv:2310.11248

    arXiv 2023

Show all 33 references
  1. [9]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861

    2023 arXiv

  2. [10]

    GitHub . 2023. GitHub Copilot -- Your AI pair programmer . https://github.com/features/copilot

    2023

  3. [11]

    GitHub . 2024. GitHub Actions: Automate your workflow from idea to production . https://github.com/features/actions

    2024

  4. [12]

    Abhishek Kadian

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. Abhishek Kadian. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  5. [13]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  6. [14]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. 2024. https://openreview.net/forum?id=VtmBAGCN7o Meta GP...

    2024

  7. [15]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654

    2024 arXiv

  8. [16]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  9. [17]

    Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Repository-level compositional code translation and validation. arXiv preprint arXiv:2410.24117

    2024 arXiv

  10. [18]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770

    2023 arXiv

  11. [19]

    Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. 2024 a . https://arxiv.org/abs/2403.08604 Devbench: A comprehensive benchmark for software devel...

    2024 arXiv

  12. [20]

    Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024 b . Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00599

    2024 arXiv

  13. [21]

    Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091

    2023 arXiv

  14. [22]

    nektos . 2024. act: Run your GitHub Actions locally . https://nektosact.com/

    2024

  15. [23]

    OpenAI . 2024 a . GPT-4o mini: advancing cost-efficient intelligence . https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  16. [24]

    OpenAI . 2024 b . Hello GPT-4o . https://openai.com/index/hello-gpt-4o/

    2024

  17. [25]

    PyPI . 2024. Find, install and publish Python packages with the Python Package Index . https://https://pypi.org/

    2024

  18. [26]

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. https://arxiv.org/abs/2307.07924 Chatdev: Communicative agents for software development . Preprint, arXiv:...

    2024 arXiv

  19. [27]

    tree-sitter . 2024. Tree-sitter . https://tree-sitter.github.io/tree-sitter/

    2024

  20. [28]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brenna...

    2024 arXiv

  21. [29]

    Qinyun Wu, Chao Peng, Pengfei Gao, Ruida Hu, Haoyu Gan, Bo Jiang, Jinhe Tang, Zhiwen Deng, Zhanming Guan, Cuiyun Gao, et al. 2024. Repomastereval: Evaluating code completion via real-world repositories. arXiv preprint arXiv:2408.03519

    2024

  22. [30]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. 2024. https://arxiv.org/abs/2410.03859 Swe-bench multimodal: Do ai systems general...

    2024 arXiv

  23. [31]

    Hongjie Ye, Wei Chen, Wensheng Dou, Guoquan Wu, and Jun Wei. 2022. https://doi.org/10.1145/3510003.3510127 Knowledge-based environment dependency inference for python programs . In 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), pages 1245--1256

    2022

  24. [32]

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570

    2023 arXiv

  25. [33]

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. 2024. https://arxiv.org/abs/2410.10934 Agent-as-a-judge: Evalua...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.