Pith. sign in

REVIEW 2 major objections 6 minor 56 references

BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new benchmark for governing interacting pull-request queues shows that current coding agents recover relations but rarely deliver a safe whole-queue plan.

desk verdict A carefully built benchmark for queue-level PR governance with a real novelty and honest evaluation; the main risks are the unreleased artifact and the unproven cross-component independence behind the exact oracle. read the letter →

arxiv 2608.02685 v1 pith:HTFE5DAV submitted 2026-08-03 cs.SE cs.AI

Zetong Xiong , Qiao Zhao , Jun Zhang , Xueying Lyu , Zhi Li , Yixiang Tu , Xiaowen Yang , Yunjie Zhang
show 15 more authors
Yufeng Wang Zhe Zhang Kaize Yu Hanwen Du Zhongkai Sun Zhuoxin Liu Zekun Lin Jianwen Yang Ruining Chen Ying Zhang Tingxuan Pan Ke Chen Shubin Han Chuanhao Sun Yehua Yang
This is my paper
classification cs.SEcs.AI
keywords codingagentspullrequestqueuemergeplanningrelationalgroundtruthsafesubsetexecutableorderbenchmarkconstructionsoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that managing a queue of interacting pull requests is a distinct task: an agent must recover which PRs conflict, depend on, duplicate, or supersede one another, choose a subset to merge, and order that subset so every committed state passes hidden safety checks. To make this measurable, the authors introduce BulkPR-Bench, comprising 581 newly authored candidate PRs on frozen snapshots of 18 real repositories, with a gold relation graph validated by actual repository execution of every registered state. An exact oracle computes the largest safe subset of each pool, and scoring follows the realized merge trace rather than the agent's stated plan. Under a rolling-release protocol, the best models beat deterministic sequential baselines on the relation-group metric RDS, but only 8 of 324 model runs complete an entire queue exactly, and 69.8% of runs end with an unsafe merge. The paper's central point is that progress on relation groups does not yet translate into dependable whole-queue governance.

What carries the argument

The load-bearing mechanism is the executable relational construction: every candidate PR is authored against the same pinned base commit, symbolic predictions about registered states are checked against real repository execution including construction-time hidden verifiers, and the gold relation graph is accepted only when predictor and repository behavior agree on all registered states. An exact maximum-safe-subset oracle then computes OPT_N, and the metrics RDS and Global-SGY are computed from the realized merge trace, not from the agent's claimed plan.

What would settle it

Run a full 2^N enumeration over one 33-PR pool using the frozen gold constraints and compare the resulting maximum safe subset with the published OPT_N; any mismatch would falsify the exact-oracle claim.

Watch

Extended reading notes

Core claim

The central claim is that queue-level pull-request governance can be benchmarked with execution-validated relational ground truth, and that current coding agents are effective at recovering and respecting individual relations but poor at converting that into a safe whole-queue plan. At the primary buffered protocol with batch size K=32, the three highest model RDS estimates are 66.6%, 62.0%, and 57.9%, compared with 53.1% for the strongest sequential baseline, while Global Safety-Gated Yield stays at or below 18.3% and only 8 of 324 model runs achieve Exact Completion.

Load-bearing premise

The gold relation graph is assumed to capture every consequential interaction among candidates, even though only registered states are executed — all nonempty subsets within each relation component plus whole-pool witnesses and selected cross-component probes — not all 2^N global subsets.

Editorial extensions

If this is right

  • If the ground truth is accepted, the reported RDS and Global-SGY numbers become meaningful measurements of an agent's queue-level governance capability.
  • Because models respect 98.1% to 100.0% of the relations they explicitly recover, the main bottleneck is incomplete relation recovery, not acting on recovered relations.
  • Gold-fed diagnostics reaching 77.5% to 99.1% RDS show that even with perfect relation information, selection and ordering leave substantial headroom.
  • With only 8 of 324 exact completions, the benchmark establishes that dependable whole-queue governance remains an open problem for current agents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exactness of the oracle depends on the registered-state coverage; an unregistered cross-component combination that violates a hidden relation would invalidate the gold graph, a risk the paper itself notes by not enumerating all 2^N subsets.
  • The RDS metric weights all relation groups equally, so a model could accumulate high RDS by correctly rejecting zero-optimum groups, which may overstate practical delivery if such refusals are easier than positive safe merges.
  • A natural extension would be to use the frozen pools to test whether larger buffers, alternative ledger schemas, or different arrival orders change relation recovery more than model scale does.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. BulkPR-Bench is an executable benchmark for queue-level pull-request governance. Each instance pins a frozen repository snapshot with N=32-33 newly authored candidate PRs released in batches of size K; an agent must recover consequential PR relations, select a subset of candidates, and produce an executable merge order under a rolling-release protocol with a bounded buffer. Ground truth is a gold relation graph (pairwise and higher-order conflicts, dependencies, all-or-none groups, must-rejects, duplicates, supersedes) validated by registered state-by-state repository execution with hidden verifiers, together with an exact oracle that computes the largest safe subset (OPT_N). The primary metric, Relational Delivery Score (RDS), scores safe delivery and correct rejection per relation group on the realized merge trace; Global-SGY, Exact Completion, CriticalRecall, and gold-fed/solver diagnostics measure whole-queue strictness and relation recovery. Across 18 repositories at K=32, the top RDS estimates are 66.6% (Claude-Opus-4.8), 62.0% (GPT-5.4), and 57.9% (GLM-5.2), versus 53.1% for the strongest sequential baseline, while only 8/324 model runs achieve Exact Completion and CriticalRecall ranges from 35.2% to 57.7%.

Significance. The benchmark targets a real gap: existing coding-agent benchmarks fix a single outcome or a prescribed change chain, whereas BulkPR-Bench requires joint subset selection, relation recovery, and ordering under bounded visibility. The construction protocol is a genuine strength: registered-state agreement between constraint prediction and repository execution, production-bound hidden verifiers with negative controls, cache-bypassed flakiness probes, a triple-recomputed oracle, drift checks, and versioned release plans are appropriate to the claims. The evaluation is honestly hedged: repository-level BCa bootstrap, explicit non-representativeness disclaimers, and diagnostic arms (gold-fed, solver) that are clearly labeled as non-causal. The headline separation between relation-group gains (RDS) and whole-queue strictness (8/324 Exact Completions) is a useful, falsifiable finding. If the gold-graph completeness concern in Major Comment 1 is resolved, this is a credible and reusable measurement instrument for agentic code integration.

major comments (2)
  1. [Section 5.2; Section 4 (Eqs. 1-4)] The exactness of OPT_N, OPT_c, and every downstream metric rests on the assumption that registered-state validation captures all consequential interactions among candidates, but the registered evidence covers only nonempty subsets within each relation component plus whole-pool witnesses and selected cross-component probes, and the text explicitly declines to enumerate all 2^N global subsets. If two PRs from different relation components interact only in an unregistered combination, the missing gold atom would (a) inflate OPT_N and OPT_c, deflating Global-SGY (Eq. 3) and Exact Completion (Eq. 4); (b) let an actually unsafe realized plan be scored as safe, inflating RDS group scores (Eq. 1); and (c) mislabel a PR as relation-free. Appendix A.3 re-runs the registered states and recomputes the oracle but does not expand the registered set, so the abstract's 'exact oracle' claim overstates what is established (Section 5.2 properly says 'exact with respect to the frozen gold constraints'). I request, at minimum: (i) per-repository counts of cross-component probes, especially all cross-component pairs, and any mismatches found; (ii) the number of candidate pools rejected or rebuilt before the agreement rule was satisfied; and (iii) either a completeness audit that samples unregistered cross-component states (all pairs plus a random sample of triples or larger subsets), or a file-disjointness/independence argument for cross-component candidates, or an explicit rescoping of the 'exact' claims in the abstract and introduction together with a statement of the bias direction on each metric.
  2. [Section 5.2 and Figure 2] The manuscript never specifies what the 'constraint predictor' is or how it was produced relative to the gold graph. If the predictor is the same hand-authored relation design that defines the gold atoms, then predictor-execution agreement on registered states functions mainly as a bookkeeping check, and the real evidential load falls on the hidden verifiers for the declared relations; if the predictor is an independent static or LLM-based analysis, the agreement test carries more weight. Please specify the predictor and its independence from the gold graph. Relatedly, define what 'whole-pool witnesses' means: the union of all N candidates, or a maximum-safe witness set derived from the gold graph? A single witnessed combination cannot certify OPT_N for all alternative combinations, so the definition determines how much the witness check can contribute to the completeness argument.
minor comments (6)
  1. [Section 7.1 and Table 3] The top three model RDS point estimates exceed the best baseline, but the repository-level 95% intervals are wide (e.g., GLM-5.2 [51.3, 64.8] contains the 53.1% baseline, and Claude-Opus-4.8 [60.4, 73.2] is consistent with a much smaller edge); since significance is explicitly not claimed, consider reporting per-repository paired differences between each model and CI-Fixedpoint so the separation can be interpreted.
  2. [Section 5.3] The stated release plan is future-tense ('We will release...'); for a benchmark whose credibility depends on frozen trial records and hidden verifiers, the editor and reviewers should be given access to the artifact during revision, even under a reviewer-only tag.
  3. [Table 3] The typesetting interleaves point estimates and their bootstrap intervals as separate lines, which makes the column correspondence hard to parse; please format each estimate and interval as a single cell or otherwise align them unambiguously.
  4. [Figure 3] The K<32 points are described as exploratory three-run means without clustered intervals; since the text already discloses this, consider adding a brief note in the caption so the lack of intervals on those points is visible without reading Section 6.1.
  5. [Section 4.3] 'Agentuses its own ledger' appears to be missing a space ('Agent uses'); please proofread the camera-ready version.
  6. [Figure 2 caption] 'File-disjoint from public diffs' is ambiguous between 'touching only files outside all public diffs' and 'touching files of the relation-target diffs only'; please define the required disjointness precisely.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the oracle and metrics are defined independently of model outputs, and the gold graph is anchored to repository execution rather than to the scores it produces.

full rationale

The paper's derivation chain is benchmark construction -> registered repository execution -> gold relation graph -> exact solver -> metrics -> agent scores. Each step is anchored outside the model outputs. The gold relation graph is accepted only when a constraint predictor and repository execution agree on registered states (Section 5.2), and hidden verifiers are bound to concrete production behavior with negative controls; this is a code-executed external anchor, not a fitted parameter or an imported self-citation. The oracle is computed by an exact solver with respect to the frozen gold constraints, and the metrics (RDS Eq. 2, Global-SGY Eq. 3, Exact Completion Eq. 4) are defined directly from realized traces and that oracle without conditioning on model results. The buffer size B=4,T=16 was selected from an offline clairvoyant grid computed from pool structure alone, explicitly without model runs (Appendix B.3), so the primary protocol is not tuned to model outcomes. The only noted gap is that registered evidence does not enumerate all 2^N global subsets (Section 5.2), which is an acknowledged completeness limitation for the gold graph as repository truth, not a circular reduction: the paper repeatedly scopes claims to the frozen gold constraints ('Passing means satisfying pinned tests, hidden verifiers, and the registered full gold, not production correctness', Section 8). No self-citations are load-bearing; no parameter fitted to target scores is renamed as a prediction. The benchmark's self-containment against external repository execution justifies a score of 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric constants are fitted to a target result; the benchmark's scoring is defined independently of the models. The strongest assumptions are domain assumptions about gold-graph validity and hidden-verifier fidelity, both located in Section 5.2.

assumptions (3)
  • domain assumption A relation graph inferred from execution agreement on all subsets within each relation component, whole-pool witnesses, and selected cross-component probes is valid for all unenumerated subsets.
    Section 5.2 states that registered evidence does not enumerate all 2^N global subsets. All safety labels, OPT_N, and downstream metrics depend on this generalization.
  • domain assumption Construction-time hidden verifiers bound to concrete production behavior correctly identify real-world safety failures and are not merely the authors' intended semantics.
    Section 5.2 describes hidden verifiers that pass on base and negative controls and fail on target states, but no independent third-party validation is described.
  • standard math Standard combinatorial and statistical tools are used correctly: the exact maximum over the gold graph and BCa bootstrap clustered by 18 repositories.
    Sections 5.2 and 6.3 rely on the exact solver and the clustered bootstrap; these are not central but are required for the reported optima and intervals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests." pith.science (2026). https://pith.science/paper/HTFE5DAV

@misc{pith2026260802685,
  author       = {Pith},
  title        = {Pith review of: BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HTFE5DAV}},
  note         = {Machine review of arXiv:2608.02685}
}
abstract

Coding-agent benchmarks increasingly cover long-horizon, end-to-end, and interactive development, but typically retain one requested outcome or a fixed change sequence. Sequential policies can process a pull-request (PR) queue one candidate at a time, but when queued PRs interact, maximizing safe delivery can require jointly deciding which changes to merge and in what order. We introduce BulkPR-Bench, an executable benchmark in which an agent must recover consequential PR relations and return a large safe subset in executable order under a rolling-release protocol. The suite contains 581 newly authored candidate PRs on frozen snapshots of 18 real repositories. Registered state-by-state repository execution, including hidden safety checks, validates the gold relation graph; an exact oracle then computes the largest safe subset. Our primary metric, Relational Delivery Score (RDS), scores safe delivery and correct rejection over relation groups from the realized merge trace; Global Safety-Gated Yield (Global-SGY) separately measures strict delivery of the realized whole-queue plan. Under the buffered primary protocol with batch size $K=32$, the three highest RDS estimates among the six models are 66.6%, 62.0%, and 57.9%, compared with 53.1% for the strongest sequential baseline. Only 8 of 324 model runs complete a queue exactly. Critical-relation recall ranges from 35.2% to 57.7%, and diagnostic runs supplied with the gold relations show substantial remaining headroom. Gains on relation groups therefore do not yet translate into dependable whole-queue governance.

Figures

Figures reproduced from arXiv: 2608.02685 by the authors.

Figure 1
Figure 1. In this episode, a greedy no-deferral policy admits a hidden-conflicting pair and rejects the all-or-none group; a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Executable relational construction. Historical PRs and expert input guide new candidate designs; candidates are [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. RDS versus batch size 𝐾. Points average three runs per eligible repository (18 for RDS; 17 for RDS (hidden)). Columns compare buffered and no-deferral; rows use differ￾ent scales. The four highest-RDS models are shown; [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Primary buffered 𝐾=32 diagnostics for the four highest-RDS models. (a) Agent RDS (three runs/cell), its gap to gold-fed RDS (one run/cell), and the remaining gap to the full-gold solver; the gaps are diagnostic, not causal. (b) Mutually exclusive outcomes (54 runs/mode…
Figure 5
Figure 5. Figure 5: Critical-relation recovery remains incomplete. Crit [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 12 canonical work pages

  1. [2]

    Ali Arabat and Mohammed Sayagh. 2024. An Empirical Study on Cross- Component Dependent Changes: A Case Study on the Components of OpenStack. Empirical Software Engineering29, 5 (2024), 109. doi:10.1007/s10664-024-10488-y

    doi:10.1007/s10664-024-10488-y 2024

  2. [3]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, Outcomes, and Chal- lenges of Modern Code Review. InProceedings of the 35th International Conference on Software Engineering. IEEE, 712–721. doi:10.1109/ICSE.2013.6606617

    arXiv 2013

  3. [4]

    Sebastian Baltes and Paul Ralph. 2022. Sampling in Software Engineering Re- search: A Critical Review and Guidelines.Empirical Software Engineering27, 4 (2022), 94. doi:10.1007/s10664-021-10072-8

    doi:10.1007/s10664-021-10072-8 2022

  4. [5]

    Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. InProceedings of the 14th International Conference on Mining Software Repositories. IEEE, 356–367. doi:10.1109/MSR.2017.62

    doi:10.1109/msr.2017.62 2017

  5. [6]

    Bender and Batya Friedman

    Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Lan- guage Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics6 (2018), 587–604. doi:10.1162/tacl_a_00041

    doi:10.1162/tacl_a_00041 2018

  6. [7]

    Ernst, and David Notkin

    Yuriy Brun, Reid Holmes, Michael D. Ernst, and David Notkin. 2011. Proactive Detection of Collaboration Conflicts. InProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engi- neering. ACM, 168–178. doi:10.1145/2025113.2025139

    arXiv 2011

  7. [8]

    Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Pran- nay Hebbar, Jiankai Sun, Yiyuan Li, Pramod Srinivasan, Ishan Gupta, Christo- pher Settles, Daniel Wang, Derek Chen, Pranav Raja, Albert Liu, Marek Šuppa, Nevasini Sasikumar, Luyang Kong, Erik Quintanilla, Xiangyi Li...

    arXiv 2026

  8. [9]

    Elizabeth Dinella, Todd Mytkowicz, Alexey Svyatkovskiy, Christian Bird, Mayur Naik, and Shuvendu K. Lahiri. 2023. DeepMerge: Learning to Merge Programs. IEEE Transactions on Software Engineering49, 4 (2023), 1599–1614. doi:10.1109/ TSE.2022.3183955

    arXiv 2023

Show all 56 references
  1. [10]

    Bradley Efron. 1987. Better Bootstrap Confidence Intervals.J. Amer. Statist. Assoc. 82, 397 (1987), 171–185. doi:10.1080/01621459.1987.10478410

    1987

  2. [11]

    C. A. Field and A. H. Welsh. 2007. Bootstrapping Clustered Data.Journal of the Royal Statistical Society: Series B (Statistical Methodology)69, 3 (2007), 369–390. doi:10.1111/j.1467-9868.2007.00593.x

    2007

  3. [12]

    Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, III, and Kate Crawford. 2021. Datasheets for Datasets. Commun. ACM64, 12 (2021), 86–92. doi:10.1145/3458723

    2021 doi

  4. [13]

    Gleiph Ghiotto, Leonardo Murta, Marcio Barros, and André van der Hoek. 2020. On the Nature of Merge Conflicts: A Study of 2,731 Open Source Java Projects Hosted by GitHub.IEEE Transactions on Software Engineering46, 8 (2020), 892–

    2020

  5. [14]

    Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An Exploratory Study of the Pull-Based Software Development Model. InProceedings of the 36th International Conference on Software Engineering. ACM, 345–355. doi:10.1145/ 2568225.2568260

    2014

  6. [15]

    Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, and Arie van Deursen

  7. [16]

    Hao Guan, Lingyue Fu, Shao Zhang, Yaoming Zhu, Kangning Zhang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang, and Yong Yu. 2026. SWE- Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle. arXiv:2605.13139 [cs.SE] https://arxiv.org/abs/2605.13139

    2026 arXiv

  8. [17]

    Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig

  9. [18]

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. 2026. DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks. arXiv:2607.07946 [cs.SE] https://arxiv.org/abs/2607.07946

    2026 arXiv

  10. [19]

    Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Con- tamination by Evaluation Benchmarks. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processin...

    2023 doi

  11. [20]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tian- jun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. InThe Thirteenth International Conference o...

    2025 arXiv

  12. [21]

    Tao Ji, Liqian Chen, Xiaoguang Mao, Xin Yi, and Jiahong Jiang. 2022. Automated Regression Unit Test Generation for Program Merges.Science China Information Sciences65, 9 (2022), 199103. doi:10.1007/s11432-019-3020-4

    2022 doi

  13. [22]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InInternational Conference on Learning Rep- resentations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  14. [23]

    Qirui Jin, Lingching Tung, Kenan Li, Qiyang Shi, Yushi She, Huanzhong Jia, Harrison Zhao, Kejing Xia, Zhenbang Du, Yikai Zhang, Jiaxin Pei, Zhenyu Zhang, Zhen Qi, Yuyan Duan, Wenke Lee, and Zijian Jin. 2026. ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenanc...

    2026 arXiv

  15. [24]

    Dhruva Juloori, Zhongpeng Lin, Matthew Williams, Eddy Shin, and Sonal Ma- hajan. 2025. CI at Scale: Lean, Green, and Fast. InProceedings of the IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice. IEEE, 437–447. doi:10.1109/ICSE-SEIP6635...

    2025

  16. [25]

    Maximilian Jungwirth, Martin Gruber, and Gordon Fraser. 2025. Improving Merge Pipeline Throughput in Continuous Integration via Pull Request Prioritization. InProceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE, 565–575. doi:10.1109/ICS...

    2025

  17. [26]

    Siegel, Nitya Nadgir, and Arvind Narayanan

    Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2025. AI Agents That Matter.Transactions on Machine Learning Research(2025). arXiv:2407.01502 https://openreview.net/forum?id=Zy4uFzMviZ

    2025 arXiv

  18. [27]

    Brendan King and Jeffrey Flanigan. 2026. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents. arXiv:2606.13995 [cs.CL] https://arxiv.org/ abs/2606.13995

    2026

  19. [28]

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan

  20. [29]

    Zhixing Li, Yue Yu, Minghui Zhou, Tao Wang, Gang Yin, Long Lan, and Huaimin Wang. 2022. Redundancy, Context, and Preference: An Empirical Study of Dupli- cate Pull Requests in OSS Projects.IEEE Transactions on Software Engineering48, 4 (2022), 1309–1335. doi:10.1109/TSE.2020.3018726

    2022

  21. [30]

    Tobias Lindenbauer, Egor Bogomolov, and Yaroslav Zharov. 2025. GitGoodBench: A Novel Benchmark for Evaluating Agentic Performance on Git. arXiv:2505.22583 https://arxiv.org/abs/2505.22583

    2025 arXiv

  22. [31]

    Chandra Shekhar Maddila, Nachiappan Nagappan, Christian Bird, Georgios Gousios, and Arie van Deursen. 2022. ConE: A Concurrent Edit Detection Tool for Large-Scale Software Development.ACM Transactions on Software Engineering and Methodology31, 2 (2022), 1–26. doi:10.1145/3478019

    2022 doi

  23. [32]

    Shane McIntosh, Yasutaka Kamei, Bram Adams, and Ahmed E. Hassan. 2014. The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects. InProceedings of the 11th Working Conference on Mining Software Repositor...

    2014

  24. [33]

    Tom Mens. 2002. A State-of-the-Art Survey on Software Merging.IEEE Transac- tions on Software Engineering28, 5 (2002), 449–462. doi:10.1109/TSE.2002.1000449

    2002 arXiv

  25. [35]

    Agustín Olmedo, Gabriela Arévalo, Ignacio Cassol, Christelle Urtado, and Sylvain Vauttier. 2022. Improving Integration Process Efficiency through Pull Request Prioritization. InProceedings of the 17th International Conference on Evaluation of Novel Approaches to Software Engin...

    2022

  26. [36]

    Perry, Harvey P

    Dewayne E. Perry, Harvey P. Siy, and Lawrence G. Votta. 1998. Parallel Changes in Large-Scale Software Development: An Observational Case Study. InProceedings of the 20th International Conference on Software Engineering. IEEE, 251–260. doi:10. 1109/ICSE.1998.671134

    1998

  27. [37]

    Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. InPro- ceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, 1776–...

    2022

  28. [38]

    Rigby and Christian Bird

    Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. InProceedings of the 9th Joint Meeting on Foundations of Software Engineering. ACM, 202–212. doi:10.1145/2491411.2491444

    2013

  29. [39]

    Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP Evaluation in Trouble: On the Need to KDD ’27, August 1–5, 2027, San Jose, CA, USA Xiong et al. Measure LLM Data Contamination for Each Benchmark. InFindings of the ...

    2023 doi

  30. [40]

    KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara, Pranoy Panda, and Chaitanya Devaguptapu. 2026. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution. arXiv:2604.03035 https://arxiv.org/ abs/2604.03035

    2026 arXiv

  31. [41]

    Jonathan Steinberg and Oren Gal. 2026. MOSAIC-Bench: Measuring Compo- sitional Vulnerability Induction in Coding Agents. arXiv:2605.03952 https: //arxiv.org/abs/2605.03952

    2026 arXiv

  32. [42]

    Jason Tsay, Laura Dabbish, and James Herbsleb. 2014. Let’s Talk About It: Eval- uating Contributions through Discussion in GitHub. InProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 144–154. doi:10.1145/2635868.2635882

    2014

  33. [43]

    Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. 2021. Towards Automating Code Review Activities. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 163–174. arXiv:2101.02518 doi:10.1109/ICSE43902.2021.00027

    2021 arXiv

  34. [44]

    Gustavo Vale, Claus Hunsen, Eduardo Figueiredo, and Sven Apel. 2022. Chal- lenges of Resolving Merge Conflicts: A Mining and Survey Study.IEEE Trans- actions on Software Engineering48, 12 (2022), 4964–4985. doi:10.1109/TSE.2021. 3130098

    2022 doi

  35. [45]

    Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integra- tion in GitHub. InProceedings of the 10th Joint Meeting on Foundations of Software Engineering. ACM, 805–816. doi:10.1145/278...

    2015

  36. [46]

    Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, and Haoli Bai. 2026. SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review. arXiv:2607.06065 https://arxiv.org/abs/2607.06065

    2026 arXiv

  37. [47]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2025. Demystifying LLM-Based Software Engineering Agents.Proceedings of the ACM on Software Engineering2, FSE (June 2025), 801–824. arXiv:2407.01489 doi:10.1145/3715754

    2025 arXiv

  38. [48]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R. Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. InAdvances in Neural In- formation Processing Systems, Vol. 37. Curran Associates...

    2024 arXiv

  39. [49]

    Yue Yu, Huaimin Wang, Vladimir Filkov, Premkumar Devanbu, and Bogdan Vasilescu. 2015. Wait for It: Determinants of Pull Request Evaluation Latency on GitHub. InProceedings of the 12th Working Conference on Mining Software Repositories. IEEE, 367–371. doi:10.1109/MSR.2015.42

    2015 doi

  40. [50]

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Shulin Xin, Linhao Zhang, Qi Liu, Aoyan Li, Lu Chen, Xiaojian Zhong, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Ming Ding, and Liang Xiang. 2025. Multi-SWE-bench: A Multilingual Bench...

    2025 arXiv

  41. [51]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. 2025. SWE-bench Goes Live! arXiv:2505.23419 [cs.SE] https://arxiv.or...

    2025 arXiv

  42. [52]

    Xin Zhang, Yang Chen, Yongfeng Gu, Weiqin Zou, Xiaoyuan Xie, Xiangyang Jia, and Jifeng Xuan. 2018. How Do Multiple Pull Requests Change the Same Code: A Study of Competing Pull Requests in GitHub. InProceedings of the IEEE International Conference on Software Maintenance and E...

    2018

  43. [53]

    Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, and Abhik Roychoudhury. 2026. Code Review Agent Benchmark. arXiv:2603.23448 https://arxiv.org/abs/2603.23448

    2026 arXiv

  44. [54]

    Qixing Zhou, Jiacheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxi- ang Zhang. 2026. FeatureBench: Benchmarking Agentic Coding for Complex Feature Development. InThe Fourteenth International Conferenc...

    2026

  45. [915]

    doi:10.1109/TSE.2018.2871083

    2018

  46. [2015]

    InProceedings of the 37th International Conference on Software Engineering

    Work Practices and Challenges in Pull-Based Development: The Integra- tor’s Perspective. InProceedings of the 37th International Conference on Software Engineering. IEEE, 358–368. doi:10.1109/ICSE.2015.55

    2015 doi

  47. [2016]

    InProceedings of the 31st IEEE/ACM International Conference on Auto- mated Software Engineering

    Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects. InProceedings of the 31st IEEE/ACM International Conference on Auto- mated Software Engineering. ACM, 426–437. doi:10.1145/2970276.2970358

  48. [2022]

    InProceed- ings of the 30th ACM Joint European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering (ESEC/FSE ’22)

    Automating Code Review Activities by Large-Scale Pre-training. InProceed- ings of the 30th ACM Joint European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering (ESEC/FSE ’22). Association for Computing Machinery, 1035–1047. arXiv:2203.0...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.