BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies

Brian Goh; Camellia Lok; Constance Tan; David Lo; Eng Lieh Ouh; Ferdian Thung; Fiona Wee; Haodi Qi; Hong Jin Kang; Jack Phan

BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.15481 v1 pith:62KN22W7 submitted 2024-01-27 cs.SE

BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies

Ratnadira Widyasari , Sheng Qin Sim , Camellia Lok , Haodi Qi , Jack Phan , Qijin Tay , Constance Tan , Fiona Wee

show 8 more authors

Jodie Ethelda Tan Yuheng Yieh Brian Goh Ferdian Thung Hong Jin Kang Thong Hoang David Lo Eng Lieh Ouh

This is my paper

classification cs.SE

keywords pythondebuggingtestingjavatoolsbenchmarkprogramsbugs

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers. One factor that contributed to the rapid growth of Java testing and debugging tools is the availability of benchmarks. A popular benchmark is the Defects4J benchmark; its initial version contained 357 real bugs from 5 real-world Java programs. Each bug comes with a test suite that can expose the bug. Defects4J has been used by hundreds of testing and debugging studies and has helped to push the frontier of research in these directions. In this project, inspired by Defects4J, we create another benchmark database and tool that contain 493 real bugs from 17 real-world Python programs. We hope our benchmark can help catalyze future work on testing and debugging tools that work on Python programs.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Hybrid Fault-Driven Mutation Testing for Python
cs.SE 2026-01 conditional novelty 6.0

Seven Python anti-pattern mutation operators, applied via hybrid static/dynamic analysis in the PyTation tool, produce mutants that complement Cosmic Ray and reveal test-suite gaps in 13 projects.