Pith. sign in

REVIEW 3 major objections 4 minor 72 references

Exploring the Jupyter Ecosystem: An Empirical Study of Bugs and Vulnerabilities

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The most common bugs in Jupyter notebooks fall into twelve root-cause categories led by incorrect configuration, data-shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors, while Jupyter deployment frameworks…

desk verdict Useful qualitative taxonomy of Jupyter notebook bugs, but the frequency ordering is unsupported by the sampling design and RQ1 has an internal contradiction. read the letter →

arxiv 2507.18833 v1 pith:BZTPLI6G submitted 2025-07-24 cs.SE

Wenyuan Jiang , Diany Pressato , Harsh Darji , Thibaud Lutellier This is my paper · ORCID
classification cs.SE
keywords JupyterNotebooksempiricalsoftwareengineeringbugtaxonomygroundedtheoryvulnerabilitiesdatascienceconfigurationerrorsAPImisuse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish, with data, what actually goes wrong in Jupyter notebooks and in the software that runs them. It argues that notebook bugs have a recognizable source-code-level signature: twelve root-cause categories, with incorrect configuration, data-shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors the most common, and that these categories appear consistently across GitHub and Kaggle notebooks. It also argues that the characteristics most associated with bug-fixing commits are social rather than technical—number of contributors and file-modification frequency outweigh code complexity and documentation metrics—and that Jupyter deployment frameworks, not notebook documents themselves, carry the security risk, through web-style vulnerabilities like CSRF and access-control gaps. If correct, the study gives tool builders and teams a concrete checklist of what to check first and shifts attention from notebook content to configuration, collaboration, and deployment hygiene.

What carries the argument

The paper's central mechanism is a grounded-theory coding pipeline for notebook bug fixes, where categories are built by iterative reading and coding rather than imposed in advance. Because notebooks are JSON documents whose line diffs are polluted by metadata and outputs, the authors serialize each notebook's Python code cells, parse them into abstract syntax trees (ASTs), and diff the trees; PCA on eight change metrics and DBSCAN clustering then produced eleven sampling clusters with a silhouette score of 0.91, from which three co-authors independently coded sampled fixes, discussed, and resampled until no new categories emerged after 230 bugs. For security, a regex filter followed by an LLM (DeepSeek-V3) mapped commit messages to CWE Top 25 entries, with manual validation reducing 323 candidates to 62. The taxonomy itself is the deliverable that carries the central claim.

What would settle it

Take a random sample of 100 commits the heuristic labels as bug-related from the 376 repositories, have two independent annotators classify each as a genuine bug fix, and check whether excluding the false positives changes the ordering of the top root-cause categories; if it does, the claim that configuration and API misuse are the most common notebook bugs needs revision.

Watch

Extended reading notes

Core claim

Based on 8,647 notebooks from 376 active GitHub repositories and 1,038 Kaggle competition notebooks, the paper derives a taxonomy of twelve bug root causes in Jupyter notebooks: incorrect configuration, data shape/structure mistakes, API misuse, incorrect syntax, wrong logic, non-determinism, exception/error/log/debugging issues, errors in test code and assertions, resource management, incomplete code, undeclared variables and typos, and documentation mistakes. The most common categories, per the authors' manual coding of 230 bug fixes, are incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors; 89 tracebacks in the Kaggle logs independently show configuration errors, compiler errors, API misuse, and logic errors. On the quantitative side, the number of contributors and the frequency of file modification correlate most strongly with bug-related commits, while cyclomatic complexity and code-to-markdown ratio are weak or negligible. On security, the authors report no security issues inside notebook documents themselves, but 62 validated security-related commits in JupyterHub, Jupyter Server, and JupyterLab repositories, mapped to CWE categories, are dominated by protection-mechanism failure (CWE-693), improper access control (CWE-284), and improper resource-lifecycle control (CWE-664), with CSRF the most recurrent single issue and a visible tension between usability and security in some fixes.

Load-bearing premise

The whole study leans on a keyword rule that calls a commit bug-related if its message contains 'fix,' 'bug,' or 'patch' and none of 'rename,' 'merge,' 'clean-up,' or 'refactor'; if that labeling is noisy for notebooks, the correlations and the taxonomy sample inherit the noise.

Editorial extensions

If this is right

  • Bug-finding and linting tools for notebooks should target configuration strings, data-shape operations, and API call sites first, since those categories dominate the taxonomy.
  • Teams adopting notebooks should treat collaboration metrics as risk signals: files touched by many contributors or modified frequently are where bug-fixing commits concentrate, so code review and change management matter more than code length in notebooks.
  • Configuration management practices such as pinning library versions, keeping environment files, and using containers would address the single largest root-cause category identified in the study.
  • Organizations exposing JupyterHub, Jupyter Server, or JupyterLab beyond a local machine should harden authentication, TLS, cross-origin policies, and dependency updates, because framework vulnerabilities are predominantly web-based.
  • The absence of security findings inside notebook documents, combined with the framework findings, shifts security attention from notebook content to the deployment layer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the commit-based method can only see bugs that were noticed and fixed; silent wrong-output bugs are likely underrepresented, so 'wrong logic' may be rarer in the ranking than in reality.
  • Beyond the paper, the strong contributor correlation may partly reflect popularity rather than collaboration: popular notebooks attract both more editors and more fixes, and normalizing by file age or views would test that alternative explanation.
  • Beyond the paper, the security analysis covers only three GitHub organizations; the reported doubling of Jupyter-related CVEs between 2023 and 2024 suggests the broader plugin and server ecosystem deserves an equivalent dependency-level audit.
  • Beyond the paper, the same taxonomy could be tested against execution traces from CI pipelines or enterprise notebook platforms to see whether configuration and API misuse still dominate outside GitHub and Kaggle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports an empirical study of bugs and vulnerabilities in the Jupyter Notebook ecosystem. The authors collect 8,647 notebooks from 376 active GitHub repositories and 1,038 Kaggle competition notebooks, extract AST-level changes, and identify 9,554 bug-related commits using a keyword heuristic (Section III-B). For RQ1, they compute Pearson correlations between notebook characteristics (complexity, natural language, contributor metrics) and bug-related commit counts. For RQ2, they follow grounded theory, manually analyze 230 bug-fixing changes, and derive a 12-category taxonomy of root causes, claiming that incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors are the most common. For RQ3, they analyze security-related commits in JupyterHub, JupyterLab, and Jupyter Server repositories, manually validating a commit-to-CWE mapping and identifying CSRF, access control, and resource management as dominant vulnerability categories.

Significance. The qualitative taxonomy is a potentially useful contribution: it is derived from manual inspection of 230 real fixes and complements existing higher-level taxonomies of notebook problems by focusing on source-code-level root causes. The security analysis is also valuable because the commit-to-CWE mapping was manually validated, and the finding that CSRF, improper access control, and resource management issues dominate is concrete and actionable. The paper ships a replication package and provides falsifiable claims about the notebook ecosystem. The main weakness is that the 'most common' ranking of root causes in RQ2 is not supported by the reported sampling design, which limits the quantitative force of the taxonomy claim.

major comments (3)
  1. [Section III-C and RQ2 Summary (Section V)] The claim that 'incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors' are the most common root causes is not supported by the methodology. The axial coding stage samples 30 or 50 bugs per cluster based on availability, which is a purposive, diversity-oriented design that does not preserve the relative frequencies of clusters or bug types; no per-category counts are reported for the 230 manually analyzed bugs. The Kaggle evidence covers only 89 notebooks with tracebacks, mapped to four categories (59 compiler errors, 20 configuration errors, 5 logic errors, 5 API misuses), so it does not include five of the six categories listed as most common. The authors should report frequencies from a random sample or rephrase the RQ2 summary and abstract to avoid claiming a ranking that the data cannot support.
  2. [Section III-B] The identification of bug-related commits relies on a keyword heuristic ('fix', 'bug', 'patch' with exclusions) adapted from general software defect prediction work (references [28], [66]) and is not validated on Jupyter Notebooks. Because this heuristic defines the dependent variable for RQ1 and the sampling pool for RQ2, false positives (e.g., 'fix typo', 'fix formatting') and false negatives (e.g., 'correct the calculation') can bias both the correlation results and the taxonomy. The authors should validate the heuristic on a random sample of commits with manual labels and report precision and recall, or use a more notebook-specific bug detection approach.
  3. [Section IV, Figure 3] The Pearson correlation analysis reports coefficients without p-values, confidence intervals, or a definition of the color thresholds for 'weak', 'medium', and 'strong' correlations. The conclusion that 'the number of contributors metric is strongly correlated' is based on a single coefficient (r = 0.41) with no significance test; count-based metrics such as number of contributors and file modification frequency are likely non-normal, so the suitability of Pearson should be justified. The authors should add significance tests or bootstrap confidence intervals to support the RQ1 claims.
minor comments (4)
  1. [Figure 3] The mapping between the printed coefficient values and the metric labels is ambiguous in the text; the figure should clearly associate each coefficient with its metric, since the reader must infer, for example, that 0.34 corresponds to libraries imported and 0.41 to number of contributors.
  2. [Section III-C and Section V] The paper reports 11 clusters for qualitative analysis but then presents a taxonomy of 12 categories; the authors should clarify how the 11 clusters map to the 12 categories.
  3. [Section III-D] The security commit pipeline reduces 400 regex-filtered commits to 323 after LLM filtering, then to 66 after removing automated scans, then to 62 after manual annotation; the paper should report how many of the 66 were removed by manual annotation and the agreement between annotators.
  4. [Abstract] The abstract states that 'configuration issues are among the most common bugs... followed by incorrect API usage', which is broader than the RQ2 summary; the abstract should be aligned with the actual support provided by the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the taxonomy, correlations, and CWE analysis are empirical outputs of manual coding and independent corpora, not derived by construction from their own inputs.

full rationale

This paper is an empirical study with no fitted parameters, closed-form derivations, or predicted quantities that reduce to its inputs. The bug-related commit labeling uses a keyword heuristic 'following previous work [28], [66]' (Section III-B); although one of those citations, [28], includes a co-author, the heuristic is a standard preprocessing filter also attributed to independent work [66], and neither the RQ1 correlations nor the RQ2 taxonomy are computed from that heuristic in a way that forces the conclusions. The RQ2 bug taxonomy was produced by manual grounded-theory coding of 230 commits plus Kaggle tracebacks; the categories are not defined in terms of the PCA/DBSCAN clustering inputs, and the Kaggle analysis is an independent secondary source. The concern that the 'most common root causes' ordering in the RQ2 Summary is not backed by per-category counts from the diversity-oriented sampling design is a validity and evidence issue, not circularity, because the paper never claims the ordering is computed from cluster sizes or from the sampling allocation. The RQ3 CWE mapping likewise comes from manually checked regex/LLM filtering of security commits in deployment repositories, and the final tally of 62 commits is an output, not an input, of the analysis. No self-definitional step, fitted-input-called-prediction step, or citation-forced conclusion is present, so the derivation chain is self-contained.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several hand-chosen thresholds and domain assumptions about dataset representativeness and bug identification; no new theoretical entities or fitted constants are introduced.

free parameters (5)
  • GitHub activity cutoff = 1 year
    Projects considered active if they had any GitHub event within the last year as of February 2024 (Section III-A). Hand-chosen threshold for dataset inclusion.
  • Commit message length cutoff = 1,000 characters
    Commits with messages longer than 1,000 characters excluded to avoid automation-generated commits (Section III-D). Hand-chosen.
  • Cluster count = 11 (8 DBSCAN + 3 manual)
    DBSCAN yielded 8 clusters; 3 additional clusters added for non-code edits, unparsable changes, and commits lacking bug-fixing keywords (Section III-C). This structure shapes the manual sampling for the taxonomy.
  • Keyword sets for bug-related commits = include {'fix','bug','patch'}, exclude {'rename','merge','clean-up','refactor'}
    Heuristic from prior work [28,66] used to label bug-related commits (Section III-B); precision and recall not validated on notebooks.
  • Kaggle error keyword set = Traceback, Error
    Logs parsed for these keywords to identify bugs in Kaggle notebooks (Section III-A).
assumptions (5)
  • domain assumption The keyword-based heuristic for identifying bug-related commits correctly labels commits that fix bugs in Jupyter notebooks.
    Used in Section III-B to build the bug-related commit dataset; if inaccurate, RQ1 correlations and RQ2 sampling are biased.
  • domain assumption AST-based differencing of concatenated Python source cells captures the meaningful code changes in notebooks; changes in outputs and metadata are ignored.
    Section III-A; notebooks store outputs and metadata in JSON, and non-code changes are discarded. Semantic changes in code cells are assumed to be captured by AST comparison.
  • domain assumption Manual grounded theory coding reached saturation after 230 bugs sampled in three iterations.
    Section III-C; the taxonomy's completeness rests on this claim, which is not quantitatively demonstrated.
  • domain assumption The LLM (DeepSeek-V3) commit-to-CWE mapping, after manual validation, correctly classifies security-related commits.
    Section III-D; 62 commits are classified into CWE pillars based on LLM output plus manual annotation; LLM errors are mitigated manually but not independently benchmarked.
  • domain assumption GitHub top-starred active repositories and top Kaggle competition notebooks are representative of the Jupyter notebook ecosystem.
    Section III-A and acknowledged in Threats to Validity; selection may introduce bias toward popular, high-quality projects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Jupyter Ecosystem: An Empirical Study of Bugs and Vulnerabilities." pith.science (2026). https://pith.science/paper/BZTPLI6G

@misc{pith2026250718833,
  author       = {Pith},
  title        = {Pith review of: Exploring the Jupyter Ecosystem: An Empirical Study of Bugs and Vulnerabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZTPLI6G}},
  note         = {Machine review of arXiv:2507.18833}
}
read the original abstract

Background. Jupyter notebooks are one of the main tools used by data scientists. Notebooks include features (configuration scripts, markdown, images, etc.) that make them challenging to analyze compared to traditional software. As a result, existing software engineering models, tools, and studies do not capture the uniqueness of Notebook's behavior. Aims. This paper aims to provide a large-scale empirical study of bugs and vulnerabilities in the Notebook ecosystem. Method. We collected and analyzed a large dataset of Notebooks from two major platforms. Our methodology involved quantitative analyses of notebook characteristics (such as complexity metrics, contributor activity, and documentation) to identify factors correlated with bugs. Additionally, we conducted a qualitative study using grounded theory to categorize notebook bugs, resulting in a comprehensive bug taxonomy. Finally, we analyzed security-related commits and vulnerability reports to assess risks associated with Notebook deployment frameworks. Results. Our findings highlight that configuration issues are among the most common bugs in notebook documents, followed by incorrect API usage. Finally, we explore common vulnerabilities associated with popular deployment frameworks to better understand risks associated with Notebook development. Conclusions. This work highlights that notebooks are less well-supported than traditional software, resulting in more complex code, misconfiguration, and poor maintenance.

Figures

Figures reproduced from arXiv: 2507.18833 by the authors.

Figure 1
Figure 1. A Notebook where the first cell may be skipped. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. First, we mine active Jupyter Notebook repositories [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 2
Figure 2. Overview of the data collection and processing pipelines for each research question. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Pearson Correlation coefficients between the number of bug-related [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 67 canonical work pages

  1. [28]

    In: Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pp

    Lutellier, T., Pham, H.V ., Pang, L., Li, Y ., Wei, M., Tan, L.: Coconut: combining context-aware neural translation models using ensemble for program repair. In: Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pp. 101–114 (2020)

    work page 2020

  2. [66]

    In: Proceedings of the 38th international conference on software engineering, pp

    Wang, S., Liu, T., Tan, L.: Automatically learning semantic features for defect prediction. In: Proceedings of the 38th international conference on software engineering, pp. 297–308 (2016)

    work page 2016

  3. [1]

    Journal of Computing Sciences in Colleges 39(5), 96–108 (2023)

    Adams, K., Vilkomir, A., Hills, M.: A comparison of machine learning code quality in python scripts and jupyter notebooks. Journal of Computing Sciences in Colleges 39(5), 96–108 (2023)

    work page 2023

  4. [2]

    Characterizing Bugs in Python and R Data Analytics Programs

    Ahmed, S., Wardat, M., Bagheri, H., Cruz, B.D., Rajan, H.: Charac- terizing bugs in python and r data analytics programs. arXiv preprint arXiv:2306.08632 (2023)

    work page Pith review arXiv 2023

  5. [3]

    IEEE Transactions on Software Engineering 47(6), 1277–1298 (2019)

    Bao, L., Xia, X., Lo, D., Murphy, G.C.: A Large Scale Study of Long- time Contributor Prediction for Github Projects. IEEE Transactions on Software Engineering 47(6), 1277–1298 (2019)

    work page 2019

  6. [4]

    Journal of Systems and Software 152, 165–181 (2019)

    Catolino, G., Palomba, F., Zaidman, A., Ferrucci, F.: Not all bugs are the same: Understanding, characterizing, and classifying bug types. Journal of Systems and Software 152, 165–181 (2019)

    work page 2019

  7. [5]

    https://github.com/cgpotts/cs224 u/commit/d582e8d057543fb4972c642007db104a15914e07 (2021)

    cgpotts: cs224u, commit d582e8d. https://github.com/cgpotts/cs224 u/commit/d582e8d057543fb4972c642007db104a15914e07 (2021). Accessed: July 2025

    work page 2021

  8. [6]

    In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp

    Chattopadhyay, S., Prasad, I., Henley, A.Z., Sarma, A., Barik, T.: What’s wrong with computational notebooks? pain points, needs, and design opportunities. In: Proceedings of the 2020 CHI conference on human factors in computing systems, pp. 1–12 (2020)

    work page 2020

Show all 72 references
  1. [7]

    In: IEEE 17th International Conference on eScience, pp

    Cunha, R.L., Real, L.C.V ., Souza, R., Silva, B., Netto, M.A.: Context- aware execution migration tool for data science jupyter notebooks on hybrid clouds. In: IEEE 17th International Conference on eScience, pp. 30–39. IEEE (2021)

    2021

  2. [8]

    https://github.com/fastai/fastai2/comm it/9cecf8192a232176a9532b785de22bcc50a09865 (2020)

    cwza: fastai2, commit 9cecf81. https://github.com/fastai/fastai2/comm it/9cecf8192a232176a9532b785de22bcc50a09865 (2020). Accessed: July 2025

    2020

  3. [9]

    URL https://arxiv.org/abs/2412.19437

    DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xi...

    2024 arXiv

  4. [10]

    https://github.com/goo gle-deepmind/deepmind-research/commit/8cc5c3966282fc677b032e8c6 7f1b89458e6d47a (2021)

    Ding, D.: deepmind-research, commit 8cc5c39. https://github.com/goo gle-deepmind/deepmind-research/commit/8cc5c3966282fc677b032e8c6 7f1b89458e6d47a (2021). Accessed: July 2025

    2021

  5. [11]

    https: //github.com/google-deepmind/deepmind-research/blob/8cc5c396628 2fc677b032e8c67f1b89458e6d47a/perceiver/colabs/optical flow.ipynb (2021)

    Ding, D., diegolascasas: Deepmind research, optical flow.ipynb. https: //github.com/google-deepmind/deepmind-research/blob/8cc5c396628 2fc677b032e8c67f1b89458e6d47a/perceiver/colabs/optical flow.ipynb (2021). Accessed: July 2025

    2021

  6. [12]

    In: 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), pp

    Dong, H., Zhou, S., Guo, J.L., K ¨astner, C.: Splitting, renaming, re- moving: A study of common cleaning activities in jupyter notebooks. In: 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), pp. 114–119. IEEE (2021)

    2021

  7. [13]

    Information and Software Technology 138, 106616 (2021)

    Eluri, V .K., Mazzuchi, T.A., Sarkani, S.: Predicting Long-time Contrib- utors for Github Projects Using Machine Learning. Information and Software Technology 138, 106616 (2021)

    2021

  8. [15]

    In: Proceedings of the 19th international conference on mining software repositories, pp

    Grotov, K., Titov, S., Sotnikov, V ., Golubev, Y ., Bryksin, T.: A large- scale comparison of python code in jupyter notebooks and scripts. In: Proceedings of the 19th international conference on mining software repositories, pp. 353–364 (2022)

    2022

  9. [16]

    Software Testing, Verification And Reliability 31(4), e1751 (2021)

    Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Besz ´edes, ´A., Ferenc, R., Mesbah, A.: Bugsjs: a benchmark and taxonomy of javascript bugs. Software Testing, Verification And Reliability 31(4), e1751 (2021)

    2021

  10. [17]

    In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp

    Hanam, Q., Brito, F.S.d.M., Mesbah, A.: Discovering bug patterns in javascript. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp. 144–156 (2016)

    2016

  11. [18]

    Journal of Software: Evolution and Process 35(2), e2507 (2023)

    Hu, M., Zhang, Y .: An empirical study of the python/c API on evolution and bug patterns. Journal of Software: Evolution and Process 35(2), e2507 (2023). DOI 10.1002/smr.2507. URL https://onlinelibrary.wiley. com/doi/10.1002/smr.2507

    2023 doi

  12. [19]

    In: Pro- ceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp

    Humbatova, N., Jahangirova, G., Bavota, G., Riccio, V ., Stocco, A., Tonella, P.: Taxonomy of real faults in deep learning systems. In: Pro- ceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1110–1121. ACM (2020). DOI 10.1145/3377811.33 80395....

    2020

  13. [20]

    Journal of Systems and Software 85(10), 2293–2304 (2012)

    Ibrahim, W.M., Bettenburg, N., Adams, B., Hassan, A.E.: On the relationship between comment update practices and software bugs. Journal of Systems and Software 85(10), 2293–2304 (2012)

    2012

  14. [21]

    URL https: //www.jetbrains.com/lp/devecosystem-2022/data-science/

    JetBrains: The State of Developer Ecosystem 2022 (2022). URL https: //www.jetbrains.com/lp/devecosystem-2022/data-science/

    2022

  15. [22]

    In: International Conference on Database Systems for Advanced Applications, pp

    Jia, L., Zhong, H., Wang, X., Huang, L., Lu, X.: An empirical study on bugs inside tensorflow. In: International Conference on Database Systems for Advanced Applications, pp. 604–620. Springer (2020)

    2020

  16. [23]

    In: 23rd Asia-Pacific Software Engineering Conference, pp

    Jimenez, M., Papadakis, M., Le Traon, Y .: An empirical analysis of vulnerabilities in openssl and the linux kernel. In: 23rd Asia-Pacific Software Engineering Conference, pp. 105–112. IEEE (2016)

    2016

  17. [24]

    The Art, Science, and Engineering of Programming 5(3) (2021)

    K ¨all´en, M., Sigvardsson, U., Wrigstad, T.: Jupyter notebooks on github: characteristics and code clones. The Art, Science, and Engineering of Programming 5(3) (2021)

    2021

  18. [25]

    In: IEEE/ACM 18th International Conference on Mining Software Repositories, pp

    Kamienski, A.V ., Palechor, L., Bezemer, C.P., Hindle, A.: Pysstubs: Characterizing single-statement bugs in popular open-source python projects. In: IEEE/ACM 18th International Conference on Mining Software Repositories, pp. 520–524. IEEE (2021)

    2021

  19. [26]

    In: Proceedings of the fifth international conference on the foundations of digital games, pp

    Lewis, C., Whitehead, J., Wardrip-Fruin, N.: What went wrong: a taxonomy of video game bugs. In: Proceedings of the fifth international conference on the foundations of digital games, pp. 108–115 (2010)

    2010

  20. [27]

    https://github.com/h uggingface/notebooks/commit/589f8aa27f57562bab4f3427183a4be77a 888459 (2022)

    lewtun: Huggingface notebooks, commit 589f8aa. https://github.com/h uggingface/notebooks/commit/589f8aa27f57562bab4f3427183a4be77a 888459 (2022). Accessed: July 2025

    2022

  21. [29]

    IEEE Transactions on Software Engineering (2023)

    Macklon, F., Viggiato, M., Romanova, N., Buzon, C., Paas, D., Bezemer, C.P.: A taxonomy of testable html5 canvas issues. IEEE Transactions on Software Engineering (2023)

    2023

  22. [30]

    In: IEEE/ACM 43rd International Conference on Software Engineering, pp

    Makhshari, A., Mesbah, A.: Iot bugs and development challenges. In: IEEE/ACM 43rd International Conference on Software Engineering, pp. 460–472. IEEE (2021)

    2021

  23. [31]

    In: IEEE Symposium on Visual Languages and Human-Centric Computing, pp

    Merino, M.V ., van Binsbergen, L.T., Seraj, M.: Making the invisible visible in computational notebooks. In: IEEE Symposium on Visual Languages and Human-Centric Computing, pp. 1–3. IEEE (2022)

    2022

  24. [32]

    https://github.com/jupyterhub/jupy terhub/commit/43a6cd0bf90c8496ed5c47eaadd2803bbb87e0b6 (2017)

    minrk: Jupyterhub, commit 43a6cd0. https://github.com/jupyterhub/jupy terhub/commit/43a6cd0bf90c8496ed5c47eaadd2803bbb87e0b6 (2017). Accessed: July 2025

    2017

  25. [33]

    https: //cwe.mitre.org/top25/archive/2024/2024 cwe top25.html (2024)

    Mitre: 2024 cwe top 25 most dangerous software weaknesses. https: //cwe.mitre.org/top25/archive/2024/2024 cwe top25.html (2024). Accessed: July 2025

    2024

  26. [34]

    Computer Fraud & Security 2022(12) (2022)

    Morag, A.: The Threats to Jupyter Notebook. Computer Fraud & Security 2022(12) (2022)

    2022

  27. [35]

    Proceedings of the 44th International Conference on Software Engineering pp

    Nahar, N., Zhou, S., Lewis, G., K ¨astner, C.: Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engi- neering, and Process. Proceedings of the 44th International Conference on Software Engineering pp. 413–425 (2022)

    2022

  28. [36]

    In: Proceedings of the 44th International Conference on Software Engineering, pp

    Patra, J., Pradel, M.: Nalin: learning from runtime behavior to find name-value inconsistencies in jupyter notebooks. In: Proceedings of the 44th International Conference on Software Engineering, pp. 1469–1481 (2022)

    2022

  29. [37]

    In: IEEE/ACM 16th international conference on mining software repositories, pp

    Pimentel, J.F., Murta, L., Braganholo, V ., Freire, J.: A large-scale study about quality and reproducibility of jupyter notebooks. In: IEEE/ACM 16th international conference on mining software repositories, pp. 507–

  30. [38]

    Empirical Software Engineering 26(4), 1–55 (2021)

    Pimentel, J.F., Murta, L., Braganholo, V ., Freire, J.: Understanding and Improving the Quality and Reproducibility of Jupyter Notebooks. Empirical Software Engineering 26(4), 1–55 (2021)

    2021

  31. [39]

    In: IEEE/ACM 18th International Conference on Mining Software Repositories, pp

    Quaranta, L., Calefato, F., Lanubile, F.: Kgtorrent: A dataset of python jupyter notebooks from kaggle. In: IEEE/ACM 18th International Conference on Mining Software Repositories, pp. 550–554. IEEE (2021)

    2021

  32. [40]

    Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–41 (2022)

    Quaranta, L., Calefato, F., Lanubile, F.: Eliciting Best Practices for Collaboration with Computational Notebooks. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–41 (2022)

    2022

  33. [41]

    In: Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI, CAIN ’22

    Quaranta, L., Calefato, F., Lanubile, F.: Pynblint: a static analyzer for python jupyter notebooks. In: Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI, CAIN ’22. ACM (2022). DOI 10.1145/3522664.3528612. URL http: //dx.doi.org/10....

    2022

  34. [42]

    In: IEEE/ACM 42nd International Conference on Software Engineering, pp

    Rahman, A., Farhana, E., Parnin, C., Williams, L.: Gang of eight: A defect taxonomy for infrastructure as code scripts. In: IEEE/ACM 42nd International Conference on Software Engineering, pp. 752–764. IEEE (2020)

    2020

  35. [43]

    In: 29th Asia-Pacific Software Engineering Conference, pp

    Ritta, N., Settewong, T., Kula, R.G., Ragkhitwetsagul, C., Sunetnanta, T., Matsumoto, K.: Reusing my own code: Preliminary results for competitive coding in jupyter notebooks. In: 29th Asia-Pacific Software Engineering Conference, pp. 457–461. IEEE (2022)

    2022

  36. [44]

    In: 36th IEEE/ACM International Conference on Automated Software Engineering, pp

    Romano, A., Liu, X., Kwon, Y ., Wang, W.: An empirical study of bugs in webassembly compilers. In: 36th IEEE/ACM International Conference on Automated Software Engineering, pp. 42–54. IEEE (2021)

    2021

  37. [45]

    Journal of computational and applied mathematics 20, 53–65 (1987)

    Rousseeuw, P.J.: Silhouettes: a Graphical Aid to the Interpretation and Validation of Cluster Analysis. Journal of computational and applied mathematics 20, 53–65 (1987)

    1987

  38. [46]

    ACM Trans

    de Santana, T.L., Neto, P.A.d.M.S., de Almeida, E.S., Ahmed, I.: Bug analysis in jupyter notebook projects: An empirical study. ACM Trans. Softw. Eng. Methodol. (2024). DOI 10.1145/3641539. URL https: //doi.org/10.1145/3641539. Just Accepted

    2024 doi

  39. [47]

    In: 29th Asia-Pacific Software Engineering Conference, pp

    Settewong, T., Ritta, N., Kula, R.G., Ragkhitwetsagul, C., Sunetnanta, T., Matsumoto, K.: Why Visualize Data When Coding? Preliminary Categories for Coding in Jupyter Notebooks. In: 29th Asia-Pacific Software Engineering Conference, pp. 462–466. IEEE (2022)

    2022

  40. [48]

    https://github.com/fastai/fastai2/co mmit/8215467935e431415f5c854e876db7af59bf0e37 (2020)

    sgugger: fastai2, commit 8215467. https://github.com/fastai/fastai2/co mmit/8215467935e431415f5c854e876db7af59bf0e37 (2020). Accessed: July 2025

    2020

  41. [49]

    https://github.com/fastai/fastai 2/commit/8ab3e702bf248a7c2545b64d05d60a9e60325672 (2020)

    sgugger: fastai2, commit 8ab3e70. https://github.com/fastai/fastai 2/commit/8ab3e702bf248a7c2545b64d05d60a9e60325672 (2020). Accessed: July 2025

    2020

  42. [50]

    In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp

    Shen, Q., Ma, H., Chen, J., Tian, Y ., Cheung, S.C., Chen, X.: A com- prehensive study of deep learning compiler bugs. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 968– 9...

    2021

  43. [51]

    In: Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, pp

    Suboti ´c, P., Miliki´c, L., Stoji ´c, M.: A static analysis framework for data science notebooks. In: Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, pp. 13–22 (2022)

    2022

  44. [52]

    Frontiers of Computer Science 15(6), 1–16 (2021)

    Sun, X., Zhou, T., Wang, R., Duan, Y ., Bo, L., Chang, J.: Experience report: investigating bug fixes in machine learning frameworks/libraries. Frontiers of Computer Science 15(6), 1–16 (2021)

    2021

  45. [53]

    In: The art and science of analyzing software data, pp

    Tan, L.: Code comment analysis for improving software quality. In: The art and science of analyzing software data, pp. 493–517. Elsevier (2015)

    2015

  46. [54]

    Empirical software engineering 19(6), 1665– 1705 (2014)

    Tan, L., Liu, C., Li, Z., Wang, X., Zhou, Y ., Zhai, C.: Bug characteristics in open source software. Empirical software engineering 19(6), 1665– 1705 (2014)

    2014

  47. [55]

    In: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, pp

    Tan, L., Yuan, D., Krishna, G., Zhou, Y .: /* icomment: Bugs or bad comments?*. In: Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, pp. 145–158 (2007)

    2007

  48. [56]

    In: IEEE Fifth International Conference on Software Testing, Verification and Validation, pp

    Tan, S.H., Marinov, D., Tan, L., Leavens, G.T.: @ tcomment: Testing javadoc comments to detect comment-code inconsistencies. In: IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 260–269. IEEE (2012)

    2012

  49. [57]

    https://github.com/tugst ugi/dl-colab-notebooks/commit/67caaeb668aa6b596b4ad3df2acf15228b 5ba0be (2020)

    tugstugi: dl-colab-notebooks, commit 67caaeb. https://github.com/tugst ugi/dl-colab-notebooks/commit/67caaeb668aa6b596b4ad3df2acf15228b 5ba0be (2020). Accessed: July 2025

    2020

  50. [58]

    In: IEEE international conference on software maintenance and evolution, pp

    Vahabzadeh, A., Fard, A.M., Mesbah, A.: An empirical study of bugs in test code. In: IEEE international conference on software maintenance and evolution, pp. 101–110. IEEE (2015)

    2015

  51. [59]

    Journal of Systems and Software 143, 44–58 (2018)

    Valdivia-Garcia, H., Shihab, E., Nagappan, M.: Characterizing and predicting blocking bugs in open source projects. Journal of Systems and Software 143, 44–58 (2018)

    2018

  52. [60]

    In: Proceedings of the 2020 ACM SIGPLAN International Sym- posium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp

    Van Binsbergen, L.T., Verano Merino, M., Jeanjean, P., Van Der Storm, T., Combemale, B., Barais, O.: A principled approach to repl inter- preters. In: Proceedings of the 2020 ACM SIGPLAN International Sym- posium on New Ideas, New Paradigms, and Reflections on Programming and ...

    2020

  53. [61]

    In: IEEE/ACM 14th International Conference on Mining Software Repositories, pp

    Wan, Z., Lo, D., Xia, X., Cai, L.: Bug characteristics in blockchain systems: a large-scale empirical study. In: IEEE/ACM 14th International Conference on Mining Software Repositories, pp. 413–424. IEEE (2017)

    2017

  54. [62]

    In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering, pp

    Wang, D., Li, S., Xiao, G., Liu, Y ., Sui, Y .: An exploratory study of autopilot software bugs in unmanned aerial vehicles. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Con- ference and Symposium on the Foundations of Software Engineering, pp...

    2021

  55. [63]

    In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp

    Wang, J., Kuo, T.y., Li, L., Zeller, A.: Assessing and Restoring Repro- ducibility of Jupyter Notebooks. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 138– 149 (2020)

    2020

  56. [64]

    In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, pp

    Wang, J., Li, L., Zeller, A.: Better Code, Better Sharing: on the Need of Analyzing Jupyter Notebooks. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, pp. 53–56 (2020)

    2020

  57. [65]

    Empirical Software Engineering 27(1), 1–35 (2022)

    Wang, P., Brown, C., Jennings, J.A., Stolee, K.T.: Demystifying regular expression bugs. Empirical Software Engineering 27(1), 1–35 (2022)

    2022

  58. [67]

    Journal of Software: Evolution and Process 33(10), e2376 (2021)

    Wei, Y ., Sun, X., Bo, L., Cao, S., Xia, X., Li, B.: A comprehensive study on security bug characteristics. Journal of Software: Evolution and Process 33(10), e2376 (2021)

    2021

  59. [68]

    https://github.com/x 4nth055/pythoncode-tutorials/commit/351ee9548a4a2981f39f557127f9 9fd0ac11f506 (2020)

    x4nth055: pythoncode-tutorials, commit 351ee95. https://github.com/x 4nth055/pythoncode-tutorials/commit/351ee9548a4a2981f39f557127f9 9fd0ac11f506 (2020). Accessed: July 2025

    2020

  60. [69]

    In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp

    Zhai, J., Xu, X., Shi, Y ., Tao, G., Pan, M., Ma, S., Xu, L., Zhang, W., Tan, L., Zhang, X.: Cpc: Automatically classifying and propagating natural language comments via program analysis. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp...

    2020

  61. [70]

    Information and software technology 86, 37–53 (2017)

    Zhao, Y ., Leung, H., Yang, Y ., Zhou, Y ., Xu, B.: Towards an under- standing of change types in bug fixing code. Information and software technology 86, 37–53 (2017)

    2017

  62. [71]

    In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp

    Zhou, X., Han, D., Lo, D.: Simple or complex? together for a more accurate just-in-time defect predictor. In: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pp. 229–240 (2022)

    2022

  63. [72]

    In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp

    Zhou, Y ., Sharma, A.: Automated identification of security issues from commit messages and bug reports. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp. 914–919 (2017)

    2017

  64. [73]

    In: Proceedings of the IEEE/ACM 36th International Conference on Automated Software Engineering, pp

    Zhu, C., Saha, R.K., Prasad, M.R., Khurshid, S.: Restoring the Exe- cutability of Jupyter Notebooks by Automatic Upgrade of Deprecated APIs. In: Proceedings of the IEEE/ACM 36th International Conference on Automated Software Engineering, pp. 240–252. IEEE (2021)

    2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.