{"id":"5e3cf790-3d57-4fee-88b4-b3cc5e787c28","arxiv_id":"2507.18833","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The most common Jupyter notebook bugs are configuration mistakes and API misuse, and notebook infrastructure suffers mainly from web-based vulnerabilities like CSRF and XSS.","lead":"Researchers analyzed over 9,000 Jupyter notebooks from GitHub and Kaggle to find what makes notebooks buggy. They built a taxonomy of 12 common bug types, led by configuration mistakes and API misuse, and mapped security weaknesses in notebook deployment frameworks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RQ2 'most common root causes' ordering is not supported by the diversity-oriented sampling design; no per-category counts are reported.","rationale":"I focused on RQ2 because the reader's strongest_claim is the taxonomy. The taxonomy categories themselves are plausible, and the paper provides a replication package plus manual validation, so I do not object to the qualitative contribution. However, the frequency claim is load-bearing and unsupported by the reported method. The reader's weakest assumption (the keyword heuristic for identifying bug-related commits) is real, but it is upstream and affects RQ1 most directly; even if that heuristic were perfect, the RQ2 sampling design would still not support the \"most common\" ordering. This is a correctness risk in the central claim, not merely a presentation issue. Requiring per-category counts and a proportional re-sample is a concrete, feasible condition that would resolve the concern. The existing CONDITIONAL verdict remains appropriate, with the condition now explicitly requiring frequency evidence or a softened claim.","tokens_in":18636,"tokens_out":5310,"duration_ms":55915,"concrete_test":"Using the replication package, extract the 230 manually analyzed samples, record the final root-cause label for each, and report the full count distribution. Check whether the six categories named in the RQ2 Summary are the six largest counts and whether the counts can distinguish the claimed ordering (e.g., via bootstrap confidence intervals or a simple proportion test). Independently, re-run the qualitative coding on a simple random sample of 230 commits drawn from the 9,554 bug-related commits, without PCA/DBSCAN stratification, and compare the resulting category proportions; if the top categories change materially, the \"most common\" claim is an artifact of the sampling design rather than a property of the underlying bug population.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central RQ2 claim that \"incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors\" are the most common root causes requires frequency information from the 230 manually analyzed bugs. The sampling protocol in Section III-C does not provide it. After PCA/DBSCAN clustering, the authors state that the co-authors \"each randomly sample 30 or 50 bugs from different categories (based on availability)\" and iterate three times until saturation. This is a purposive, diversity-oriented sample: the clusters were created to cover distinct scenarios, and the per-author allocation is not proportional to cluster sizes or to the 9,554 bug-related commits. No per-category counts are reported for the 230 bugs, so the ordering in the RQ2 Summary cannot be checked. The Kaggle evidence does not fill the gap: only 89 of 1,038 notebooks had tracebacks, they were mapped to only four categories, and compiler errors (59) dominate; the Kaggle data do not include the six claimed top categories. The keyword heuristic for bug-related commits is a further source of uncertainty, but even with perfect labeling, the stratified sampling design invalidates frequency claims. The taxonomy may still be a useful qualitative contribution, but the quantitative \"most common\" assertion is load-bearing and currently unsupported by the reported method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of bugs and vulnerabilities in the Jupyter Notebook ecosystem. The authors collect 8,647 notebooks from 376 active GitHub repositories and 1,038 Kaggle competition notebooks, extract AST-level changes, and identify 9,554 bug-related commits using a keyword heuristic (Section III-B). For RQ1, they compute Pearson correlations between notebook characteristics (complexity, natural language, contributor metrics) and bug-related commit counts. For RQ2, they follow grounded theory, manually analyze 230 bug-fixing changes, and derive a 12-category taxonomy of root causes, claiming that incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors are the most common. For RQ3, they analyze security-related commits in JupyterHub, JupyterLab, and Jupyter Server repositories, manually validating a commit-to-CWE mapping and identifying CSRF, access control, and resource management as dominant vulnerability categories.","tokens_in":18889,"tokens_out":9896,"duration_ms":85142,"significance":"The qualitative taxonomy is a potentially useful contribution: it is derived from manual inspection of 230 real fixes and complements existing higher-level taxonomies of notebook problems by focusing on source-code-level root causes. The security analysis is also valuable because the commit-to-CWE mapping was manually validated, and the finding that CSRF, improper access control, and resource management issues dominate is concrete and actionable. The paper ships a replication package and provides falsifiable claims about the notebook ecosystem. The main weakness is that the 'most common' ranking of root causes in RQ2 is not supported by the reported sampling design, which limits the quantitative force of the taxonomy claim.","major_comments":[{"comment":"The claim that 'incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors' are the most common root causes is not supported by the methodology. The axial coding stage samples 30 or 50 bugs per cluster based on availability, which is a purposive, diversity-oriented design that does not preserve the relative frequencies of clusters or bug types; no per-category counts are reported for the 230 manually analyzed bugs. The Kaggle evidence covers only 89 notebooks with tracebacks, mapped to four categories (59 compiler errors, 20 configuration errors, 5 logic errors, 5 API misuses), so it does not include five of the six categories listed as most common. The authors should report frequencies from a random sample or rephrase the RQ2 summary and abstract to avoid claiming a ranking that the data cannot support.","section":"Section III-C and RQ2 Summary (Section V)"},{"comment":"The identification of bug-related commits relies on a keyword heuristic ('fix', 'bug', 'patch' with exclusions) adapted from general software defect prediction work (references [28], [66]) and is not validated on Jupyter Notebooks. Because this heuristic defines the dependent variable for RQ1 and the sampling pool for RQ2, false positives (e.g., 'fix typo', 'fix formatting') and false negatives (e.g., 'correct the calculation') can bias both the correlation results and the taxonomy. The authors should validate the heuristic on a random sample of commits with manual labels and report precision and recall, or use a more notebook-specific bug detection approach.","section":"Section III-B"},{"comment":"The Pearson correlation analysis reports coefficients without p-values, confidence intervals, or a definition of the color thresholds for 'weak', 'medium', and 'strong' correlations. The conclusion that 'the number of contributors metric is strongly correlated' is based on a single coefficient (r = 0.41) with no significance test; count-based metrics such as number of contributors and file modification frequency are likely non-normal, so the suitability of Pearson should be justified. The authors should add significance tests or bootstrap confidence intervals to support the RQ1 claims.","section":"Section IV, Figure 3"}],"minor_comments":[{"comment":"The mapping between the printed coefficient values and the metric labels is ambiguous in the text; the figure should clearly associate each coefficient with its metric, since the reader must infer, for example, that 0.34 corresponds to libraries imported and 0.41 to number of contributors.","section":"Figure 3"},{"comment":"The paper reports 11 clusters for qualitative analysis but then presents a taxonomy of 12 categories; the authors should clarify how the 11 clusters map to the 12 categories.","section":"Section III-C and Section V"},{"comment":"The security commit pipeline reduces 400 regex-filtered commits to 323 after LLM filtering, then to 66 after removing automated scans, then to 62 after manual annotation; the paper should report how many of the 66 were removed by manual annotation and the agreement between annotators.","section":"Section III-D"},{"comment":"The abstract states that 'configuration issues are among the most common bugs... followed by incorrect API usage', which is broader than the RQ2 summary; the abstract should be aligned with the actual support provided by the data.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The main obstruction to acceptance is the RQ2 frequency claim. The taxonomy itself is a reasonable qualitative product, and the security analysis is solid. If the authors can either provide frequency counts from a representative sample or reframe the claim as a taxonomy without ranking, the paper could be acceptable. I also note that the keyword heuristic from the authors' own prior work (Lutellier et al. [28]) is not inherently problematic, but validation is needed because it feeds both the correlation analysis and the taxonomy sampling."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the 12-category taxonomy of source-level bugs in Jupyter notebooks, built from 230 manually inspected fixes, plus the manually validated CWE mapping for Jupyter infrastructure commits. Both are real contributions. The categories (incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, documentation errors) are plausible, well-illustrated with concrete fixes, and complement the higher-level problem taxonomy from De Santana et al. The security analysis is careful: regex + LLM filtering followed by manual validation, ending with 62 commits, and the claims there are backed by counts and examples.\n\nThe soft spots are real and load-bearing. First, the RQ2 \"most common root causes\" claim in the summary is not supported by the reported method. The 230 bugs were sampled to maximize diversity across DBSCAN clusters: each author sampled 30 or 50 bugs per category, iterating to saturation. That is a purposive sample, not a representative one, so you cannot rank root causes by frequency from it. No per-category counts are given for the 230 bugs. The Kaggle data does not rescue this: only 89 notebooks had tracebacks, mapped to four categories, with compiler errors dominating. The sentence naming the six \"most common\" categories should be withdrawn or replaced with counts from a random sample. Second, RQ1 is internally inconsistent: the text says the number of contributors is \"strongly correlated\" with bug-related commits, but Fig. 3 reports r = -0.08 for exactly that variable. File modification frequency (r = 0.41) is actually the strongest correlate. The section also uses causal language for correlations and reports no significance tests. These problems are fixable but need an honest rewrite. Third, the bug-related commit label relies on the \"fix/bug/patch\" keyword heuristic without precision/recall validation on notebooks; that is a minor concern here given the qualitative nature of the main contribution, but it still affects RQ1 and the RQ2 pool.\n\nNone of this kills the qualitative taxonomy, which is the paper's best asset. If the authors reframe RQ2 as a qualitative taxonomy without frequency ranking, fix the RQ1 wording and add significance reporting, this becomes a solid empirical software-engineering-for-data-science paper. As it stands, it deserves a serious referee with a conditional decision. I would send it out.","headline":"Useful qualitative taxonomy of Jupyter notebook bugs, but the frequency ordering is unsupported by the sampling design and RQ1 has an internal contradiction.","tokens_in":19412,"tokens_out":2813,"would_cite":true,"duration_ms":27930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The most common bugs in Jupyter notebooks fall into twelve root-cause categories led by incorrect configuration, data-shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors, while Jupyter deployment frameworks…","keywords":["Jupyter Notebooks","empirical software engineering","bug taxonomy","grounded theory","software vulnerabilities","data science","configuration errors","API misuse"],"falsifier":"Take a random sample of 100 commits the heuristic labels as bug-related from the 376 repositories, have two independent annotators classify each as a genuine bug fix, and check whether excluding the false positives changes the ordering of the top root-cause categories; if it does, the claim that configuration and API misuse are the most common notebook bugs needs revision.","tokens_in":18443,"feed_emoji":"🐛","tokens_out":8108,"duration_ms":74729,"temperature":0.7,"pith_summary":"The paper tries to establish, with data, what actually goes wrong in Jupyter notebooks and in the software that runs them. It argues that notebook bugs have a recognizable source-code-level signature: twelve root-cause categories, with incorrect configuration, data-shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors the most common, and that these categories appear consistently across GitHub and Kaggle notebooks. It also argues that the characteristics most associated with bug-fixing commits are social rather than technical—number of contributors and file-modification frequency outweigh code complexity and documentation metrics—and that Jupyter deployment frameworks, not notebook documents themselves, carry the security risk, through web-style vulnerabilities like CSRF and access-control gaps. If correct, the study gives tool builders and teams a concrete checklist of what to check first and shifts attention from notebook content to configuration, collaboration, and deployment hygiene.","feed_headline":"Configuration mistakes top the list of Jupyter notebook bugs","feed_subtitle":"An analysis of thousands of notebooks yields a 12-category bug taxonomy and points to web-style risks in Jupyter servers.","key_machinery":"The paper's central mechanism is a grounded-theory coding pipeline for notebook bug fixes, where categories are built by iterative reading and coding rather than imposed in advance. Because notebooks are JSON documents whose line diffs are polluted by metadata and outputs, the authors serialize each notebook's Python code cells, parse them into abstract syntax trees (ASTs), and diff the trees; PCA on eight change metrics and DBSCAN clustering then produced eleven sampling clusters with a silhouette score of 0.91, from which three co-authors independently coded sampled fixes, discussed, and resampled until no new categories emerged after 230 bugs. For security, a regex filter followed by an LLM (DeepSeek-V3) mapped commit messages to CWE Top 25 entries, with manual validation reducing 323 candidates to 62. The taxonomy itself is the deliverable that carries the central claim.","core_discovery":"Based on 8,647 notebooks from 376 active GitHub repositories and 1,038 Kaggle competition notebooks, the paper derives a taxonomy of twelve bug root causes in Jupyter notebooks: incorrect configuration, data shape/structure mistakes, API misuse, incorrect syntax, wrong logic, non-determinism, exception/error/log/debugging issues, errors in test code and assertions, resource management, incomplete code, undeclared variables and typos, and documentation mistakes. The most common categories, per the authors' manual coding of 230 bug fixes, are incorrect configuration, data shape mistakes, API misuse, incomplete code, wrong logic, and documentation errors; 89 tracebacks in the Kaggle logs independently show configuration errors, compiler errors, API misuse, and logic errors. On the quantitative side, the number of contributors and the frequency of file modification correlate most strongly with bug-related commits, while cyclomatic complexity and code-to-markdown ratio are weak or negligible. On security, the authors report no security issues inside notebook documents themselves, but 62 validated security-related commits in JupyterHub, Jupyter Server, and JupyterLab repositories, mapped to CWE categories, are dominated by protection-mechanism failure (CWE-693), improper access control (CWE-284), and improper resource-lifecycle control (CWE-664), with CSRF the most recurrent single issue and a visible tension between usability and security in some fixes.","pith_inferences":["Beyond the paper, the commit-based method can only see bugs that were noticed and fixed; silent wrong-output bugs are likely underrepresented, so 'wrong logic' may be rarer in the ranking than in reality.","Beyond the paper, the strong contributor correlation may partly reflect popularity rather than collaboration: popular notebooks attract both more editors and more fixes, and normalizing by file age or views would test that alternative explanation.","Beyond the paper, the security analysis covers only three GitHub organizations; the reported doubling of Jupyter-related CVEs between 2023 and 2024 suggests the broader plugin and server ecosystem deserves an equivalent dependency-level audit.","Beyond the paper, the same taxonomy could be tested against execution traces from CI pipelines or enterprise notebook platforms to see whether configuration and API misuse still dominate outside GitHub and Kaggle."],"forward_implications":["Bug-finding and linting tools for notebooks should target configuration strings, data-shape operations, and API call sites first, since those categories dominate the taxonomy.","Teams adopting notebooks should treat collaboration metrics as risk signals: files touched by many contributors or modified frequently are where bug-fixing commits concentrate, so code review and change management matter more than code length in notebooks.","Configuration management practices such as pinning library versions, keeping environment files, and using containers would address the single largest root-cause category identified in the study.","Organizations exposing JupyterHub, Jupyter Server, or JupyterLab beyond a local machine should harden authentication, TLS, cross-origin policies, and dependency updates, because framework vulnerabilities are predominantly web-based.","The absence of security findings inside notebook documents, combined with the framework findings, shifts security attention from notebook content to the deployment layer."],"supporting_citations":[{"why":"The concurrent, higher-level taxonomy of Jupyter notebook problems that this paper's source-code-level taxonomy complements.","marker":"[46]"},{"why":"Supplies the keyword heuristic used to identify bug-related commits from commit messages.","marker":"[28]"},{"why":"Also provides the keyword-based bug-commit labeling method, cited alongside [28].","marker":"[66]"},{"why":"DeepSeek-V3, the LLM used to map security commit messages to CWE entries.","marker":"[9]"},{"why":"The 2024 CWE Top 25 list that defines the classification target for security-related commits.","marker":"[33]"},{"why":"The regex-based method for filtering security-related commits, the first stage of the RQ3 pipeline.","marker":"[72]"}],"fun_headline_variants":["Configuration errors top Jupyter notebook bug list","Jupyter notebooks: 12 bug types, configuration most common","JupyterHub and JupyterLab carry web-style security risks","Study of 8,647 notebooks uncovers Jupyter bug taxonomy","Notebook configuration mistakes beat API misuse in bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole study leans on a keyword rule that calls a commit bug-related if its message contains 'fix,' 'bug,' or 'patch' and none of 'rename,' 'merge,' 'clean-up,' or 'refactor'; if that labeling is noisy for notebooks, the correlations and the taxonomy sample inherit the noise.","fun_headline_variants_meta":{"raw":{"variants":["Configuration errors top Jupyter notebook bug list","Jupyter notebooks: 12 bug types, configuration most common","JupyterHub and JupyterLab carry web-style security risks","Study of 8,647 notebooks uncovers Jupyter bug taxonomy","Notebook configuration mistakes beat API misuse in bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1413,"prompt_tokens":1049,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":665,"tokens_out":364,"duration_ms":3894,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:07:07.158971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 100 commits the heuristic labels as bug-related from the 376 repositories, have two independent annotators classify each as a genuine bug fix, and check whether excluding the false positives changes the ordering of the top root-cause categories; if it does, the claim that configuration and API misuse are the most common notebook bugs needs revision.","supporting_citations":[{"cited_title":"In: Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the keyword heuristic used to identify bug-related commits from commit messages."},{"cited_title":"In: Proceedings of the 38th international conference on software engineering, pp","cited_arxiv_id":null,"evidence_quote":"Also provides the keyword-based bug-commit labeling method, cited alongside [28]."},{"cited_title":"https: //cwe.mitre.org/top25/archive/2024/2024 cwe top25.html (2024)","cited_arxiv_id":null,"evidence_quote":"The 2024 CWE Top 25 list that defines the classification target for security-related commits."},{"cited_title":"In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp","cited_arxiv_id":null,"evidence_quote":"The regex-based method for filtering security-related commits, the first stage of the RQ3 pipeline."}],"review_version":2}