Pith. sign in

REVIEW 3 major objections 6 minor 36 references

CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CRABS recovers a notebook's information flow graph at 98.57% F1 and its transitive cell execution dependencies at 99.67% F1, all without executing the notebook, by bounding the inter-cell I/O set syntactically and letting an LLM resolve…

desk verdict A solid, well-scoped neuro-symbolic method for notebook dataflow extraction; the headline numbers are real but measured on a curated subset, and the LLM prompts share the annotation rubric. read the letter →

arxiv 2507.11742 v2 pith:TIBNXCRO submitted 2025-07-15 cs.CL cs.AI

Meng Li , Timothy M. McPhillips , Dingmin Wang , Shin-Rong Tsai , Bertram Ludäscher This is my paper · ORCID
classification cs.CLcs.AI
keywords informationflowgraphcellexecutiondependenciesPythonnotebooksdataflowanalysisLLMcodeunderstandingsyntactic-semantichybridzero-shotpromptingnotebook
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CRABS aims to show that the information flows inside a Python notebook—which cells send which variables to which later cells—can be recovered accurately without executing the notebook or resolving its software dependencies. The paper proposes treating each cell's inter-cell inputs and outputs as a set that syntactic analysis can bracket from below and above, leaving only a small, well-scoped set of ambiguities for a large language model to resolve with simple yes/no questions. On 50 annotated notebooks, the strategy reaches 98.57% F1 for cell-to-cell information flows and 99.67% F1 for transitive execution dependencies, with exact graph matches for 74% and 82% of notebooks. The significance is that a cheap symbolic phase plus a focused semantic phase can outperform asking an LLM to read the whole notebook, while also eliminating hallucinated variables and long-context failures. The paper's claims are conditional on a curated evaluation set that respects the syntactic phase's assumptions.

What carries the argument

The central object is the inter-cell I/O set: the set of all triples (cell, variable, in/out) recording which variables flow into and out of each cell. CRABS's load-bearing mechanism is the pair of lower and upper estimates of this set produced by shallow AST analysis—the lower estimate includes only flows that are syntactically certain, the upper estimate includes every syntactically possible flow, including in-place mutations, loop iterators, and shared references—so that the true I/O set is guaranteed to lie between them. The difference between the two estimates is exactly the ambiguity budget given to the LLM, which resolves each ambiguous member with a constrained zero-shot yes/no prompt rather than free-form description. This bounding-first, resolving-second design is what carries the argument: it converts an open-ended whole-notebook comprehension task into a finite set of small classification tasks.

What would settle it

Take 50 notebooks that violate at least one of CRABS's stated assumptions (e.g., notebooks containing %%capture magic commands, global variables read inside function bodies, or a variable renamed as a function after definition), hand-annotate their information flows, and check whether the syntactic lower estimate remains a subgraph and the upper estimate a supergraph of the ground truth. If any violation causes the bounds to fail to bracket the true flow set, or if the LLM resolution accuracy drops well below 98%, the paper's central boundedness and accuracy claims are falsified outside the curated regime.

Watch

Extended reading notes

Core claim

The central discovery is that the notebook understanding task—constructing an information flow graph and the induced cell execution dependency graph—can be solved to high accuracy, with F1 scores above 98%, by a two-phase pincer. The syntactic phase analyzes only the visible code in each cell to produce a lower estimate (flows that are certain) and an upper estimate (flows that are certain or possible) of the inter-cell I/O set, which bracket the ground truth as subgraph and supergraph respectively. The semantic phase then feeds the ambiguous members of this bracketed set to an LLM one cell at a time, asking only 'is this variable an input?' or 'is this an output candidate?'—binary questions whose answers are selected from the syntactic candidates, so the model cannot invent variables. The result is that 1397 of 1425 (98%) individual ambiguities are resolved correctly, and the final graphs match human annotations on 74–82% of notebooks. The paper frames this as evidence that a syntactic-semantic pincer, which bounds both the LLM's workload and the space of its answers, is a viable general strategy for neuro-symbolic code understanding.

Load-bearing premise

The evaluation set was deliberately curated so that every notebook satisfies CRABS's core assumptions—cells execute strictly top-to-bottom, variable and function names never clash, globals are not used inside functions unless passed in, and no non-Python (magic or shell) statements appear—so the guaranteed bounds and the reported 98% accuracy are only established for notebooks that meet these criteria.

Editorial extensions

If this is right

  • If CRABS generalizes beyond the curated set, notebook comprehension tools can map data provenance without executing cells, enabling reuse and debugging of notebooks whose dependencies cannot be resolved.
  • The lower/upper estimates provide certified bounds on cell-to-cell flows for notebooks satisfying the stated assumptions, so downstream tools can know which flows are certain, which are possible, and which are excluded.
  • The cell-by-cell prompting strategy's 98% resolution accuracy suggests that LLM effort is best spent on many small, syntax-bounded questions rather than one large analysis.
  • The dependency graph derived from the flow graph reaches 99.67% F1, implying that transitive 'what affects what' queries about notebook cells become reliably answerable.
  • Ablations show that removing either pincer arm degrades performance substantially, indicating both syntactic bounding and cell-by-cell resolution are load-bearing, not incidental.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bounded-estimate idea could be lifted to other artifact types—SQL scripts, shell pipelines, and computational documents—where a cheap static analysis can delimit the candidate dataflow edges and a language model resolves the residue.
  • A natural stress test the paper does not run: apply CRABS to notebooks that violate its assumptions (magic commands, globals read inside functions, reused names) and measure how often the lower/upper bounds still bracket the ground truth; if the brackets break, the guarantee is an artifact of the curated sample.
  • Because the LLM resolves ambiguities independently per cell, the per-cell questions could be parallelized, and the paper's own latency appendix already points to a concurrent implementation being consistently faster than the baseline; a practical interactive notebook inspector seems within reach.
  • The 74% exact-match rate on flow graphs suggests that residual errors concentrate in a few hard patterns (likely in-place mutations and aliasing); specializing the resolution prompts toward those patterns, or adding a third syntactic pass for hidden modifications, could close the remaining gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CRABS, a two-phase strategy for recovering cell-to-cell information flows and execution dependencies from Python notebooks without executing them. A syntactic phase builds an AST and computes lower and upper estimates of the inter-cell I/O set; a semantic-aware phase then prompts an LLM to resolve the ambiguous candidates cell by cell with binary questions. The authors contribute a hand-annotated dataset of 50 curated Kaggle notebooks, report 98.57% F1 for information flows and 99.67% F1 for transitive dependencies (Table 2), and show through ablations that both the syntactic phase and cell-by-cell prompting contribute to performance. The paper also includes experiments with GPT-4o-mini, Qwen3-8B, and Qwen2.5-Coder.

Significance. If the reported results are taken at face value within the paper's stated assumptions, CRABS is a useful demonstration that a small amount of symbolic analysis can constrain an LLM on a structured code-understanding task: cell-by-cell prompting avoids the long-context failures of the baseline, the binary-question format prevents variable hallucination, and the syntactic bounds make the method's behavior transparent. The public release of code and data, the multi-LLM evaluation, and the ablation design are concrete strengths. The main caveat is that the evaluation is restricted to notebooks deliberately selected to satisfy the method's assumptions, so the headline numbers should not be read as evidence about general notebook corpora until that scope is tested or clearly stated.

major comments (3)
  1. [Section 5.1, Table 5; Section 4.1] The evaluation set is not representative of 'Python notebooks' in general. The 50 notebooks were selected from the 104 most up-voted Python notebooks by applying three criteria: an ML workflow, no global-variable-in-function or name-reuse patterns, and a single connected information flow with no external-file routing. Table 5 shows that 54 of the 104 notebooks were excluded for violating these criteria. Since the lower/upper estimate guarantees in Section 4.1 are explicitly conditional on the same assumptions (strict top-to-bottom execution, distinct data/code names, no hidden globals, no non-Python cells), the Table 2 results hold only for the curated subset. Moreover, the upper estimate is not a true superset on notebooks that violate these assumptions: for example, Appendix A.2.2 (Figure 12) shows that globals accessed inside functions are not recognized, so the upper estimate misses the flow (cell 1, cell 2, data_file_path), and the LLM cannot recover it because it is not an ambiguous candidate. The abstract and contributions claim that CRABS 'bound[s] LLM interpretation of Python notebooks' without this qualification. Please either (a) consistently scope the claims to assumption-satisfying notebooks, (b) measure performance on the unfiltered top-104 corpus and report error modes on the violating notebooks, or (c) provide an automatic checker for the assumptions and quantify how common violations are in a larger sample.
  2. [Section 4.2, Figures 15 and 16] The semantic-phase prompts contain the same semantic rules that the authors used to create the ground-truth annotations. For example, Figure 16 states 'If a method (or a function) modifies the object in place..., the object is an output candidate' and enumerates shared-reference propagation rules. The 1397/1425 (98%) ambiguity-resolution accuracy therefore largely measures whether the LLM can apply the provided rules to a snippet, not whether the rules themselves are valid or whether the LLM discovers them independently. This is compounded by the fact that the ground truth is self-annotated (Section 5.1) with no reported inter-annotator agreement. To support the claim that the LLM contributes semantic insight, the authors should either test with prompts that do not encode the target rules, provide an annotation codebook with inter-annotator agreement, or reframe the resolution phase as a rule-following check and adjust the significance accordingly.
  3. [Section 5.3, Tables 2 and 10] The headline results are reported as point estimates without confidence intervals, while the per-notebook scores in Table 10 show substantial variability (e.g., notebook 03 at 96.58% vs. many at 100%). Given that the comparison with the baseline and ablations rests on aggregated differences of a few percentage points (Table 3), the paper should report confidence intervals or a paired significance test. This is particularly important because the exact-match metric is based on only 37/50 and 41/50 notebooks, so a small number of notebooks changes EM substantially.
minor comments (6)
  1. [Introduction] Page 2 contains a typo: 're-exeution' should be 're-execution'.
  2. [Section 4.1] The notation S ⊂ {(c, ik, tag) | c ∈ {cs, ct}, ...} is confusing because cs and ct are not introduced in the same scope; define c as a generic cell identifier or explain that cs and ct are the source and target cells of the flow.
  3. [Section 4.2] The abstract says 'zero-shot learning' and Section 4.2 says 'zero-shot in-context learning'; since no few-shot examples are provided, please use 'zero-shot prompting' consistently.
  4. [Section 5.1] The statement that the 50 selected notebooks are 'representative of general data science and machine learning notebooks' is not supported by the exclusion criteria; please rephrase to 'representative of notebooks satisfying the stated criteria' or provide additional evidence about the broader corpus.
  5. [Appendix A.2] The execution-time analysis is presented as a preliminary result in the Future Work section; consider moving it to a clearly labeled experiments appendix or a separate section, and report confidence intervals for the regression slopes.
  6. [Figures 15 and 16] The 'Important Cases' blocks in the prompts are effectively rule specifications; consider labeling them as 'heuristics' or 'provided rules' in the text to avoid implying that the LLM derives them independently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CRABS' syntactic bounds are fixed rule-based estimates, the ground truth is an external human-annotated dataset, and the LLM resolves concrete per-cell ambiguities not already contained in its syntactic inputs.

full rationale

CRABS' derivation chain is self-contained. The syntactic phase computes lower and upper estimates from fixed AST rules with no fitted parameters and no use of the ground-truth annotations; the upper estimate's superset property is an over-approximation design goal subject to the paper's stated assumptions, not a quantity fitted to the target data. The ground truth is an external, hand-annotated dataset produced independently of CRABS' outputs (Section 5.1), and the LLM resolves per-cell ambiguities zero-shot against that external truth. Although the prompts in Figures 15 and 16 encode general definitions of in-place modification and shared references that align with the annotation concept, this is task specification rather than a reduction: the LLM must still decide for each concrete cell whether a method mutates an object, and those decisions are not read off from the lower/upper estimates. The paper explicitly scopes its guarantees in Section 4.1 and the appendix (global variables, non-Python cells, hidden modifications), and the exclusion of 54 notebooks from the evaluation set is a validity limitation rather than a circular step. YesWorkflow and other self-citations are motivational/related work and carry no load-bearing proof. No step in the paper's derivation reduces by construction to its own inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

CRABS introduces no fitted numerical parameters and no new physical or conceptual entities. It rests on a set of stated modeling assumptions about notebook execution and name usage, plus a curated dataset that guarantees these assumptions hold.

assumptions (5)
  • domain assumption Cells are executed in strict top-to-bottom order from a cleared initial state.
    Stated in Section 3; defines the information flow graph with s < t. Real notebooks can be executed out of order, which would break the model.
  • domain assumption All code is reachable and every branch of a conditional may execute.
    Stated in Section 4.1; used to define union/intersection rules for conditional statements. In practice, some branches may never execute for real inputs.
  • domain assumption Function and class names are never reused as variable names, and globals are not accessed inside functions unless passed as arguments.
    Stated in Section 4.1; the implementation cannot detect flows through such name reuse or implicit globals, so notebooks violating this are excluded from the dataset.
  • domain assumption Non-Python cells such as shell commands and IPython magic commands are ignored.
    Stated in Section 4.1 as a limitation; this can miss flows, as shown in Appendix A.1.4.
  • ad hoc to paper The 50 evaluation notebooks were curated to satisfy the above assumptions.
    Section 5.1 lists criteria C1-C3 and Table 5 documents exclusions; the reported F1 scores apply only to this curated set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks." pith.science (2026). https://pith.science/paper/TIBNXCRO

@misc{pith2026250711742,
  author       = {Pith},
  title        = {Pith review of: CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIBNXCRO}},
  note         = {Machine review of arXiv:2507.11742}
}
abstract

Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating a notebook via re-execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due to hallucinations and long-context challenges. To address these issues, we propose a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of a pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O set$\unicode{x2014}$the flows of information into or out of cells via variables$\unicode{x2014}$then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 notebooks, CRABS achieves average F1 scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependencies.

Figures

Figures reproduced from arXiv: 2507.11742 by the authors.

Figure 1
Figure 1. An overview of CRABS using the example notebook in Figure [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Ground-truth information flow graph for the example notebook ( [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. An example Python notebook with six code cells, representing a simplified data preparation pipeline for symptom burden prediction using smartphone records. phone cell 1 cell 2 cell 4 cell 5 cell 6 phone survey survey phone cell 3 ec ec [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figures from the paper (14 more)
Figure 5
Figure 5. Figure 5: An example shared reference issue. Figure 5: An example shared reference issue. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Information flow graphs derived from the (a) [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 10
Figure 10. Figure 10: An example case depending on runtime information. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: An example case for reusing a data name as a code name. Figure 10: An example case depending on runtime information [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: An example case for the simplified function parsing approach. Figure 9 2 datafilepath = "data.csv" 1 # cell 2 [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: An example case for the mixed code issue. pppg , cell 1). [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: An example case for conditional statements in a loop. 2 count = 1 time for a notebook, defined as the total of the latencies of the individual requests involved [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 12
Figure 12. Figure 12: Scatter plot of execution time vs. number of ambiguities per notebook with least [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Scatter plot of execution time vs. number of information flows per notebook with [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: An example baseline prompt (in part). The real prompt includes all code cells. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: An example CRABS prompt for resolving ambiguous inputs [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: An example CRABS prompt for resolving ambiguous output candidates [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: An example prompt for the ablation study S1 (i.e., w/o syntactic phase). [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Part of an example prompt for the ablation study S2 (i.e., w/o cell-by-cell [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 21 canonical work pages

  1. [1]

    Your notebook is not crumby enough, REP lace it

    Michael Brachmann, William Spoth, Oliver Kennedy, Boris Glavic, Heiko Mueller, Sonia Castelo, Carlos Bautista, and Juliana Freire. Your notebook is not crumby enough, REP lace it. In 10th Annual Conference on Innovative Data Systems Research (CIDR 2020), 2020

    work page 2020

  2. [2]

    Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37: 0 107703--107744, 2024

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37: 0 107703--107744, 2024

    2024

  3. [3]

    Niw: Converting notebooks into workflows to capture dataflow and provenance

    Lucas AMC Carvalho, Regina Wang, Yolanda Gil, and Daniel Garijo. Niw: Converting notebooks into workflows to capture dataflow and provenance. In K-CAP Workshops, pp.\ 12--16, 2017

    work page 2017

  4. [4]

    Dataflow-guided retrieval augmentation for repository-level code completion

    Wei Cheng, Yuhan Wu, and Wei Hu. Dataflow-guided retrieval augmentation for repository-level code completion. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7957--7977, Bangkok, Thailand, August 2024. Association for Computationa...

    doi:10.18653/v1/2024.acl-long.431 2024

  5. [5]

    Applications of link prediction in social networks: A review

    Nur Nasuha Daud, Siti Hafizah Ab Hamid , Muntadher Saadoon, Firdaus Sahran, and Nor Badrul Anuar. Applications of link prediction in social networks: A review. Journal of Network and Computer Applications, 166: 0 102716, 2020. ISSN 1084-8045. doi:https://doi.org/10.1016/j.jnca.2020.102716. URL https://www.sciencedirect.com/science/article/pii/S1084804520301909

    arXiv 2020

  6. [6]

    Jeremy Goecks, Anton Nekrutenko, James Taylor, and Galaxy Team team@ galaxyproject. org. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome biology, 11: 0 1--13, 2010

    work page 2010

  7. [7]

    Graphcodebert: Pre-training code representations with data flow

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations, ...

    work page 2021

  8. [8]

    Analyzing the Performance of Large Language Models on Code Summarization

    Rajarshi Haldar and Julia Hockenmaier. Analyzing the Performance of Large Language Models on Code Summarization . In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics , Language Resources and Evaluation ( LREC - CO...

    work page 2024

Show all 36 references
  1. [9]

    A review on evaluation metrics for data classification evaluations

    Mohammad Hossin and Md Nasir Sulaiman. A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5 0 (2): 0 1, 2015

    2015

  2. [10]

    Execution-based evaluation for data science code generation models

    Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. Execution-based evaluation for data science code generation models. In Eduard Dragut, Yunyao Li, Lucian Popa, Slobodan Vucetic, and Shashank Srivastava (eds.), ...

    2022

  3. [11]

    Junjie Huang, Daya Guo, Chenglong Wang, Jiazhen Gu, Shuai Lu, Jeevana Priya Inala, Cong Yan, Jianfeng Gao, Nan Duan, and Michael R. Lyu. Contextualized data-wrangling code generation in computational notebooks. In Proceedings of the 39th IEEE/ACM International Conference on Au...

    2024

  4. [12]

    Suggesting code edits in interactive machine learning notebooks using large language models

    Bihui Jin, Jiayue Wang, and Pengyu Nie. Suggesting code edits in interactive machine learning notebooks using large language models. arXiv preprint arXiv:2501.09745, 2025

    2025 arXiv

  5. [13]

    Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B

    Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P \'e rez, Brian E. Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B. Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Dami \'a n Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. Jupyter...

    2016

  6. [14]

    Information retrieval on the web

    Mei Kobayashi and Koichi Takeda. Information retrieval on the web. ACM computing surveys (CSUR), 32 0 (2): 0 144--173, 2000

    2000

  7. [15]

    Dataflow notebooks: encoding and tracking dependencies of cells

    David Koop and Jay Patel. Dataflow notebooks: encoding and tracking dependencies of cells. In Proceedings of the 9th USENIX Conference on Theory and Practice of Provenance , TaPP '17, pp.\ 17, USA, June 2017. USENIX Association

    2017

  8. [16]

    Snakemake—a scalable bioinformatics workflow engine

    Johannes Köster and Sven Rahmann. Snakemake—a scalable bioinformatics workflow engine. Bioinformatics, 28 0 (19): 0 2520--2522, 08 2012. ISSN 1367-4803. doi:10.1093/bioinformatics/bts480. URL https://doi.org/10.1093/bioinformatics/bts480

    2012 doi

  9. [17]

    Python code generation by asking clarification questions

    Haau-Sing (Xiaocheng) Li, Mohsen Mesgar, Andr \'e Martins, and Iryna Gurevych. Python code generation by asking clarification questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational L...

    2023 doi

  10. [18]

    Demonstration of elasticnotebook: Migrating live computational notebook states

    Zhaoheng Li, Supawit Chockchowwat, Hanxi Fang, Ribhav Sahu, Sumay Thakurdesai, Kantanat Pridaphatrakun, and Yongjoo Park. Demonstration of elasticnotebook: Migrating live computational notebook states. In Companion of the 2024 International Conference on Management of Data, pp...

    2024

  11. [19]

    Scientific workflow management and the kepler system

    Bertram Lud \"a scher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A Lee, Jing Tao, and Yang Zhao. Scientific workflow management and the kepler system. Concurrency and computation: Practice and experience, 18 0 (10): 0 1039--1065, 2006

    2006

  12. [20]

    A large language model-assisted education tool to provide feedback on open-ended responses

    Jordan K Matelsky, Felipe Parodi, Tony Liu, Richard D Lange, and Konrad P Kording. A large language model-assisted education tool to provide feedback on open-ended responses. arXiv preprint arXiv:2308.02439, 2023

    2023 arXiv

  13. [21]

    Kyle Bocinsky, Yang Cao, James Cheney, Fernando Chirigati, Saumen Dey, Juliana Freire, Christopher Jones, James Hanken, Keith W

    Timothy McPhillips, Tianhong Song, Tyler Kolisnik, Steve Aulenbach, Khalid Belhajjame, R. Kyle Bocinsky, Yang Cao, James Cheney, Fernando Chirigati, Saumen Dey, Juliana Freire, Christopher Jones, James Hanken, Keith W. Kintigh, Timothy A. Kohler, David Koop, James A. Macklin, ...

    2015

  14. [22]

    Using an llm to help with code understanding

    Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an llm to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Mac...

    2024

  15. [23]

    Taverna: a tool for the composition and enactment of bioinformatics workflows

    Tom Oinn, Matthew Addis, Justin Ferris, Darren Marvin, Martin Senger, Mark Greenwood, Tim Carver, Kevin Glover, Matthew R Pocock, Anil Wipat, et al. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20 0 (17): 0 3045--3054, 2004

    2004

  16. [24]

    A large-scale study about quality and reproducibility of jupyter notebooks

    João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. A large-scale study about quality and reproducibility of jupyter notebooks. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp.\ 507--517, 2019. doi:10.1109/MSR....

    2019

  17. [25]

    Meta kaggle code, 2023

    Jim Plotts and Megan Risdal. Meta kaggle code, 2023. URL https://www.kaggle.com/ds/3240808. Accessed: 27 March 2025

    2023

  18. [26]

    Visualising data science workflows to support third-party notebook comprehension: an empirical study

    Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. Visualising data science workflows to support third-party notebook comprehension: an empirical study. Empirical Software Engineering, 28 0 (3): 0 58, March 2023. ISSN 1573-7616. doi:10.1007/s1...

    2023 doi

  19. [27]

    Meta kaggle, 2022

    Megan Risdal and Timo Bozsolik. Meta kaggle, 2022. URL https://www.kaggle.com/ds/9. Accessed: 27 March 2025

    2022

  20. [28]

    Autodrug: fully automated macromolecular crystallography workflows for fragment-based drug discovery

    Yingssu Tsai, Scott E McPhillips, Ana Gonz \'a lez, Timothy M McPhillips, Daniel Zinn, Aina E Cohen, Michael D Feese, David Bushnell, Theresa Tiefenbrunn, CDavid Stout, et al. Autodrug: fully automated macromolecular crystallography workflows for fragment-based drug discovery....

    2013

  21. [29]

    LLMDFA : Analyzing Dataflow in Code with Large Language Models

    Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA : Analyzing Dataflow in Code with Large Language Models . In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Proce...

    2024

  22. [30]

    Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of- Thought Prompting Elicits Reasoning in Large Language Models . Advances in Neural Information Processing Systems, 35: 0 24824--24837, December 2022. URL...

    2022

  23. [31]

    Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure

    John Wenskovitch, Jian Zhao, Scott Carter, Matthew Cooper, and Chris North. Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure . In 2019 IEEE Visualization in Data Science ( VDS ) , pp.\ 1--10, Vancouver, BC, Canada, October 2019. IEEE. ISBN...

    2019

  24. [32]

    Natural Language to Code Generation in Interactive Data Science Notebooks

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural Language to Code Generation in Interactive Data Science Notebooks . In Anna Rogers, J...

    2023

  25. [33]

    R epo C oder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. R epo C oder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 20...

    2023 doi

  26. [34]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  27. [35]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  28. [36]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.