{"id":"dbf875ac-7dc0-4204-b1a2-53641028a2f5","arxiv_id":"2507.11742","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CRABS combines AST bounds with an LLM to reconstruct information flow and execution dependency graphs for Python notebooks, reaching 98% F1 on 50 curated Kaggle notebooks.","lead":"CRABS is a hybrid method that combines lightweight Python code analysis with an LLM to map how data and variables flow between cells in Jupyter notebooks, without running the notebook. On 50 curated Kaggle notebooks it recovers cell-to-cell information flows with 98% F1, which could make notebook reuse and auditing practical at scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98.57% F1 result is measured on 50 notebooks hand-selected to satisfy CRABS' syntactic assumptions; the 54 excluded top-Kaggle notebooks are untested, so the claim that CRABS bounds interpretation of Python notebooks is not yet supported on realistic, unfiltered notebooks.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the evaluation set was constructed to satisfy CRABS' core assumptions, so the high F1 does not yet demonstrate usefulness on realistic notebooks that violate them. I agree with the CONDITIONAL verdict. The paper is transparent about the limitations and provides code/data, but the selection rate (50/104) and the nature of the excluded notebooks (globals, name reuse, external-file routing, multiple workflows) mean the central generalization claim is untested. No internal inconsistency was found in the reported numbers for the curated set; the concern is external validity. A concrete next step is to evaluate on the excluded notebooks or an unfiltered random sample, which would settle whether CRABS is a general bounding strategy or a well-engineered method for a restricted notebook class.","tokens_in":24919,"tokens_out":8854,"duration_ms":109347,"concrete_test":"Run CRABS on the 54 notebooks excluded in Section 5.1 (or on a random sample of 50 Meta Kaggle notebooks not screened by the three inclusion criteria), manually annotate cell inputs/outputs with the same rubric, and compare F1/EM for information flows and transitive dependencies against Table 2. If the mean F1 drops materially below 98.57% / 99.67%, the headline result is selection-dependent and the paper must restrict its generalization claim to notebooks satisfying the syntactic assumptions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 assembles the evaluation set by inspecting the 104 most up-voted Python notebooks and keeping only the 50 that satisfy three criteria: an ML workflow, no global-variable-in-function or name-reuse patterns, and a single connected information flow with no external-file routing. Table 5 shows 54 notebooks excluded for violating these criteria. The syntactic-phase guarantees in Section 4.1 (lower estimate as a subset, upper estimate as a superset) are stated only under these assumptions; CRABS explicitly does not recognize globals accessed inside functions, non-Python/magic cells, or hidden modifications beyond simple aliasing. On any notebook violating these conditions, the upper estimate need not be a superset of the true flows, so the LLM resolution phase cannot recover the missing flows and the reported 98% F1 is not indicative. The paper calls the 50 notebooks 'representative,' but they are a minority (48%) of the top-104 sample and were chosen because they fit the method. The central claim should therefore be scoped to notebooks satisfying the stated assumptions until performance on unfiltered notebooks is measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRABS, a two-phase strategy for recovering cell-to-cell information flows and execution dependencies from Python notebooks without executing them. A syntactic phase builds an AST and computes lower and upper estimates of the inter-cell I/O set; a semantic-aware phase then prompts an LLM to resolve the ambiguous candidates cell by cell with binary questions. The authors contribute a hand-annotated dataset of 50 curated Kaggle notebooks, report 98.57% F1 for information flows and 99.67% F1 for transitive dependencies (Table 2), and show through ablations that both the syntactic phase and cell-by-cell prompting contribute to performance. The paper also includes experiments with GPT-4o-mini, Qwen3-8B, and Qwen2.5-Coder.","tokens_in":25146,"tokens_out":7755,"duration_ms":89573,"significance":"If the reported results are taken at face value within the paper's stated assumptions, CRABS is a useful demonstration that a small amount of symbolic analysis can constrain an LLM on a structured code-understanding task: cell-by-cell prompting avoids the long-context failures of the baseline, the binary-question format prevents variable hallucination, and the syntactic bounds make the method's behavior transparent. The public release of code and data, the multi-LLM evaluation, and the ablation design are concrete strengths. The main caveat is that the evaluation is restricted to notebooks deliberately selected to satisfy the method's assumptions, so the headline numbers should not be read as evidence about general notebook corpora until that scope is tested or clearly stated.","major_comments":[{"comment":"The evaluation set is not representative of 'Python notebooks' in general. The 50 notebooks were selected from the 104 most up-voted Python notebooks by applying three criteria: an ML workflow, no global-variable-in-function or name-reuse patterns, and a single connected information flow with no external-file routing. Table 5 shows that 54 of the 104 notebooks were excluded for violating these criteria. Since the lower/upper estimate guarantees in Section 4.1 are explicitly conditional on the same assumptions (strict top-to-bottom execution, distinct data/code names, no hidden globals, no non-Python cells), the Table 2 results hold only for the curated subset. Moreover, the upper estimate is not a true superset on notebooks that violate these assumptions: for example, Appendix A.2.2 (Figure 12) shows that globals accessed inside functions are not recognized, so the upper estimate misses the flow (cell 1, cell 2, data_file_path), and the LLM cannot recover it because it is not an ambiguous candidate. The abstract and contributions claim that CRABS 'bound[s] LLM interpretation of Python notebooks' without this qualification. Please either (a) consistently scope the claims to assumption-satisfying notebooks, (b) measure performance on the unfiltered top-104 corpus and report error modes on the violating notebooks, or (c) provide an automatic checker for the assumptions and quantify how common violations are in a larger sample.","section":"Section 5.1, Table 5; Section 4.1"},{"comment":"The semantic-phase prompts contain the same semantic rules that the authors used to create the ground-truth annotations. For example, Figure 16 states 'If a method (or a function) modifies the object in place..., the object is an output candidate' and enumerates shared-reference propagation rules. The 1397/1425 (98%) ambiguity-resolution accuracy therefore largely measures whether the LLM can apply the provided rules to a snippet, not whether the rules themselves are valid or whether the LLM discovers them independently. This is compounded by the fact that the ground truth is self-annotated (Section 5.1) with no reported inter-annotator agreement. To support the claim that the LLM contributes semantic insight, the authors should either test with prompts that do not encode the target rules, provide an annotation codebook with inter-annotator agreement, or reframe the resolution phase as a rule-following check and adjust the significance accordingly.","section":"Section 4.2, Figures 15 and 16"},{"comment":"The headline results are reported as point estimates without confidence intervals, while the per-notebook scores in Table 10 show substantial variability (e.g., notebook 03 at 96.58% vs. many at 100%). Given that the comparison with the baseline and ablations rests on aggregated differences of a few percentage points (Table 3), the paper should report confidence intervals or a paired significance test. This is particularly important because the exact-match metric is based on only 37/50 and 41/50 notebooks, so a small number of notebooks changes EM substantially.","section":"Section 5.3, Tables 2 and 10"}],"minor_comments":[{"comment":"Page 2 contains a typo: 're-exeution' should be 're-execution'.","section":"Introduction"},{"comment":"The notation S ⊂ {(c, ik, tag) | c ∈ {cs, ct}, ...} is confusing because cs and ct are not introduced in the same scope; define c as a generic cell identifier or explain that cs and ct are the source and target cells of the flow.","section":"Section 4.1"},{"comment":"The abstract says 'zero-shot learning' and Section 4.2 says 'zero-shot in-context learning'; since no few-shot examples are provided, please use 'zero-shot prompting' consistently.","section":"Section 4.2"},{"comment":"The statement that the 50 selected notebooks are 'representative of general data science and machine learning notebooks' is not supported by the exclusion criteria; please rephrase to 'representative of notebooks satisfying the stated criteria' or provide additional evidence about the broader corpus.","section":"Section 5.1"},{"comment":"The execution-time analysis is presented as a preliminary result in the Future Work section; consider moving it to a clearly labeled experiments appendix or a separate section, and report confidence intervals for the regression slopes.","section":"Appendix A.2"},{"comment":"The 'Important Cases' blocks in the prompts are effectively rule specifications; consider labeling them as 'heuristics' or 'provided rules' in the text to avoid implying that the LLM derives them independently.","section":"Figures 15 and 16"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for COLM, and the authors deserve credit for releasing code and data and for running a multi-LLM evaluation. My concerns are concentrated in two areas: the evaluation is on a hand-filtered subset of notebooks, and the semantic prompts encode the same rules used to create the ground truth, making the 98% resolution accuracy partly a measure of rule-following. Both issues are addressable in revision by scoping the claims, adding an unfiltered or at least more diverse evaluation, and reporting inter-annotator agreement. I would support publication if these points are convincingly handled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The pincer idea is the real contribution: AST-derived lower and upper bounds on inter-cell I/O, with the LLM resolving only the differences, is genuinely new relative to YesWorkflow, Albireo, and prior LLM dataflow work. The paper is also well-engineered: the ablations isolate the syntactic phase and cell-by-cell prompting, the artifact is public, and the failure analysis is honest. The 98.57% F1 for information flows is real on the dataset they built, and the 74% exact-match rate is strong.\n\nThe main soft spot is dataset selection. The 50 notebooks were hand-picked from 104 top-voted ones; 54 were excluded because they violate the method's assumptions. The bounds are only guaranteed under top-to-bottom execution, no name reuse, no globals-in-functions, and no magic cells. On notebooks outside that envelope, the upper bound need not be a superset of the true flows, and the LLM resolution phase cannot recover what the syntactic phase missed. The paper calls the 50 notebooks representative, but they are a minority of the sample and were chosen because they fit. This is a scoping issue, not a fatal one, because the criteria and failure modes are described clearly, but it does mean the central claim should be stated as conditional on those assumptions until an unfiltered evaluation is done.\n\nThe second concern is the prompt design. The semantic-phase prompts (Figures 15 and 16) hand the LLM the same in-place modification and shared-reference rules used to create the ground-truth annotations. So the 98% resolution accuracy partly measures rule-following rather than independent semantic insight. It is not fully circular—the LLM still decides per case—but the number should not be read as the LLM discovering dataflow semantics from scratch.\n\nMinor gaps: no confidence intervals, single-annotator ground truth without inter-annotator agreement, and the latency analysis is tangential. The paper is transparent about all of this, which is to its credit.\n\nThis deserves a serious referee. The pincer strategy is a reusable recipe, the artifacts are there, and the limitations are honestly reported. What it needs before publication is an out-of-distribution or unfiltered evaluation, even a small one, to show how the bounds degrade, plus an agreement check on the annotations. I would accept it with major revisions.","headline":"A solid, well-scoped neuro-symbolic method for notebook dataflow extraction; the headline numbers are real but measured on a curated subset, and the LLM prompts share the annotation rubric.","tokens_in":25641,"tokens_out":1671,"would_cite":true,"duration_ms":21687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRABS recovers a notebook's information flow graph at 98.57% F1 and its transitive cell execution dependencies at 99.67% F1, all without executing the notebook, by bounding the inter-cell I/O set syntactically and letting an LLM resolve…","keywords":["information flow graph","cell execution dependencies","Python notebooks","dataflow analysis","LLM code understanding","syntactic-semantic hybrid","zero-shot prompting","notebook understanding"],"falsifier":"Take 50 notebooks that violate at least one of CRABS's stated assumptions (e.g., notebooks containing %%capture magic commands, global variables read inside function bodies, or a variable renamed as a function after definition), hand-annotate their information flows, and check whether the syntactic lower estimate remains a subgraph and the upper estimate a supergraph of the ground truth. If any violation causes the bounds to fail to bracket the true flow set, or if the LLM resolution accuracy drops well below 98%, the paper's central boundedness and accuracy claims are falsified outside the curated regime.","tokens_in":24738,"feed_emoji":"🦀","tokens_out":9512,"duration_ms":92941,"temperature":0.7,"pith_summary":"CRABS aims to show that the information flows inside a Python notebook—which cells send which variables to which later cells—can be recovered accurately without executing the notebook or resolving its software dependencies. The paper proposes treating each cell's inter-cell inputs and outputs as a set that syntactic analysis can bracket from below and above, leaving only a small, well-scoped set of ambiguities for a large language model to resolve with simple yes/no questions. On 50 annotated notebooks, the strategy reaches 98.57% F1 for cell-to-cell information flows and 99.67% F1 for transitive execution dependencies, with exact graph matches for 74% and 82% of notebooks. The significance is that a cheap symbolic phase plus a focused semantic phase can outperform asking an LLM to read the whole notebook, while also eliminating hallucinated variables and long-context failures. The paper's claims are conditional on a curated evaluation set that respects the syntactic phase's assumptions.","feed_headline":"Pincer strategy reads notebook dataflow at 98.57% F1","feed_subtitle":"Syntactic bounds plus an LLM recover cell flows and execution dependencies without running the notebook","key_machinery":"The central object is the inter-cell I/O set: the set of all triples (cell, variable, in/out) recording which variables flow into and out of each cell. CRABS's load-bearing mechanism is the pair of lower and upper estimates of this set produced by shallow AST analysis—the lower estimate includes only flows that are syntactically certain, the upper estimate includes every syntactically possible flow, including in-place mutations, loop iterators, and shared references—so that the true I/O set is guaranteed to lie between them. The difference between the two estimates is exactly the ambiguity budget given to the LLM, which resolves each ambiguous member with a constrained zero-shot yes/no prompt rather than free-form description. This bounding-first, resolving-second design is what carries the argument: it converts an open-ended whole-notebook comprehension task into a finite set of small classification tasks.","core_discovery":"The central discovery is that the notebook understanding task—constructing an information flow graph and the induced cell execution dependency graph—can be solved to high accuracy, with F1 scores above 98%, by a two-phase pincer. The syntactic phase analyzes only the visible code in each cell to produce a lower estimate (flows that are certain) and an upper estimate (flows that are certain or possible) of the inter-cell I/O set, which bracket the ground truth as subgraph and supergraph respectively. The semantic phase then feeds the ambiguous members of this bracketed set to an LLM one cell at a time, asking only 'is this variable an input?' or 'is this an output candidate?'—binary questions whose answers are selected from the syntactic candidates, so the model cannot invent variables. The result is that 1397 of 1425 (98%) individual ambiguities are resolved correctly, and the final graphs match human annotations on 74–82% of notebooks. The paper frames this as evidence that a syntactic-semantic pincer, which bounds both the LLM's workload and the space of its answers, is a viable general strategy for neuro-symbolic code understanding.","pith_inferences":["The bounded-estimate idea could be lifted to other artifact types—SQL scripts, shell pipelines, and computational documents—where a cheap static analysis can delimit the candidate dataflow edges and a language model resolves the residue.","A natural stress test the paper does not run: apply CRABS to notebooks that violate its assumptions (magic commands, globals read inside functions, reused names) and measure how often the lower/upper bounds still bracket the ground truth; if the brackets break, the guarantee is an artifact of the curated sample.","Because the LLM resolves ambiguities independently per cell, the per-cell questions could be parallelized, and the paper's own latency appendix already points to a concurrent implementation being consistently faster than the baseline; a practical interactive notebook inspector seems within reach.","The 74% exact-match rate on flow graphs suggests that residual errors concentrate in a few hard patterns (likely in-place mutations and aliasing); specializing the resolution prompts toward those patterns, or adding a third syntactic pass for hidden modifications, could close the remaining gap."],"forward_implications":["If CRABS generalizes beyond the curated set, notebook comprehension tools can map data provenance without executing cells, enabling reuse and debugging of notebooks whose dependencies cannot be resolved.","The lower/upper estimates provide certified bounds on cell-to-cell flows for notebooks satisfying the stated assumptions, so downstream tools can know which flows are certain, which are possible, and which are excluded.","The cell-by-cell prompting strategy's 98% resolution accuracy suggests that LLM effort is best spent on many small, syntax-bounded questions rather than one large analysis.","The dependency graph derived from the flow graph reaches 99.67% F1, implying that transitive 'what affects what' queries about notebook cells become reliably answerable.","Ablations show that removing either pincer arm degrades performance substantially, indicating both syntactic bounding and cell-by-cell resolution are load-bearing, not incidental."],"supporting_citations":[{"why":"Quantifies that only one in four notebooks re-executes without error, motivating why execution-free understanding is needed.","marker":"[Pimentel et al., 2019]"},{"why":"Introduces the information-flow-graph model of scripts that CRABS adapts to notebook cells.","marker":"[McPhillips et al., 2015]"},{"why":"Supplies the dataset of 50 notebooks used for evaluation.","marker":"[Plotts & Risdal, 2023]"},{"why":"Provides kernel version IDs used to retrieve the evaluation notebooks.","marker":"[Risdal & Bozsolik, 2022]"},{"why":"The LLM model used in the semantic-aware phase to resolve ambiguities.","marker":"[OpenAI, 2024a]"},{"why":"Chain-of-thought prompting used to construct the whole-notebook baseline that CRABS is compared against.","marker":"[Wei et al., 2022]"}],"fun_headline_variants":["A syntactic pincer bounds LLM reading to 98% F1 on notebooks","CRABS uses AST bounds and zero-shot LLM to map notebook dataflow without execution","LLM resolves 98% of notebook I/O ambiguities after syntactic bounding","Without running code CRABS maps notebook flows at 98% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation set was deliberately curated so that every notebook satisfies CRABS's core assumptions—cells execute strictly top-to-bottom, variable and function names never clash, globals are not used inside functions unless passed in, and no non-Python (magic or shell) statements appear—so the guaranteed bounds and the reported 98% accuracy are only established for notebooks that meet these criteria.","fun_headline_variants_meta":{"raw":{"variants":["A syntactic pincer bounds LLM reading to 98% F1 on notebooks","CRABS uses AST bounds and zero-shot LLM to map notebook dataflow without execution","LLM resolves 98% of notebook I/O ambiguities after syntactic bounding","Without running code CRABS maps notebook flows at 98% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3637,"prompt_tokens":1098,"completion_tokens":2539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":2454}},"tokens_in":714,"tokens_out":2539,"duration_ms":23612,"temperature":1.0,"reasoning_tokens":2454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:02:50.880841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 50 notebooks that violate at least one of CRABS's stated assumptions (e.g., notebooks containing %%capture magic commands, global variables read inside function bodies, or a variable renamed as a function after definition), hand-annotate their information flows, and check whether the syntactic lower estimate remains a subgraph and the upper estimate a supergraph of the ground truth. If any violation causes the bounds to fail to bracket the true flow set, or if the LLM resolution accuracy drops well below 98%, the paper's central boundedness and accuracy claims are falsified outside the curated regime.","supporting_citations":[{"cited_title":"Meta kaggle code, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 50 notebooks used for evaluation."},{"cited_title":"Kyle Bocinsky, Yang Cao, James Cheney, Fernando Chirigati, Saumen Dey, Juliana Freire, Christopher Jones, James Hanken, Keith W","cited_arxiv_id":null,"evidence_quote":"Introduces the information-flow-graph model of scripts that CRABS adapts to notebook cells."},{"cited_title":"Meta kaggle, 2022","cited_arxiv_id":null,"evidence_quote":"Provides kernel version IDs used to retrieve the evaluation notebooks."},{"cited_title":"Le, and Denny Zhou","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting used to construct the whole-notebook baseline that CRABS is compared against."}],"review_version":1}