Snakemaker: Seamlessly transforming ad-hoc analyses into sustainable Snakemake workflows with generative AI

Alessandro Leone; Ivan Molineris; Johannes K\"oster; Marco Masera

REVIEW 3 major objections 3 minor 1 cited by

Snakemaker: Seamlessly transforming ad-hoc analyses into sustainable Snakemake workflows with generative AI

T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Snakemaker uses generative AI to convert recorded terminal commands and monolithic IPython notebooks into maintainable, validated Snakemake workflows without altering how researchers prototype.

desk verdict A genuinely useful-looking tool with an honest writeup, but the 'high-quality workflows' claim needs an evaluation before it will convince. read the letter →

arxiv 2505.02841 v1 pith:GU3R6VXR submitted 2025-04-26 cs.SE cs.AI

Marco Masera , Alessandro Leone , Johannes Köster , Ivan Molineris This is my paper · ORCID

classification cs.SEcs.AI

keywords generativeAISnakemakeworkflowgenerationreproducibilityIPythonnotebooksread/writesetanalysisbioinformaticsLLM-assistedcodetransformation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Snakemaker, a tool that uses large language models to convert recorded shell usage and IPython notebooks into Snakemake workflows. The aim is to lower the effort needed to turn throwaway prototype code into sustainable, reproducible pipelines, a transition bioinformatics often skips. Snakemaker claims to do this non-invasively, without changing how researchers prototype, while producing workflows that follow Snakemake best practices such as Conda environment tracking, wildcard-based generic rules, configuration extraction, and per-rule logging. If the approach works, it would give bioinformatics labs a practical bridge from ad-hoc scripts to maintainable pipelines.

What carries the argument

The central mechanism is a multi-pass LLM pipeline grounded in explicit file and variable dependencies: it extracts read/write sets from terminal commands or notebook cells, builds a DAG by connecting each read to the closest prior write, then drafts Snakemake rules with wildcards and loop unrolling, extracts configuration, post-processes with regex and small automata, and iteratively validates the workflow by running the Snakemake binary and feeding errors back to the model.

What would settle it

Run Snakemaker on a benchmark of notebooks and shell histories whose true file dependencies are known, and check whether the validated workflows recover exactly those dependencies; a single class of inputs where missing writes or hallucinated reads survive the correction loop and change the final outputs would falsify the central claim.

Watch

Extended reading notes

Core claim

The paper argues that unstructured, level-0 analyses—shell command histories and monolithic IPython notebooks—can be semi-automatically refactored into level-2 Snakemake workflows without the researcher changing how they work. For shell histories, Snakemaker records commands, return codes, and Conda environments, filters out incidental commands, and drafts generic rules that merge repeated commands and unroll loops. For notebooks, it unrolls the global execution state into a DAG of discrete file-based dependencies by estimating each cell's read and write sets, which is undecidable in general and therefore delegated to LLMs with a second pass to catch missing dependencies. The resulting workflows are validated iteratively against the Snakemake engine, with one or two correction loops reported as sufficient in most cases.

Load-bearing premise

The load-bearing premise is that, after iterative correction, the LLM reliably identifies which files and variables each command or cell reads and writes; if that fails, the dependency graph and the generated rules are wrong regardless of how clean the syntax is.

Editorial extensions

If this is right

Researchers can reconstruct reproducible workflows from past terminal sessions without having manually documented them.
Notebooks that currently need full re-execution can be split into modular scripts whose dependencies are explicit, so only affected steps rerun.
Generated Snakefiles are integrated into existing workflows, matching their style and avoiding duplicate rules.
The iterative validation loop means the final workflow is checked by the actual execution engine, not only by the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The core read/write-set unrolling is engine-agnostic: the same DAG construction could drive other file-based workflow systems, so the mechanism generalizes beyond Snakemake.
A structured benchmark with ground-truth dependencies across varied notebooks would be needed to confirm the correction loop converges; the paper reports qualitative experience, not quantitative accuracy.
The reported model-size dependence suggests that as open-weight models improve, local deployment becomes feasible, and adding Snakemake documentation to the prompt is a natural test of whether small models can close the gap.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing you should know: this is a tool paper that describes a clever combination—non-invasive terminal tracking, LLM-based rule generation, and notebook-to-DAG conversion—but it never measures whether the resulting workflows are actually correct or sustainable. The software is available on GitHub, the architecture is plausible, and the authors are candid about several failure modes. What it is not is an evidence-backed demonstration that Snakemaker reliably produces high-quality workflows.

What is new: I have not seen another tool that tracks your shell history in VSCode and converts it to Snakemake rules, nor one that unrolls a notebook's global state into a DAG using LLM-inferred read/write sets. The separate config-extraction pass and the regex/FSM post-processing are nice engineering touches. The paper also does something I appreciate: it openly admits where LLMs are shaky—read/write set inference is 'error-prone,' small models are 'near-unusable'—and it designs a human-in-the-loop review process around those limitations. The citation pattern is appropriate: Molder on sustainable Snakemake, Coelho on software levels, and the step-back prompting paper are all relevant.

Soft spots. The biggest one is the missing evaluation. The abstract and section 3 claim 'high-quality' workflows, adherence to best practices, and reliable command classification, but no numbers are given. No benchmark, no baseline against manual conversion, no user study, no error rates. The stress-test note you passed along is on point: the iterative validation through Snakemake can catch syntax errors and missing files, but it will not catch a wrong edge in the DAG when the upstream output happens to exist or the files are scheduled in a benign order. A second LLM pass looking at 'missing dependencies' is not a correctness guarantee. The paper's own Section 2.2 admits the problem, so this is not a hidden flaw—it is a known unquantified risk. I would also say the word 'seamlessly' in the title overpromises; the detailed GUI and manual-editing steps tell a more accurate story.

That said, the paper is not dishonest. It is a software description with clear limitations, and for a tools venue that has a place—if the code is usable and the claims are scoped. As written, the central claim crosses from 'we built this and here is how' into 'this works well,' and that latter part is unsupported. The fix is straightforward: run Snakemaker on a few real-world repositories, compare the generated DAGs against hand-crafted ones, and report correctness, user time, and how often the human had to fix things. If the authors supply that, I would be much more comfortable.

Who is this for: anyone building reproducibility tooling for bioinformatics, or a reviewer deciding whether to cite Snakemaker as a practical alternative to hand-writing Snakemake files. The paper deserves peer review, but it should be a conditional accept at best. I would not cite it for its empirical claims yet, only as a pointer to the tool.

Referee Report

3 major / 3 minor

Summary. The paper introduces Snakemaker, a VSCode extension that uses LLMs to convert ad-hoc analyses (terminal command histories and IPython notebooks) into Snakemake workflows. The system records terminal activity, feeds commands and metadata to an LLM to draft rules, separates configuration generation, applies regex/FSA post-processing, and performs iterative validation against the Snakemake binary. For notebooks, the system computes read/write sets per cell, builds a DAG of cell dependencies, and generates per-rule scripts with prefix/suffix code blocks. A chat assistant gives users natural-language control. The authors claim that Snakemaker generates high-quality workflows adhering to best practices such as Conda environment tracking, generic rule generation, and loop unrolling, lowering the barrier from prototype (level-0) to sustainable (level-2) software.

Significance. If the central claims were substantiated, Snakemaker would address a real and widely felt problem in computational biology: converting prototype code into maintainable, reproducible workflows. The architecture is described clearly and has credible, thoughtful components: human-in-the-loop DAG review, separation of configuration generation, finite-state post-processing, and explicit support for both bash and notebook inputs. The manuscript is honest about known limitations, including LLM error-proneness and the need for large models. The software is released under an open-source license with a public repository. However, the paper currently provides no quantitative evaluation: no benchmark dataset, no baseline comparison, no user study, and no measurement of workflow correctness. The central claim of generating 'high-quality' workflows therefore rests on anecdotal statements rather than evidence, and the strongest contribution of the work—its evaluation—is missing.

major comments (3)

[Section 3 (Results)] The Results section contains no controlled evaluation supporting the central claim of generating high-quality workflows. The sentence 'Tests with different LLMs show that they are reliable at distinguishing workflow-relevant commands from incidental ones' reports no dataset, no metrics, no baseline, and no error rates. Similarly, claims about optimal performance with specific model sizes are stated without experimental protocol. Because the paper's contribution is a tool whose core promise is correctness and reliability, the lack of any benchmark, case study, or user study leaves the main claim unsupported.
[Section 2.2 (Notebook conversion)] The notebook conversion pipeline depends critically on LLM-computed read and write sets, yet the manuscript states that 'identifying the read and write sets has proven to be an error-prone endeavor even for high performance models, with occasional missed writes or hallucinated reads.' The only mitigation is a second LLM pass that 'looks for errors in the sets explicitly related to the missing dependencies.' No quantitative evidence is provided that this second pass reliably recovers missed dependencies, nor is any measurement of the residual error rate after correction reported. Since the entire DAG construction depends on the accuracy of these sets, this is a load-bearing assumption that must be evaluated, for example on a labeled corpus of notebooks with known ground-truth dependencies.
[Section 2.1 (Shell tracking and conversion), step 5 (Validation)] The iterative validation step passes the workflow through the Snakemake binary, which can detect syntax errors and missing files only if they manifest at validation time. A missing dependency edge between rules may not produce an error if the upstream output already exists on disk or if rule scheduling happens to be benign. Thus the statement that 'one or two iterations are sufficient to fix all errors in most cases' does not establish semantic correctness of the generated DAG. The paper needs to define a concrete correctness criterion and evaluate how often the generated workflows actually capture the intended data dependencies, rather than relying on the Snakemake binary as a correctness oracle.

minor comments (3)

[Section 2.2] The phrase 'coats of paints approach' appears to be a typo; it should likely be 'coats of paint' or 'layers of paint'. Please correct this and proofread for similar language slips.
[Section 3] Figure 3 is included as a screenshot of the GUI but is never referenced in the text. Please add a reference to the figure in the Results section, or remove it if it is not needed.
[Section 3] The phrase 'direct acyclic graph' should be 'directed acyclic graph' in the first paragraph of the Results section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LLM outputs are validated against the external Snakemake binary and user review; the cited best-practice reference is not a load-bearing derivation input.

full rationale

The paper makes no derivational claim of the kind that can reduce to its own inputs. Snakemaker is an engineering tool that converts terminal histories and IPython notebooks into Snakemake workflows; the generated rules are checked by running them through the external Snakemake binary and by user inspection, not by a criterion defined in terms of the LLM's own output. The admitted fallibility of LLM read/write-set extraction in Section 2.2, with a second LLM pass looking for missing dependencies, is a self-correction loop rather than circular reasoning, because the final arbiter of workflow executability is the Snakemake runtime and the human user. The citation of Mölder et al. [3], authored partly by co-author Köster, is a normative reference to Snakemake best practices and does not supply a result whose conclusion is assumed in the premise. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no known result is relabeled. The absence of a quantitative benchmark for LLM accuracy is a soundness and validation weakness, not a circularity, and therefore does not raise the circularity score.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no mathematical derivation and no fitted parameters. Its central claim depends on the practical reliability of LLMs for code analysis and generation, plus the adequacy of terminal tracking. These are domain assumptions that the paper acknowledges but does not quantitatively validate, leaving the tool's effectiveness unproven.

assumptions (3)

domain assumption LLM-generated read/write sets and rule drafts are sufficiently accurate after iterative correction.
Section 2.2 admits errors and relies on a second LLM pass; no quantitative success rate is given.
domain assumption Terminal command history with return codes and active environments provides enough context to infer workflow-relevant steps.
Section 2.1 relies on filtering out one-off commands; the claimed reliability is not measured.
domain assumption VSCode extension APIs can non-invasively capture the required terminal and environment metadata.
Section 2.1 describes this as the foundation of the bash feature, but no technical validation is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Snakemaker: Seamlessly transforming ad-hoc analyses into sustainable Snakemake workflows with generative AI." pith.science (2026). https://pith.science/paper/GU3R6VXR

@misc{pith2026250502841,
  author       = {Pith},
  title        = {Pith review of: Snakemaker: Seamlessly transforming ad-hoc analyses into sustainable Snakemake workflows with generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GU3R6VXR}},
  note         = {Machine review of arXiv:2505.02841}
}

read the original abstract

Reproducibility and sustainability present significant challenges in bioinformatics software development, where rapidly evolving tools and complex workflows often result in short-lived or difficult-to-adapt pipelines. This paper introduces Snakemaker, a tool that leverages generative AI to facilitate researchers build sustainable data analysis pipelines by converting unstructured code into well-defined Snakemake workflows. Snakemaker non-invasively tracks the work performed in the terminal by the researcher, analyzes execution patterns, and generates Snakemake workflows that can be integrated into existing pipelines. Snakemaker also supports the transformation of monolithic Ipython Notebooks into modular Snakemake pipelines, resolving the global state of the notebook into discrete, file-based interactions between rules. An integrated chat assistant provides users with fine-grained control through natural language instructions. Snakemaker generates high-quality Snakemake workflows by adhering to the best practices, including Conda environment tracking, generic rule generation and loop unrolling. By lowering the barrier between prototype and production-quality code, Snakemaker addresses a critical gap in computational reproducibility for bioinformatics research.

Figures

Figures reproduced from arXiv: 2505.02841 by the authors.

**Figure 1.** Bash commands conversion pipeline. The current workflow is included in the model’s context (a) before generating the new rules (b). A second LLM pass extracts the config (c) and a post-processing step performs some fixes and merges rules with the previous workflow (d). The new workflow is fed to Snakemake (e.1); if errors arise (e.1b) they are either fed to the LLM for direct fixing (e.2b) or follow a two-step pipel… view at source ↗

**Figure 2.** Notebook conversion pipeline. The input notebook is parsed with regex and the LLM (a.1, a.2) to transform and annotate cells with read and write sets and cells’ states. The DAG is built from the annotated cells (b), and in case of errors a second LLM pass is performed (b.1). The user can modify the cells’ annotations (b.2) and consequently the DAG. The DAG is then fed to the LLM to generate additional code blocks an… view at source ↗

**Figure 3.** Screenshot from Snakemaker GUI. Bash conversion feature. Snakemaker allows to build Snakemake workflows from the recorded activity of the integrated bash terminal and from Ipython Notebooks [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Compendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python
q-bio.GN 2025-05 conditional novelty 4.0 of 10

Compendium Manager is a command-line orchestration layer that automates the launch, monitoring, and result collection of many Snakemake pipelines for large-scale bioinformatics meta-analyses.

Reference graph

Works this paper leans on

10 extracted references · 9 canonical work pages · cited by 1 Pith paper

[1]

Reproducible bioinformatics project: A community for reproducible bioinformatics analysis pipelines

Neha Kulkarni, Luca Alessandr `ı, Riccardo Panero, Maddalena Arigoni, Martina Olivero, Giulio Ferrero, Francesca Cordero, Marco Beccuti, and Raffaele Calogero. Reproducible bioinformatics project: A community for reproducible bioinformatics analysis pipelines. BMC Bioinformatics, 19:211–219, 10 2018

work page 2018
[2]

Ten simple rules for reproducible computational research

Geir Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten simple rules for reproducible computational research. PLoS computational biology, 9:e1003285, 10 2013

work page 2013
[3]

Sustainable data analysis with snakemake

Felix M¨older, Kim Jablonski, Brice Letcher, Michael Hall, Christopher Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven Twardziok, Alexander Kanitz, Andreas Wilm, Manuel Holtgrewe, Sven Rahmann, Sven Nahnsen, and Johannes K¨oster. Sustainable data analysis with snakemake. F1000Research, 10:33, 01 2021

work page 2021
[4]

For long-term sustainable software in bioinformatics

Luis Pedro Coelho. For long-term sustainable software in bioinformatics. PLOS Computational Biology , 20:e1011920, 03 2024

work page 2024
[5]

Improving bioinformatics software quality through teamwork

Katalin Ferenc, Ieva Rauluseviciute, Ladislav Hovan, Vipin Kumar, Marieke L Kuijjer, and Anthony Mathelier. Improving bioinformatics software quality through teamwork. Bioinformatics, 40(11):btae632, 10 2024

work page 2024
[6]

Le, and Denny Zhou

Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed Huai hsin Chi, Quoc V . Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models. ArXiv, abs/2310.06117, 2023. 8

arXiv 2023
[7]

A second step towards complexity-theoretic analogs of rice’s theorem

Lane Hemaspaandra and J¨org Rothe. A second step towards complexity-theoretic analogs of rice’s theorem. Theoretical Computer Science, 244:205–217, 08 1999

work page 1999
[8]

Whitepaper on Prompt Engineering by Google

Lee Boonstra. Whitepaper on Prompt Engineering by Google . Google, 2025

work page 2025

Show all 10 references

[9]

Text generation and prompting

OpenAI. Text generation and prompting. https://platform.openai.com/docs/guides/text
[10]

JSON Repair

Jos De Jong. JSON Repair. https://github.com/josdejong/jsonrepair

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Reproducible bioinformatics project: A community for reproducible bioinformatics analysis pipelines

Neha Kulkarni, Luca Alessandr `ı, Riccardo Panero, Maddalena Arigoni, Martina Olivero, Giulio Ferrero, Francesca Cordero, Marco Beccuti, and Raffaele Calogero. Reproducible bioinformatics project: A community for reproducible bioinformatics analysis pipelines. BMC Bioinformatics, 19:211–219, 10 2018

work page 2018

[2] [2]

Ten simple rules for reproducible computational research

Geir Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. Ten simple rules for reproducible computational research. PLoS computational biology, 9:e1003285, 10 2013

work page 2013

[3] [3]

Sustainable data analysis with snakemake

Felix M¨older, Kim Jablonski, Brice Letcher, Michael Hall, Christopher Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven Twardziok, Alexander Kanitz, Andreas Wilm, Manuel Holtgrewe, Sven Rahmann, Sven Nahnsen, and Johannes K¨oster. Sustainable data analysis with snakemake. F1000Research, 10:33, 01 2021

work page 2021

[4] [4]

For long-term sustainable software in bioinformatics

Luis Pedro Coelho. For long-term sustainable software in bioinformatics. PLOS Computational Biology , 20:e1011920, 03 2024

work page 2024

[5] [5]

Improving bioinformatics software quality through teamwork

Katalin Ferenc, Ieva Rauluseviciute, Ladislav Hovan, Vipin Kumar, Marieke L Kuijjer, and Anthony Mathelier. Improving bioinformatics software quality through teamwork. Bioinformatics, 40(11):btae632, 10 2024

work page 2024

[6] [6]

Le, and Denny Zhou

Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed Huai hsin Chi, Quoc V . Le, and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models. ArXiv, abs/2310.06117, 2023. 8

arXiv 2023

[7] [7]

A second step towards complexity-theoretic analogs of rice’s theorem

Lane Hemaspaandra and J¨org Rothe. A second step towards complexity-theoretic analogs of rice’s theorem. Theoretical Computer Science, 244:205–217, 08 1999

work page 1999

[8] [8]

Whitepaper on Prompt Engineering by Google

Lee Boonstra. Whitepaper on Prompt Engineering by Google . Google, 2025

work page 2025