{"id":"cf03d5a5-ce98-47f1-897e-7921b8529ba8","arxiv_id":"2507.20122","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A qualitative study of ten bioinformatics workflows finds LLMs can generate usable Galaxy and Nextflow pipelines, with Gemini best for Galaxy and DeepSeek-V3 best for Nextflow.","lead":"The authors tested three large language models, GPT-4o, Gemini 2.5 Flash, and DeepSeek-V3, on writing bioinformatics analysis pipelines for Galaxy and Nextflow. Gemini produced the clearest Galaxy workflows, DeepSeek produced the most implementation-ready Nextflow pipelines, and better prompts generally improved results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparative rankings rest on unblinded manual scoring plus an escalation design that gives failing models extra prompting chances; per-workflow narratives do not consistently support the aggregate rankings.","rationale":"The reader correctly identifies the subjectivity of manual scoring as the weakest assumption. My stress-test sharpens this into two concrete, load-bearing problems that go slightly beyond the reader's formulation. First, the tiered escalation design (§4.1) confounds model quality with prompt effort: models that fail early receive richer prompts, while models that produce plausible-but-wrong outputs are not escalated. This asymmetry can systematically inflate or deflate measured performance in a way that manual scoring alone would not capture. Second, the aggregate rankings (Gemini for Galaxy, DeepSeek for Nextflow) are not derivable from the paper's own per-workflow narratives, which credit different models on different Nextflow tasks (Gemini on W2, GPT-4o on W3 with role-based prompting, DeepSeek on W4/W5) without any quantitative weighting or stated decision rule. This makes the headline comparisons internally inconsistent, not merely subjective. The paper still provides useful qualitative evidence that LLMs can assist workflow development, and the successful execution of several generated workflows supports the broad assistive claim. Therefore the appropriate verdict remains CONDITIONAL: the paper should be published only with the model-level comparative claims explicitly downgraded to exploratory observations, or with a blinded, quantitative, pre-registered evaluation that settles whether the rankings hold. My agreement with the reader is partial because the reader focused on scoring subjectivity; I emphasize the escalation confound and the aggregation inconsistency as the more precise load-bearing weaknesses, which a concrete re-run with all prompt types and independent blinded raters would resolve.","tokens_in":39944,"tokens_out":2081,"duration_ms":25499,"concrete_test":"Reconstruct the evaluation with a fixed protocol: (1) run every model on every workflow with all three prompt types (instruction-only, role-based, CoT), not just until the authors deem an output acceptable; (2) have at least three independent raters, blinded to model identity, score each output with a pre-registered quantitative rubric (e.g., number of required steps present, tool availability in Galaxy/nf-core, executability, parameter correctness); (3) report inter-rater agreement (Cohen's kappa) and rank models by mean score per platform. If DeepSeek no longer ranks first for Nextflow, or Gemini no longer ranks first for Galaxy, under the blinded quantitative rubric, the paper's headline comparative claims are not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central comparative claims (Gemini 2.5 Flash best for Galaxy, DeepSeek-V3 best for Nextflow) are not supported by a reproducible, internally consistent evaluation. Two issues are load-bearing. First, the tiered prompting strategy in §4.1 (instruction-only → role-based → chain-of-thought) creates an asymmetry: a model whose initial output is judged incomplete gets additional, more detailed prompts, while a model that produces a superficially complete but subtly wrong workflow is never escalated and is scored as a success. Because the escalation decision is itself a subjective expert judgment (§6: 'our criteria for escalation may still carry implicit judgment calls'), measured 'capability' is confounded with the number of prompting attempts the authors chose to invest in each model. Second, the aggregate rankings do not follow from the per-workflow narratives. For Nextflow, §4.1.2 reports that Gemini 'outperforms the others' on W2 (fetchngs), that GPT-4o 'produces the most technically precise and nf-core-compliant workflow' on W3 under role-based prompting, and that all three models succeed on W5, with GPT-4o 'highest fidelity' and DeepSeek 'most comprehensive and adaptable.' Yet the summary in §4.1.2 and RQ2 declare DeepSeek 'the most capable model' for Nextflow with no quantitative aggregation or stated weighting rule. The same pattern holds for Galaxy, where Gemini is declared best even though GPT-4o and DeepSeek also produce executable workflows on four of five tasks and the differences are described in qualitative terms like 'stands out' and 'more comprehensive.' Given that the authors are also the sole raters, with no inter-rater reliability metric and no blinded protocol (both acknowledged in §6), the model-level conclusions are vulnerable to confirmation bias and cannot be distinguished from prompt-effort artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an exploratory study of three LLMs (GPT-4o, Gemini 2.5 Flash, DeepSeek-V3) for generating bioinformatics workflows in Galaxy and Nextflow. The authors select ten community-curated workflows (five per platform), design tiered prompts (instruction-only, role-based, chain-of-thought) with escalation when initial outputs are judged inadequate, and manually assess the generated workflows for completeness, correctness, and usability against GTN and nf-core baselines. The main claims are that LLMs can substantially assist workflow development, that Gemini 2.5 Flash produces the most accurate and usable Galaxy workflows, that DeepSeek-V3 excels at Nextflow pipeline generation, and that prompt design materially affects output quality. The paper includes qualitative per-workflow narratives and comparative tables, and makes all prompts and outputs available via Zenodo.","tokens_in":40189,"tokens_out":7224,"duration_ms":65070,"significance":"The study addresses a timely question and has a sensible object of study: real, community-maintained workflows rather than synthetic tasks. Its strengths include the use of GTN and nf-core baselines, a tiered prompting protocol that mirrors how non-expert users actually interact with LLMs, and a candid discussion of threats to validity in Section 6. If the rankings were robust, the paper would provide useful guidance on model and prompt selection for bioinformatics workflow generation. However, the evidence is currently qualitative, unblinded, and in places internally inconsistent; the comparative conclusions are therefore not yet reproducible. The contribution at this stage is best characterized as an exploratory qualitative account with actionable prompt-engineering patterns.","major_comments":[{"comment":"The tiered prompting protocol confounds model capability with the number and content of prompting attempts. Escalation is triggered by the authors' subjective judgment that an output is 'incomplete, incorrect, or misaligned with the baseline' (§3, 'To address RQ2'); a model whose initial draft is judged complete is never escalated, while a model whose initial draft is judged deficient receives additional, richer prompts. Section 6 concedes that 'our criteria for escalation may still carry implicit judgment calls.' Because RQ2 compares the 'completeness, correctness, and usability' of outputs across models, the protocol does not compare like with like: a model that produces a plausible but subtly wrong workflow can be scored as a success without any opportunity to correct it, while another model's measured performance reflects the authors' investment in additional prompting. The authors should either (i) compare models on instruction-only outputs only, (ii) report the number of escalation rounds per model/workflow as a separate usability metric, or (iii) define pre-specified, observable escalation criteria so that the decision rule is reproducible.","section":"§3 (Study Design) and §6 (Threats to Validity)"},{"comment":"The aggregate rankings do not follow from the per-workflow narratives. For Nextflow, the W2 narrative states that Gemini 2.5 Flash 'outperforms the others' (fetchngs), the W3 narrative concludes that GPT-4o produces 'the most technically precise and nf-core-compliant workflow' (under role-based prompting), and the W5 narrative says GPT-4o 'ensures the highest fidelity' while DeepSeek-V3 is 'most comprehensive and adaptable.' Yet the summary in §4.1.2 and the RQ2 answer declare DeepSeek-V3 'the most capable model' for Nextflow with no stated weighting or aggregation rule. For Galaxy, the W4 narrative identifies DeepSeek-V3 as producing 'the most comprehensive and customizable workflow,' while the §4.1.1 summary says Gemini 2.5 Flash 'consistently performs best.' Without a defined scoring rubric or an explicit qualitative weighting rationale, these summary rankings are not derivable from the reported evidence. Please either provide per-workflow scores and an aggregation rule, or restrict the conclusions to per-workflow observations.","section":"§4.1.2 and §4.2"},{"comment":"The evaluation lacks quantitative measurement and inter-rater reliability. Section 4.2 says the workflows 'are then manually assessed by two domain experts,' but no numeric scores, rubric, or agreement statistic are reported; the comparison tables (Tables 8–11) use qualitative checkmarks and symbols. Section 6 itself acknowledges that 'human interpretation is inherently susceptible to bias' and that 'the absence of quantitative metrics for some usability aspects introduces potential inconsistencies.' Because every comparative claim in RQ2 rests on these manual judgments, the rankings are not reproducible. I recommend adding a defined scoring rubric, reporting per-workflow scores for completeness/correctness/usability, and computing inter-rater agreement (e.g., Cohen's kappa or percentage agreement on at least a subset of workflows). If the authors prefer to keep the study qualitative, the comparative rankings should be explicitly reframed as exploratory observations rather than findings.","section":"§4.2 and §6"},{"comment":"The executability evidence is incomplete for several of the central comparisons. For Galaxy W5, the paper states that GPT-4o's instruction-only workflow 'included tools not available within the Galaxy platform, such as ISMapper, MobileElementFinder, and Trackster,' so the authors 'are unable to develop the workflow'; DeepSeek-V3's W5 response similarly proposes Infernal, which is not available in Galaxy. For Nextflow, several narratives (e.g., DeepSeek-V3 on W3) say the generated steps 'are correct, and we can develop the workflow following the instructions,' but no execution logs, error reports, or tool-availability checks are provided in the manuscript or the supplementary description. Since the abstract and RQ1 claim 'executable' workflows, the paper should report per-workflow execution status (executed successfully, executed with modifications, not executable) with evidence, or qualify the executability claims accordingly.","section":"§4.1.1 (Galaxy W5) and §4.1.2 (Nextflow)"}],"minor_comments":[{"comment":"The text contains a duplicated bracket in 'grounded in recent literature on prompt engineering and LLM behavior [e.g., [28, 29, ...]'; this citation formatting should be fixed.","section":"§4.2.1"},{"comment":"The Galaxy Training Network reference appears twice as [35] and [80]; these should be merged into a single citation.","section":"References"},{"comment":"The tables contain a garbled symbol rendered as '/exclamati⌢n-triangle'; the intended warning glyph should be typeset correctly.","section":"Tables 8–11"},{"comment":"The name of the DeepSeek model is inconsistent ('DeepSeek-V3' vs 'DeepSeek-v3'); please use one spelling consistently.","section":"Throughout"},{"comment":"The statement that 'GPT-4o performed nicely with structured prompts' is vague and does not match the level of detail in the results; consider replacing it with a concrete summary of GPT-4o's observed behavior.","section":"Abstract"},{"comment":"The text says a 'consistent prompting strategy is applied across all workflows,' but Tables 6 and 7 show role-based and chain-of-thought prompts only for a subset; clarify that these were used upon escalation, and consider reporting which prompt tier each model/workflow reached.","section":"§3 (Study Design)"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its limitations, which is commendable. The main risk is that the title, abstract, and RQ2 conclusions claim comparative rankings that the current evidence cannot support. If the authors cannot add quantitative scoring within a revision, the journal may wish to consider whether a qualitative-exploration framing is acceptable for its readership; otherwise I would treat the rankings as the main obstacle to acceptance. I also note that the authors' prior work on Galaxy reusability ([16], [20]) is cited appropriately and does not create circularity; the novel contribution here is the LLM evaluation itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading if you care about LLM-assisted workflow generation. The paper is a systematic qualitative comparison of GPT-4o, Gemini 2.5 Flash, and DeepSeek-V3 on ten bioinformatics workflows across Galaxy and Nextflow, with prompts and outputs in a Zenodo supplement and execution against GTN and nf-core baselines. The benchmark design is sensible and the per-workflow write-ups are detailed. The authors deserve credit for the candid threats-to-validity section, which admits manual scoring is subjective, escalation criteria involve implicit judgment calls, and some usability aspects have no quantitative metric.\n\nThe soft spot is the mismatch between the narratives and the aggregate conclusions. For Nextflow, the text says Gemini outperforms the others on fetchngs, GPT-4o produces the most technically precise nf-core-compliant workflow on methylseq under role-based prompting, and on phyloplace GPT-4o has highest fidelity while DeepSeek is most comprehensive and adaptable. Yet RQ2 declares DeepSeek the most capable model for Nextflow with no stated aggregation rule. For Galaxy, DeepSeek is described as producing the most comprehensive QC workflow on W4, but Gemini is declared the consistent winner. The rankings do not follow from the reported evidence.\n\nThe tiered-prompting design adds a second issue. Escalation from instruction-only to role-based to chain-of-thought happens only when the authors judge the initial output incomplete or incorrect. A model that emits a superficially complete but subtly wrong workflow is never escalated and is scored as a success, while a model that visibly struggles gets extra prompting chances. That confounds capability with prompt effort. The authors acknowledge the escalation decisions are judgment calls.\n\nThese flaws do not sink the qualitative exploration. The broad claim that LLMs can assist workflow development is supported. But the specific model rankings should be treated as hypotheses, not results. A revision needs a numeric rubric, inter-rater reliability, a blinded protocol, and either a fixed escalation rule or a separate comparison at each prompting level. If that lands, this becomes a useful reference for practitioners choosing models and prompts.\n\nI would not desk reject this. The benchmark design and supplementary materials are worth refereeing, and the problems are fixable. I would not cite the current rankings, but I would keep an eye on the dataset after revision.","headline":"Solid qualitative evaluation with honest limitations, but the headline rankings don't follow from the per-workflow narrative; treat the model comparisons as testable hypotheses.","tokens_in":40799,"tokens_out":3703,"would_cite":false,"duration_ms":34387,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can turn plain-language prompts into usable bioinformatics workflows.","keywords":["scientific workflows","large language models","bioinformatics","prompt engineering","Galaxy","Nextflow","nf-core","workflow generation"],"falsifier":"Re-run the same ten workflows with an independent panel of bioinformatics workflow experts who score the outputs blind, and check whether Gemini still wins on Galaxy and DeepSeek-V3 on Nextflow; additionally attempt an end-to-end execution of every 'usable' generated workflow in a fresh Galaxy instance and a clean Nextflow installation to see whether they run without hand repair.","tokens_in":39707,"feed_emoji":"🧬","tokens_out":5778,"duration_ms":52523,"temperature":0.7,"pith_summary":"This paper asks whether large language models can move from answering questions about bioinformatics to actually generating complete, executable scientific workflows. It claims the answer is yes: across ten representative Galaxy and Nextflow tasks, all three tested models produced logically correct workflows from natural-language prompts, with Gemini 2.5 Flash strongest in the graphical Galaxy environment and DeepSeek-V3 strongest for script-based Nextflow pipelines. The paper further claims that escalating prompts from plain instructions to role-based and chain-of-thought formats measurably improves completeness and correctness. If these results hold, researchers who do not program could hand a task description to an LLM and receive a workflow that follows community-established conventions, lowering the entry barrier to reproducible analysis.","feed_headline":"From prompt to pipeline: LLMs build working bioinformatics workflows","feed_subtitle":"Gemini leads on Galaxy, DeepSeek-V3 on Nextflow, and role-based prompts lift quality in a ten-workflow benchmark.","key_machinery":"The load-bearing mechanism is a three-tier prompting escalation applied uniformly across models and tasks: each workflow begins with an instruction-only prompt; if the output is incomplete or incorrect, the model is re-prompted as a domain expert (role-based), and finally with a chain-of-thought prompt that asks for step-by-step planning and justification. Quality is then judged by two domain experts on completeness, correctness, and usability against GTN and nf-core baselines. This tiered design is what lets the paper attribute differences in output quality to model choice and prompt design rather than to task difficulty.","core_discovery":"On its own terms, the paper establishes that state-of-the-art LLMs can act as workflow co-developers in bioinformatics, generating outputs that align with community-curated baselines from the Galaxy Training Network and nf-core. Gemini 2.5 Flash consistently performs best for Galaxy, producing complete, tool-aware, executable workflows with minimal prompting; DeepSeek-V3 emerges as the most capable model for Nextflow, generating implementation-ready, nf-core-aligned pipelines with configuration and container details. GPT-4o is described as performing well when given structured, role-based prompts but tends to omit setup steps or recommend unavailable tools when prompted minimally. Across the ten workflows, role-based and chain-of-thought prompting improved correctness and completeness compared with instruction-only prompts.","pith_inferences":["A direct test of the accessibility claim would be a user study with novices who have no Galaxy or Nextflow training, measuring time to first successful execution of an LLM-generated workflow; the paper reports expert judgment, not end-user behaviour.","Because the winning models are described as of early 2025, the platform-specific rankings are likely unstable as model versions update; the methodology, not the winner list, is the durable contribution.","The tiered escalation could be automated into an agent loop that checks executability and re-prompts until success, turning the manual ladder into a quantitative search over prompts.","Quantifying inter-rater agreement between the two evaluators, alongside execution pass rates, would let other groups reproduce the comparison on the same ten workflows."],"forward_implications":["Novice users could generate a first-draft Galaxy workflow from a single paragraph describing their analysis goal, then refine it with role-based follow-ups.","Model selection becomes platform-dependent: Gemini for graphical Galaxy, DeepSeek-V3 for script-based Nextflow, and GPT-4o when users can supply structured prompts.","Workflow developers should expect to escalate prompts; the paper's tiered strategy offers a concrete fallback ladder from instruction-only to chain-of-thought.","LLM-generated workflows still need human review because models hallucinate tool availability, so the practical role is assistive co-development rather than full autonomy.","Community-curated workflows remain the benchmark ground truth, which opens a low-cost way to pre-screen candidate pipelines before investing expert effort."],"supporting_citations":[{"why":"Prior qualitative assessment of ChatGPT for workflow design; the study this paper extends to multiple models and platforms.","marker":"[3]"},{"why":"System card for GPT-4o, one of the three evaluated models.","marker":"[23]"},{"why":"Report defining Gemini 2.5 Flash, the model the paper finds best for Galaxy workflows.","marker":"[24]"},{"why":"Technical report defining DeepSeek-V3, the model the paper finds best for Nextflow pipelines.","marker":"[25]"},{"why":"Prompt-engineering guide that motivates the tiered instruction/role/chain-of-thought strategy.","marker":"[28]"},{"why":"Chain-of-thought prompting paper that supplies the third escalation tier.","marker":"[31]"},{"why":"Galaxy Training Network, source of the community-curated Galaxy baselines used for comparison.","marker":"[35]"},{"why":"nf-core framework, source of the Nextflow baseline pipelines used for comparison.","marker":"[36]"}],"fun_headline_variants":["Gemini leads on Galaxy, DeepSeek-V3 on Nextflow for LLM workflows","Role-based prompts sharpen LLM-built bioinformatics pipelines","LLMs build runnable bioinformatics workflows with the right prompts","LLMs turn prompts into executable bioinformatics pipelines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings rest on the two authors' manual, non-quantified expert scoring of completeness, correctness, and usability; if a different expert panel scored the same outputs, the model winners could change.","fun_headline_variants_meta":{"raw":{"variants":["Gemini leads on Galaxy, DeepSeek-V3 on Nextflow for LLM workflows","Role-based prompts sharpen LLM-built bioinformatics pipelines","LLMs build runnable bioinformatics workflows with the right prompts","LLMs turn prompts into executable bioinformatics pipelines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00118,"raw_usage":{"total_tokens":4913,"prompt_tokens":1020,"completion_tokens":3893,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":3820}},"tokens_in":636,"tokens_out":3893,"duration_ms":29664,"temperature":1.0,"reasoning_tokens":3820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:48:17.340782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same ten workflows with an independent panel of bioinformatics workflow experts who score the outputs blind, and check whether Gemini still wins on Galaxy and DeepSeek-V3 on Nextflow; additionally attempt an end-to-end execution of every 'usable' generated workflow in a fresh Galaxy instance and a clean Nextflow installation to see whether they run without hand repair.","supporting_citations":[{"cited_title":"Ekin, Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices, Authorea Preprints (2023)","cited_arxiv_id":null,"evidence_quote":"Prompt-engineering guide that motivates the tiered instruction/role/chain-of-thought strategy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting paper that supplies the third escalation tier."},{"cited_title":"Hiltemann, et al., Galaxy training: A powerful framework for teaching!, PLoS compu- tational biology 19 (1) (2023) e1010752","cited_arxiv_id":null,"evidence_quote":"Galaxy Training Network, source of the community-curated Galaxy baselines used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"nf-core framework, source of the Nextflow baseline pipelines used for comparison."}],"review_version":1}