{"id":"d4494870-061f-4b8d-a20c-a3025352fdb0","arxiv_id":"2411.10890","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A topic-modeling analysis of Stack Overflow and GitHub data identifies the most common and most difficult challenges in scientific workflow system development, with workflow execution hardest on Stack Overflow.","lead":"We analyzed 35,619 Stack Overflow posts and 163,118 GitHub issues and pull requests about scientific workflow systems, using topic modeling to map the challenges developers discuss. The resulting topic and difficulty map can guide where to improve workflow tools, documentation, and training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GitHub bot-authored PRs are counted as developer challenges; rerunning Table 17 after bot filtering could change the difficulty ranking and topic set.","rationale":"The central claim depends on two empirical rankings: Workflow Execution as most difficult on Stack Overflow, and System Redesign/API Migration as most difficult on GitHub. The SO side is comparatively well supported: data collection is manually vetted for ambiguous names, question-type annotation reaches kappa 0.82, and Table 15's ranking is internally consistent with the quoted examples. The GitHub side is the vulnerable leg. The paper's own topic labels and representative examples show bot-generated PRs are in the dataset: 'precommitci autoupdate', 'Auto Compress Images by Calibre's image-actions', and 'Automated Performance and Tool Update Integration'. These records are not expressions of human developer challenges, yet BERTopic will cluster them because they share templated vocabulary, and Table 17's difficulty metrics will count them as easy, resolved items. The replication pipeline in Section 3.2 removes PRs linked to issues but has no bot-author filter. A concrete rerun with bot accounts excluded would settle whether the GitHub topic set and the 117.95 h ranking survive. The reader's weakest assumption identifies exactly this confound, and the recommended conditional verdict stands, so I keep the verdict unchanged. I would add one editorial note: the abstract's sentence claiming 'data structures and operations is the most difficult' on GitHub contradicts Table 17, where System Redesign and API Migration has the highest median resolution time; this should be corrected regardless of the bot-filter outcome.","tokens_in":41765,"tokens_out":7274,"duration_ms":70785,"concrete_test":"Use the GitHub REST API (or the replication package data) to identify all issue/PR authors whose login ends in '[bot]' or matches known bot accounts (e.g., pre-commit-ci[bot], calibreapp-image-actions[bot], dependabot[bot], renovate[bot], github-actions[bot]); remove their records from the 162,169-item GitHub dataset; rerun BERTopic with the same hyperparameters and recompute Tables 10 and 17 (unresolved percentage and median resolution time). If System Redesign and API Migration no longer has the highest median resolution time, or if bot-dominated topics disappear or shrink by more than 10% of their posts, the GitHub difficulty ranking is an artifact of automated activity. If the ranking and the human-only topic set remain stable, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The GitHub-side conclusions rest on treating every issue and pull request as a human-raised challenge. That assumption is not secure. Table 10 contains topics that are essentially bot templates: 'Automated Performance and Tool Update Integration' (0.29% unresolved, median 0.10 h in Table 17), 'Image Compression & Optimization' (0.0% unresolved, 0.97 h), and 'Automating Code Quality Checks' (0.01% unresolved, 16.63 h). These correspond to recurring PRs such as 'precommitci autoupdate' and 'Auto Compress Images by Calibre's image-actions', which the paper itself mentions in Section 4.1 without treating them as a confound. Since BERTopic clusters on text, bot-generated PRs create coherent but non-human topics, and their near-zero unresolved rates and short resolution times distort the difficulty distribution. The Section 3.2 pipeline filters PRs linked to issues but never filters bot accounts. Consequently, the claim that 'System Redesign and API Migration' is the most challenging GitHub topic (117.95 h) is not yet established: the topic model and the Table 17 metrics are computed over a mixture of human and automated activity. If bot traffic is substantial, the GitHub topic set and difficulty rankings do not represent genuine developer challenges.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical study of developer challenges in Scientific Workflow Systems (SWSs) by mining 35,619 Stack Overflow posts and 163,118 GitHub issues and pull requests related to eleven SWSs. The authors apply BERTopic after manual filtering and preprocessing, identify 10 SO topics and 13 GitHub topics, classify SO questions into How/Why/What/Other types, and measure topic difficulty using the percentage of unanswered/unresolved items and median times to answer/resolution. The main reported findings are that Workflow Execution is the most challenging SO topic (68.50% unanswered, 23.58 hours), while System Redesign and API Migration has the highest median GitHub resolution time (117.95 hours); the paper also compares SWSs with other software engineering domains and draws implications for practitioners, educators, and researchers.","tokens_in":41969,"tokens_out":2935,"duration_ms":32263,"significance":"If the results withstand scrutiny, the paper provides a useful, community-grounded map of SWS development pain points and a large, reusable dataset plus replication package. The authors give credit for the careful manual filtering of ambiguous SO tags (e.g., the Galaxy case), the inter-annotator agreement (Cohen's kappa 0.82) on question-type classification, and the explicit reporting of BERTopic hyperparameters and coherence scores. The cross-platform comparison between Stack Overflow and GitHub is a methodological strength that goes beyond many single-platform topic-mining studies. However, the GitHub-side analysis currently contains a potentially load-bearing confound because bot-generated pull requests appear to be included in the topic model and difficulty metrics.","major_comments":[{"comment":"The GitHub analysis does not filter bot-authored issues and pull requests, and this appears to distort the topic set and difficulty rankings. Topics such as 'Automated Performance and Tool Update Integration', 'Image Compression & Optimization', and 'Automating Code Quality Checks' in Table 10 correspond to recurring automated PRs (e.g., 'precommitci autoupdate' and 'Auto Compress Images by Calibre's image-actions'), which the paper itself mentions in §4.1 without treating them as a confound. Table 17 shows that these topics have near-zero unresolved rates and very short median resolution times (0.1 h, 0.97 h, and 16.63 h), which is exactly what bot-generated PRs would produce. Since no bot-account filtering is described in §3.2, the GitHub topic model and the Table 17 difficulty metrics are computed over a mixture of human and automated activity; the claim that 'System Redesign and API Migration' is the most challenging GitHub topic (117.95 h) is therefore not yet established. The authors should rerun the GitHub pipeline after filtering bot accounts, or at least quantify the proportion of bot traffic and show that the topic structure and difficulty rankings are robust to its removal.","section":"§3.2, §4.1, Table 10, Table 17"},{"comment":"The abstract states that the GitHub analysis 'discovered that data structures and operations is the most difficult', but this is not supported by any GitHub result in the paper: Table 10 does not contain a GitHub topic named 'Data Structures and Operations', and Table 17 reports System Redesign and API Migration as having the highest median resolution time (117.95 hours) while Browser Compatibility and HDFS Integration Issues have the highest unresolved rate (25.0%). This inconsistency concerns the paper's central summary of its own findings and must be resolved by aligning the abstract with the reported results.","section":"Abstract and §4.3/Table 17"},{"comment":"The use of 'open state' as the sole indicator of an unresolved GitHub issue or pull request, combined with the bot contamination, may also conflate maintenance workflow with difficulty. For example, topics with very low unresolved rates (Dependencies 0.02%, Managing Releases 0.38%, Automating Code Quality Checks 0.01%) could reflect PRs that are automatically closed or merged without representing genuine developer challenges. The paper should report how many items in each topic are PRs versus issues, how many are authored by known bot accounts, and how the difficulty metrics change when only human-authored, issue-type items are considered.","section":"§4.3, Table 17"}],"minor_comments":[{"comment":"The description for topic 9, 'Chord Execution and Task Coordination Issues', says 'Improving code quality through refactoring, and error fixes', which does not match the topic label or its keywords; this appears to be a copy-paste error and should be corrected.","section":"Table 10"},{"comment":"The evolution claims in Figures 6-8 are based on raw counts over time, which can be influenced by the overall growth of Stack Overflow and GitHub; the paper should either normalize by platform-wide activity or explicitly acknowledge this threat when interpreting the 'divergence' between SO and GitHub.","section":"§5.1"},{"comment":"There is a typo in the column header 'Agv Score'; it should read 'Avg Score'.","section":"Table 18"},{"comment":"The description of the celery filtering process is confusing: the text reports 9,499 tagged posts, then mentions 139 remaining posts and manual scrutiny, but the arithmetic leading from the initial 9,628 posts to these intermediate numbers is not clearly explained; adding a small summary table or explicit counts for each filtering step would improve reproducibility.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of empirical software engineering venues and is generally well-structured, but the GitHub-side conclusions currently rest on an unfiltered mixture of human and automated activity. The abstract inconsistency is easy to fix, but the bot filtering issue requires rerunning the analysis, so I recommend major revision rather than rejection; the SO-side analysis and the replication package are valuable and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful first map of SWS developer discussions, and the Stack Overflow side largely delivers. The GitHub side does not, because bot-authored PRs are mixed into the topic model and the difficulty metrics.\n\nWhat is actually new here: the first large-scale topic map for scientific workflow systems, with 10 Stack Overflow topics, 13 GitHub topics, difficulty rankings, a cross-platform comparison, and a question-type distribution. The data work is real: 35,619 SO posts, over 162,000 GitHub issues and PRs, careful manual filtering for ambiguous names like Galaxy, high annotator agreement (kappa 0.82), and a replication package. The topic labels look plausible, and the BERTopic setup is sensible.\n\nThe weak spot is load-bearing: the GitHub analysis counts bot-generated PRs as human developer challenges. Table 10 contains topics like “Automated Performance and Tool Update Integration,” “Image Compression & Optimization,” and “Automating Code Quality Checks,” which are essentially bot templates. The paper itself quotes recurring patterns like “precommit autoupdate update GitHub actions” and “Auto Compress Images by Calibre” without treating them as a confound. Those topics then show near-zero unresolved rates and sub-hour median resolution times in Table 17, so the difficulty distribution is partly measuring automation, not human effort. The claim that System Redesign and API Migration is the most challenging GitHub topic (117.95 hours) might survive a re-analysis, but it is not established from a mixture of human and automated activity. Bot filtering and re-running the topic model are needed before the GitHub-side conclusions are credible.\n\nOther concerns are minor. The unresolved-rate metric conflates “no accepted answer” with “difficult,” but that is a standard proxy in this literature. The selected SWS list is dominated by Airflow, Celery, Dask, and Galaxy, so the findings skew to those ecosystems. Neither issue undercuts the Stack Overflow results.\n\nWho is this for? Researchers working on scientific workflow tooling, empirical software engineering, and developer-support analytics. The paper is honest and methodical, and the SO findings alone justify a hearing. Send it to peer review, but a serious referee should require a bot-filtered GitHub re-analysis before accepting the difficulty rankings.","headline":"Useful first topic map for SWS developer discussions, but GitHub bot traffic contaminates the difficulty rankings and needs fixing before the GitHub conclusions can be trusted.","tokens_in":42530,"tokens_out":2989,"would_cite":false,"duration_ms":33106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Scientific workflow systems' hardest topic is workflow execution, with 68.5% of Stack Overflow posts unanswered.","keywords":["scientific workflow systems","Stack Overflow","GitHub","topic modeling","BERTopic","developer challenges","empirical software engineering"],"falsifier":"Re-run the GitHub analysis after removing activity from automated bot accounts (e.g., 'precommitci autoupdate' and 'Auto Compress Images by Calibre' PRs) and check whether the near-zero unresolved rates and sub-hour resolution times for those topics persist, and whether System Redesign and API Migration still has the longest median resolution time.","tokens_in":41527,"feed_emoji":"⚙️","tokens_out":6367,"duration_ms":59719,"temperature":0.7,"pith_summary":"Scientific workflow systems orchestrate complex computational pipelines, but what developers find hardest to build and maintain is not well documented. This paper mines 35,619 Stack Overflow posts and 163,118 GitHub issues and pull requests with BERTopic topic modeling to map the field's pain points. Its central finding is a ranked list of challenges: on Stack Overflow, workflow execution stands out as the most difficult topic, with 68.50% of posts unanswered and a median time to an accepted answer of 23.58 hours; on GitHub, system redesign and API migration takes the longest to resolve, at a median of 117.95 hours. The paper also finds that How-type questions dominate across all topics, pointing to a broad need for procedural guidance rather than conceptual documentation. If the rankings hold, they give tool builders, maintainers, and educators a concrete priority list for where SWS support is most lacking.","feed_headline":"Workflow execution tops hardest developer questions","feed_subtitle":"Mining 35k Stack Overflow posts and 163k GitHub issues ranks the pain points.","key_machinery":"The argument is carried by a BERTopic pipeline—a transformer-based topic-modeling method that embeds documents with a sentence-transformer model (multi-qa-MiniLM-L6-dot-v1), reduces dimensions with UMAP, clusters with HDBSCAN, and labels topics from count-vectorized keywords—combined with two difficulty metrics borrowed from prior Stack Overflow studies: the percentage of posts or issues without an accepted answer or resolution, and the median time to answer or resolve. The same pipeline is applied to GitHub issue and pull-request text. A complementary manual step classifies a statistically sampled set of 2,933 Stack Overflow posts into How, Why, What, and Other question types (Cohen's kappa 0.82), and fine-grained second-pass BERTopic runs decompose the largest topics into subtopics, which is how the paper operationalizes 'challenge' as something measurable and rankable.","core_discovery":"The paper claims that mining Stack Overflow and GitHub with BERTopic reveals the principal developer challenges in scientific workflow system development: ten topics on Stack Overflow (workflow creation and scheduling, distributed task management, workflow execution, data structures and operations, and others) and thirteen on GitHub (errors and bug fixing, documentation, dependency management, system redesign and API migration, and others). Using two difficulty metrics—the percentage of posts or issues without an accepted answer or resolution, and the median time to answer or resolve—the paper finds that workflow execution is the most challenging topic on Stack Overflow (68.50% unanswered, 23.58 hours median) and that system redesign and API migration has the longest median resolution time on GitHub (117.95 hours), while browser compatibility and HDFS integration has the highest unresolved rate (25.0%). The paper also reports that How-type questions dominate across all topics (60.97% on average), indicating a need for procedural guidance, and that several topics—data structures and operations, task management, and workflow scheduling—recur on both platforms.","pith_inferences":["The near-zero unresolved rates and sub-hour resolution times reported for automated GitHub topics (e.g., automated tool updates, image compression) are likely an artifact of bot-generated pull requests; excluding bot activity would probably raise the measured difficulty of those topics.","The same two-platform mining approach could be applied to other emerging engineering fields to locate support gaps before their knowledge bases mature; the low duplicate rate suggests SWS is still early in that maturation.","A testable extension is to correlate the How-type dominance with documentation coverage: if official SWS docs already describe a topic, the share of How questions on that topic should be lower.","The paper's difficulty metrics are proxy signals; connecting them to actual user frustration (e.g., through surveys or issue-closing comments) would test whether median hours genuinely capture perceived difficulty."],"forward_implications":["Developer-support investment in SWSs should target workflow execution tooling and debugging aids, since that topic has the highest unanswered rate and slowest answer time on Stack Overflow.","SWS maintainers should expect and plan for long-running refactoring and API migration work, as this topic shows the highest median resolution time on GitHub.","The dominance of How-type questions implies that step-by-step tutorials and worked examples would address a community-wide need more directly than reference-style documentation.","The unusually low duplicate-question rate (0.37%) suggests many SWS questions are novel; building a structured, searchable knowledge base could materially reduce the 60% unanswered-post rate.","Cross-platform topics such as data structures and operations, task management, and workflow scheduling are shared pain points, so improvements there benefit both Q&A users and issue-tracker communities."],"supporting_citations":[{"why":"Supplies the BERTopic topic-modeling method used to extract topics from Stack Overflow posts and GitHub issues.","marker":"[93]"},{"why":"Supplies the popularity and difficulty metrics (percentage unanswered, median time to answer or resolve) adapted for RQ3.","marker":"[21, 22, 24–27]"},{"why":"Supplies the How/Why/What/Other question-type taxonomy and the manual classification procedure used in RQ2.","marker":"[21–23]"},{"why":"Precedent for applying topic modeling to Stack Overflow posts and GitHub issues to identify challenges in another software engineering domain.","marker":"[26]"},{"why":"The replication package containing the curated dataset and scripts that the analysis and difficulty computations rely on.","marker":"[53]"},{"why":"The GitHub REST API used to collect the issues and pull requests that form the GitHub-side dataset.","marker":"[137]"}],"fun_headline_variants":["Workflow execution is developers' top Stack Overflow hurdle","How-type questions plague scientific workflow devs","GitHub and SO reveal workflow execution as pain point","Scientific workflow devs struggle most with execution","BERTopic mines 35k posts to rank workflow challenges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The GitHub difficulty rankings assume the issues and pull requests in the dataset are genuine human developer challenges; if bot-generated updates (like 'precommitci autoupdate' or 'Auto Compress Images by Calibre' pull requests) make up a large share of several topics, those rankings reflect automation, not human difficulty.","fun_headline_variants_meta":{"raw":{"variants":["Workflow execution is developers' top Stack Overflow hurdle","How-type questions plague scientific workflow devs","GitHub and SO reveal workflow execution as pain point","Scientific workflow devs struggle most with execution","BERTopic mines 35k posts to rank workflow challenges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1648,"prompt_tokens":1054,"completion_tokens":594,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":670,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":670,"tokens_out":594,"duration_ms":6138,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:11:03.924028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the GitHub analysis after removing activity from automated bot accounts (e.g., 'precommitci autoupdate' and 'Auto Compress Images by Calibre' PRs) and check whether the near-zero unresolved rates and sub-hour resolution times for those topics persist, and whether System Redesign and API Migration still has the longest median resolution time.","supporting_citations":[{"cited_title":"Online; last accessed May, 2025 (2025)","cited_arxiv_id":null,"evidence_quote":"The replication package containing the curated dataset and scripts that the analysis and difficulty computations rely on."},{"cited_title":"Online;January 2024 (2024)","cited_arxiv_id":null,"evidence_quote":"The GitHub REST API used to collect the issues and pull requests that form the GitHub-side dataset."}],"review_version":1}