{"id":"9180b4f8-afe4-48c4-a489-14e22dcae0e5","arxiv_id":"2412.03595","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The PLUMED community built an open, continuously tested tutorial platform that links learning materials to the software's own documentation.","lead":"PLUMED Tutorials is a new community-driven website where molecular simulation researchers build, share, and continuously update guided tutorials for the PLUMED software. The project uses automated testing and documentation links to keep tutorial examples usable as the software changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learning-path claim lacks quantitative support: the paper never reports how many of the 47 tutorials actually appear in the guidance graph.","rationale":"The reader's weakest assumption—that the learning path depends on contributors actively creating and maintaining links—is precisely the load-bearing point. My concern is the same, but made more concrete and testable: the paper does not quantify the actual connectivity of the tutorial graph, so the central claim of a structured learning path is not supported by the data presented. This does not change the verdict: CONDITIONAL remains appropriate because the condition is exactly that the authors demonstrate the learning-path claim with data (or substantially qualify it). The paper has real strengths—a live website, 47 tutorials, and automated parse-checking infrastructure—but those do not by themselves establish that the corpus is interconnected. The proposed test is straightforward and could be run by the authors or an independent reader from the public website and repositories. No ad hominem is intended; this is a request for evidence that the paper's own indexing section suggests may be missing.","tokens_in":14555,"tokens_out":2470,"duration_ms":23505,"concrete_test":"Scrape the live PLUMED Tutorials site (or parse the NAVIGATION.md/EMBED.yml files from the 47 tutorial archives listed in Table 1) and compute the directed graph of internal tutorial-to-tutorial links. Then report: number of tutorials with at least one incoming or outgoing edge, number of weakly connected components, size of the largest component, and the result after applying the paper's stated cycle-removal and transitive-reduction procedure. If the largest component contains fewer than roughly 75% of the 47 tutorials, the 'structured learning path' claim should be softened to 'partially connected collection.' Also check whether the cycle-removal step is deterministic by running it on the same graph twice with different edge orderings; if the resulting graph differs, the construction is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that tutorials are 'interconnected to form a structured learning path.' The only evidence offered is Fig. 5, which shows 'selected tutorials,' and the text in the Indexing section. The paper itself acknowledges that a tutorial 'does not appear in the graph in Fig. 5' if it has no incoming links and its contributor adds no outgoing links. However, the paper never reports: (i) how many of the 47 tutorials are present in the guidance graph, (ii) how many edges exist among them, (iii) the size of the largest connected component, or (iv) how circular dependencies are removed before transitive reduction. Without these data, the graph could consist of a small connected component plus many isolated tutorials, in which case describing the resource as having 'a structured learning path' is an overstatement. This is not an external-consensus dispute; it is a gap between the claim and the reported evidence. The infrastructure demonstrably supports linking, but whether the current corpus actually forms a learning path is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents PLUMED Tutorials, a community-driven infrastructure for developing, sharing, and updating online tutorials for the PLUMED molecular simulation package. After reviewing the limitations of past in-person PLUMED schools and static online tutorials, the authors describe the contribution workflow: contributors package markdown files, Python notebooks, PDFs, videos, and PLUMED input files in a zip archive with NAVIGATION.md and EMBED.yml files; a GitHub Actions workflow builds the site and renders each tutorial with a Mermaid navigation graph. The paper also details the automatic rendering of PLUMED input files, including parse-status badges for the current and development versions of PLUMED, hover tooltips with keyword documentation, expansion of shortcut actions, and highlighting of data passed between actions. It then describes the searchable index and a guidance graph constructed from inter-tutorial links via cycle removal and transitive reduction. The live site currently contains 47 tutorials, and the authors position the platform as a model for other scientific software communities.","tokens_in":14742,"tokens_out":6542,"duration_ms":56561,"significance":"The paper's concrete strengths are that the resource is public and working, the contribution and testing workflow is described in sufficient detail to be reproduced, and the automated annotation and syntax-checking features are genuinely useful for keeping training materials connected to a fast-moving code base. If the claims about the current corpus are supported, the platform would be a valuable template for community-maintained training materials in computational science. However, the paper provides no quantitative evidence for two central claims: that the 47 tutorials actually form a structured learning path, and that continuous integration ensures ongoing compatibility. It also presents no evaluation of educational effectiveness. These gaps are fixable and do not undermine the value of the infrastructure description, but they currently leave the headline claims stronger than the evidence.","major_comments":[{"comment":"The abstract claims that the tutorials 'are interconnected to form a structured learning path,' but the only support is Fig. 5, which displays 'selected tutorials,' and the text explicitly states that tutorials without incoming links and without outgoing links do not appear in that graph. The manuscript does not report how many of the 47 tutorials are nodes in the guidance graph, how many directed edges exist after transitive reduction, or the sizes of the connected components. Without these numbers, the current corpus could consist of a small linked core and many isolated tutorials, which would make 'a structured learning path' an overstatement. Please report the graph statistics or soften the claim to describe the linking mechanism as a capability.","section":"Indexing the tutorials"},{"comment":"The abstract states that continuous integration is used to 'ensure compatibility with software updates,' but the described test only checks whether the current release and master branch can parse the input files (the green/red badges in Fig. 2). This verifies syntactic parseability, not that the tutorial runs correctly, produces the intended results, or remains numerically compatible. Moreover, no data are reported on how many of the 47 tutorials currently parse under both versions, so the claim of ensured compatibility is stronger than the evidence. Please either report the current pass rates and clarify the scope of the checks, or revise the wording to 'check syntactic compatibility.'","section":"Linking the tutorials with the documentation"},{"comment":"The paper asserts without supporting evidence that PLUMED Tutorials 'offer a more comprehensive overview of the software's functionalities' and that the tooltips and hyperlinks make it easier for students to understand and modify inputs. These are testable claims about the resource's educational value, but no learning-outcome data, user surveys, or comparison with previous tutorial formats are provided. Since the paper is submitted to a physics education venue and the abstract positions the initiative as addressing training challenges, some form of evaluation—even a small usability or learning study—would substantially strengthen the central claims. If the paper is intended as an infrastructure description rather than an efficacy study, the claims should be correspondingly hedged.","section":"Conclusions"}],"minor_comments":[{"comment":"The file name 'NA VIGATION.md' appears with a space in two places; it should read 'NAVIGATION.md'.","section":"How to contribute a PLUMED Tutorial"},{"comment":"The phrase 'S.E.H. is founded by a Roux-Cantarini fellowship' should be 'funded by'.","section":"Acknowledgement"},{"comment":"The caption of Fig. 5 says 'Selected tutorials'; since the text describes the figure as the guidance graph for all tutorials, please state in the caption whether the figure is complete or a curated subset and how the selection was made.","section":"Indexing the tutorials"},{"comment":"The phrase 'how some of complex methods are constructed' should be 'how some complex methods are constructed'.","section":"Linking the tutorials with the documentation"},{"comment":"The sentence describing the HTML string '%20' as a separator is imprecise; it is the URL-encoding of a space. Consider rewording for clarity.","section":"Indexing the tutorials"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is authored by the creators and maintainers of the platform, and several authors are also contributors to the tutorials. This is not a conflict of interest problem in the narrow sense, but it means the absence of independent data matters more: the paper's evidence is largely the authors' own description of their system. The journal fit is also worth considering: the paper is closer to a community-resource report than to a traditional education study, and the physics.ed.ph venue may expect some form of evaluation. I would recommend the editors request the quantitative graph statistics and a section on known limitations and sustainability before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper describes a genuinely useful thing: a community-run PLUMED tutorial site built on Git, continuous integration, and automatic syntax checking. The site is live, has 47 tutorials, and the design is thoughtful. The novel part is the combination—tested input files, tooltips that link to the manual, and contributor-maintained links—not any single feature. That combination is not present in the LiveCoMS, MolSSI, or Galaxy models they cite, so the contribution is real.\n\nThe paper is honest about its limits in places. The Indexing section explicitly says a tutorial without incoming or outgoing links does not appear in the guidance graph. But the abstract claims the tutorials are \"interconnected to form a structured learning path,\" and the paper never reports how many of the 47 actually show up in that graph, how many edges exist, or the size of the largest connected component. Figure 5 shows only \"selected tutorials.\" So the central selling point is asserted on the basis of the infrastructure rather than the actual corpus. That is a genuine soft spot, and it is fixable with a paragraph of numbers.\n\nThe other soft spots are minor for this genre. There is no measurement of learning outcomes or tutorial maintenance rates; the paper leans on \"we believe\" and self-report. But this is a platform description, not a hypothesis test, and the authors do acknowledge that collective projects depend on sustained contribution and credit. The citation pattern is appropriate—they cite the prior tutorial initiatives, and the PLUMED references are the standard ones.\n\nBottom line: the infrastructure contribution is real and reproducible—the site is live and the automated testing is concrete. The paper deserves a serious referee. I would ask for a quantitative description of the tutorial graph and a more careful phrasing of the \"learning path\" claim before publication, but I would not desk-reject it.","headline":"A real, useful infrastructure paper whose main overreach is calling the tutorial graph a 'structured learning path' without reporting how connected it actually is.","tokens_in":15553,"tokens_out":2143,"would_cite":false,"duration_ms":19796,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PLUMED Tutorials is a collaborative infrastructure in which tutorial pages are rebuilt from contributor archives, automatically tested against the current and development versions of the PLUMED software, and interconnected through a…","keywords":["PLUMED","online tutorials","community-driven education","continuous integration","enhanced sampling","molecular dynamics","learning path","open source software training"],"falsifier":"Watch the live tutorial site across one or two PLUMED release cycles and compute the fraction of rendered input blocks showing red parse badges after each release; if that fraction stays high for months, or if the number of contributions stops growing, the claim that continuous integration keeps tutorials compatible and that the community sustains the collection would be contradicted.","tokens_in":14414,"feed_emoji":"🧪","tokens_out":5571,"duration_ms":45619,"temperature":0.7,"pith_summary":"The paper presents PLUMED Tutorials, an open infrastructure for developing, sharing, and updating online tutorials for the PLUMED molecular-dynamics software. The authors argue that this collaborative model overcomes three limitations of traditional training: in-person schools reach few students and are time-constrained, static online tutorials go stale as the software evolves, and manually curated material rarely covers the full range of implemented methods. In their design, tutorial files live in contributor-controlled archives, a continuous-integration build assembles the site, and every example input is automatically parsed by both the release and development versions of PLUMED so that obsolete material is flagged. Tutorials can link to one another, and a graph distilled from those links gives newcomers a suggested learning path. If the model works, it gives the community a living training resource that stays in step with the code and complements journal articles.","feed_headline":"Community-built tutorials keep PLUMED training evergreen","feed_subtitle":"A GitHub-Actions build tests every example input and links lessons into a learning path.","key_machinery":"The central mechanism is the PLUMED Tutorials build-and-render pipeline. A GitHub Actions workflow downloads contributor archives from GitHub or Zenodo and constructs a GitHub Pages site, reading NAVIGATION.md (written in GitHub markdown plus Mermaid diagram syntax) for the landing page and EMBED.yml to resolve embedded videos, external links, and links to other tutorials. The pipeline's renderer for PLUMED input files carries the argument: it parses each code block with the current release and master versions of PLUMED to produce compatibility badges, attaches keyword tooltips linked to the PLUMED manual, expands shortcut actions into their full underlying inputs, and color-codes values passed between actions. A second mechanism, the indexing step, creates a directed graph from contributor-supplied tutorial links, removes circular dependencies, applies a transitive reduction, and thereby produces a democratically determined learning path; tutorials with no incoming links simply do not appear in that graph.","core_discovery":"The central claim is that a software community can maintain its own training ecosystem through repository management and continuous integration rather than through centralized authorship. The authors show that PLUMED Tutorials pages are constructed automatically from zip archives contributed by any researcher; each archive contains markdown, notebooks, PDFs, videos, a NAVIGATION.md file that specifies a Mermaid diagram, and an EMBED.yml file that declares embedded and external resources. The renderer turns every PLUMED input block into an annotated, interactive object: it tests whether the current release and the master branch can parse the input, shows a green or red badge accordingly, adds hover tooltips that explain keywords and link to manual pages, expands shortcut actions to reveal their defaults and inner workings, and highlights the values passed between actions. Instead of imposing a fixed curriculum, the site lets contributors link their tutorials to others, and the indexing layer builds a directed graph from those links, removes circular dependencies, and applies a transitive reduction to suggest an order to work through the material. The authors argue this makes the collection more comprehensive than any single school, keeps it compatible with an evolving codebase, and provides students with both context and guidance.","pith_inferences":["Beyond the paper, the same pipeline could be applied to other scientific software with a well-defined input language: the combination of CI-driven parsing and manual-linked tooltips is not PLUMED-specific.","The compatibility badges only test whether inputs parse; a natural extension would be to run the tutorials' commands end-to-end on small systems, catching runtime and numerical breakage that parsing misses.","The action-usage index that accumulates from rendered inputs could become a measurable record of method adoption across the community, useful for studying how simulation techniques spread.","If the guidance graph were augmented with prerequisite relations inferred from action usage, the site could recommend remedial tutorials to a learner before they start a target tutorial."],"forward_implications":["Learners anywhere can work through the same material asynchronously, and instructors can assign tutorials before an in-person school so meeting time is spent on projects and discussions.","Because every example input is checked against the current and development versions of PLUMED, syntax changes become visible immediately as red badges, prompting updates before tutorials break silently.","The set of tutorials a student sees is not fixed by editors; it grows and reorganizes as contributors add or link tutorials, so the learning path is community-determined.","Developers can use the automatically built action and module indexes to see which features are used in tutorials and in PLUMED-NEST, and manual pages can link to live examples.","The same tooling renders more than 1,600 example inputs from tutorials and PLUMED-NEST, so the annotated corpus of working examples becomes a richer training resource than a manual alone."],"supporting_citations":[{"why":"Identifies the original PLUMED software that the tutorial ecosystem is built around.","marker":"[1]"},{"why":"Describes PLUMED 2, the software version whose release and master branches drive the parse tests and whose syntax appears in every example input.","marker":"[2]"},{"why":"Introduces the PLUMED consortium and PLUMED-NEST, the community repository model that PLUMED Tutorials extends.","marker":"[11]"},{"why":"Supplies the transitive-reduction algorithm used to convert contributor-created tutorial links into a non-redundant learning-path graph.","marker":"[48]"},{"why":"Motivates the paper's claim that tutorials complement journal articles by supporting FAIR sharing of methods rather than only data.","marker":"[49]"}],"fun_headline_variants":["Community-built, auto-tested, linked PLUMED tutorials","PLUMED community keeps tutorials evergreen via GitHub automation","Auto-tested, community-curated tutorials for PLUMED","How PLUMED's community keeps its tutorials evergreen","PLUMED tutorials evolve with community testing and links"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ecosystem's value depends on sustained voluntary contributions and on contributors linking their tutorials to others, since unlinked tutorials drop out of the learning-path graph and the collection would otherwise go stale.","fun_headline_variants_meta":{"raw":{"variants":["Community-built, auto-tested, linked PLUMED tutorials","PLUMED community keeps tutorials evergreen via GitHub automation","Auto-tested, community-curated tutorials for PLUMED","How PLUMED's community keeps its tutorials evergreen","PLUMED tutorials evolve with community testing and links"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3077,"prompt_tokens":929,"completion_tokens":2148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2068}},"tokens_in":545,"tokens_out":2148,"duration_ms":14405,"temperature":1.0,"reasoning_tokens":2068,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T06:00:32.533559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Watch the live tutorial site across one or two PLUMED release cycles and compute the fraction of rendered input blocks showing red parse badges after each release; if that fraction stays high for months, or if the number of contributions stops growing, the claim that continuous integration keeps tutorials compatible and that the community sustains the collection would be contradicted.","supporting_citations":[{"cited_title":"A.; Parrinello, M","cited_arxiv_id":null,"evidence_quote":"Identifies the original PLUMED software that the tutorial ecosystem is built around."},{"cited_title":"A.; Bonomi, M.; Branduardi, D.; Camilloni, C.; Bussi, G","cited_arxiv_id":null,"evidence_quote":"Describes PLUMED 2, the software version whose release and master branches drive the parse tests and whose syntax appears in every example input."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the PLUMED consortium and PLUMED-NEST, the community repository model that PLUMED Tutorials extends."},{"cited_title":"V.; Garey, M","cited_arxiv_id":null,"evidence_quote":"Supplies the transitive-reduction algorithm used to convert contributor-created tutorial links into a non-redundant learning-path graph."}],"review_version":1}