{"id":"761cc8e4-66c5-4ba7-a2c0-4cf477877918","arxiv_id":"2508.00836","paper_version":6,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Rxiv-Maker is a Markdown-based template engine that runs embedded code during PDF compilation to produce self-updating, reproducible scientific manuscripts.","lead":"Rxiv-Maker converts Markdown files into publication-ready PDFs, executing embedded code so figures and statistics refresh automatically from source data. It aims to make academic manuscripts reproducible and synchronized with evolving code and data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-updating and fully reproducible pull in opposite directions unless code, data, and environment are pinned; the abstract supplies no evidence they are.","rationale":"The abstract provides only high-level claims, so the verdict must remain provisional. However, the specific conceptual tension between continuous updates and static reproducibility is identifiable directly from the abstract and does not require the full text. The reader's weakest assumption about deterministic environments is related but not identical: the deeper issue is the conflict between 'always current' and 'fully reproducible', which would need explicit version-pinning and provenance metadata to resolve. Because no full text is available, the appropriate verdict remains UNVERDICTED, and no adjustment to the reader's assessment is needed.","tokens_in":588,"tokens_out":2811,"duration_ms":34466,"concrete_test":"Compile a minimal Rxiv-Maker manuscript containing an embedded random-number or date-dependent computation twice in identical conditions from the same source, and compare SHA-256 checksums of the produced PDFs; then recompile after changing the underlying data. If the two identical-source builds differ, or if the PDF metadata lacks a build manifest (dependency lock, random seed, commit hash, data snapshot), the reproducibility claim is unsupported. This directly tests whether 'always current' and 'fully reproducible' are simultaneously realized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim contains an internal tension: 'the final document is always current' requires recompiling when data or code change, while 'fully reproducible' requires that a fixed input state deterministically yields the same artifact. If embedded code executes at compilation time without a hermetic environment, then the same Markdown source cannot reliably reproduce the same PDF across time or machines. The abstract does not state how Rxiv-Maker pins dependency versions, records random seeds, snapshots external data, or embeds provenance (e.g., the Git commit hash) into the generated PDF. It also does not describe error handling when code fails or output changes due to environment updates. Without these mechanisms, 'fully reproducible' is unsupported; at best the system produces a current document whose provenance is unspecified, and 'always current' may be purchased at the cost of reproducibility. This is load-bearing because the entire contribution rests on the conjunction of these two properties.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Rxiv-Maker, a framework that converts Markdown files into publication-ready PDFs by executing embedded code during compilation. The central claim is that the resulting documents are simultaneously 'always current' and 'fully reproducible,' because figures and statistics are regenerated directly from source data. The paper further argues that building on Git and Visual Studio Code provides a transparent, collaborative workflow. This review is based solely on the abstract; the full text was not available for assessment.","tokens_in":721,"tokens_out":1772,"duration_ms":20898,"significance":"If the framework delivers on its claims, it could meaningfully reduce the technical burden of preparing computational manuscripts and help keep papers synchronized with evolving code and data. The emphasis on standard tools (Git, VS Code) is plausible and lowers adoption barriers. However, the significance hinges on the abstract's two core claims, neither of which is supported by evidence at this level of detail. The paper would be strengthened by concrete demonstrations of reproducibility across environments and update scenarios, which are not visible in the abstract.","major_comments":[{"comment":"The claim that the final document is 'always current and fully reproducible' is load-bearing, but the abstract does not state how reproducibility is achieved when embedded code executes during compilation. There is no mention of environment pinning, dependency versioning, random-seed management, or provenance embedding (e.g., a Git commit hash in the PDF). Without such mechanisms, the same Markdown source may produce different PDFs across machines or over time, so the reproducibility claim is unsupported by the information provided.","section":"Abstract"},{"comment":"The 'always current' property implies that recompilation occurs whenever data or code change, but the abstract does not describe the trigger mechanism or the behavior when code fails, produces warnings, or yields different output because of external updates. The paper must specify how errors are handled and under what conditions 'current' is defined; otherwise the two advertised properties may pull in opposite directions, as the stress-test note correctly observes.","section":"Abstract"},{"comment":"The abstract presents no evaluation or test cases. For a tool paper, the full text should include at least one demonstration of (a) the same input producing identical artifacts in a controlled environment and (b) an update to source data or code producing a refreshed document, ideally with recorded provenance. Without such evidence, the central claims remain unverified.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'always current' is an absolute that may be better stated as 'current as of the latest compilation,' to avoid overclaiming in the absence of continuous integration details.","section":"Abstract"},{"comment":"The abstract uses 'synchronised' (British spelling) while the rest of the text is likely American; please ensure consistent spelling throughout.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based on the abstract only because the full text was not provided. The stress-test concern about the tension between 'always current' and 'fully reproducible' is well grounded in the abstract and should be explicitly addressed in the full paper. I recommend sending the full manuscript for a complete review before a final decision, since the abstract alone is insufficient to judge soundness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is an abstract-only look, so take everything with salt. What I can see: Rxiv-Maker is an executable-Markdown tool that compiles embedded code into PDFs, aimed at keeping computational manuscripts in sync with data. That problem is real, but the solution category is well-trodden—R Markdown, Quarto, and Jupyter already do this. The distinctive claims—'always current' and 'fully reproducible'—sit in tension unless the tool pins code, data, and environment, and the abstract gives no evidence it does.\n\nCredit where it's due: the paper frames manuscript maintenance as a software-engineering problem, builds on Git and VS Code, and targets a pain point computational researchers actually feel. The writing is clear, and the scope is modest if it holds. That is genuinely useful framing, even if the execution is not novel.\n\nThe main soft spot is load-bearing: the reproducibility claim is unsupported. Executing code at compile time does not make a document reproducible. You need dependency locking, seed control, data snapshots, provenance metadata, and defined failure behavior. None of that appears in the abstract. Also, there is no comparison to existing tools, no benchmarks, no evaluation, no example output. The citation pattern can't be assessed because no references are visible. These omissions may be in the full text, but as presented the central claim is assertion, not demonstration. The stress-test tension is real: if the document is always current, then its reproducibility depends on how the environment is frozen; the two properties are not jointly delivered by 'execute code during compilation' alone. A full paper might easily resolve this by showing how the tool records Git commit hashes, locks dependency versions, or handles non-determinism. But the abstract doesn't say any of that exists.\n\nWho is this for? A computational researcher who wants version-controlled manuscripts with live figures. They would get value if the tool works and is maintained. Should it be peer reviewed? Yes—it deserves a serious technical referee who can run the code and test the reproducibility claims. My own verdict is skeptical pending full text, but I would send it out rather than desk reject.","headline":"Abstract-only look at a plausible but unproven executable-Markdown tool; the 'always current' and 'fully reproducible' claims are in tension unless environments are pinned, and the abstract shows no evidence of pinning.","tokens_in":1216,"tokens_out":1383,"would_cite":false,"duration_ms":17347,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rxiv-Maker compiles Markdown into self-updating, publication-ready PDFs by executing embedded code at build time.","keywords":["reproducible research","Markdown","manuscript automation","embedded code execution","preprint authoring","literate programming","version control","publication-ready PDFs"],"falsifier":"Recompile the same Rxiv-Maker manuscript twice in a clean environment, including after a fresh clone from its Git repository, and compare the generated figures and numerical values; if the two PDFs differ without any intentional change to data or code, the self-updating and reproducibility claims collapse.","tokens_in":439,"feed_emoji":"📄","tokens_out":3488,"duration_ms":36042,"temperature":0.7,"pith_summary":"Rxiv-Maker is a template engine that converts simple Markdown files into professionally typeset, publication-ready PDFs. Its core feature is executing embedded code during compilation, so figures and statistical values are generated directly from source data and the manuscript updates itself whenever that data changes. The paper argues this resolves the growing technical burden on authors in computational research, where manuscripts must stay synchronized with evolving data and code. If the approach works, researchers gain a transparent, collaborative authoring workflow that applies software-engineering practices to academic writing and makes the final PDF fully reproducible.","feed_headline":"Rxiv-Maker compiles Markdown into self-updating PDFs","feed_subtitle":"Figures and statistics regenerate from source data on every build, keeping manuscripts reproducible and current.","key_machinery":"The core mechanism is embedded code execution at compile time: Markdown files contain executable code blocks, and the compilation pipeline runs those blocks against the source data to generate figures and statistical values. This turns the manuscript into a program whose output is the final PDF, so the document is always derived from the current state of the data and code. Rxiv-Maker wraps this pipeline with Git-based version control and VS Code integration to deliver a collaborative, software-engineering-style workflow.","core_discovery":"The paper's central claim is that a compilation pipeline which executes embedded code can keep a scientific manuscript permanently synchronized with its underlying data and code. In Rxiv-Maker, a Markdown source containing inline executable blocks is compiled into a publication-ready PDF, and each build runs the code against the current source data to produce the figures and statistics that appear in the document. The result is a self-updating manuscript: whenever data or code change, recompiling yields an updated, reproducible PDF without manual figure regeneration or copy-pasting of numbers. The framework builds on standard tools such as Git and Visual Studio Code to make the authoring process transparent and collaborative.","pith_inferences":["A natural extension is to apply the same compile-time code execution to other long-form documents such as theses, technical reports, and lab notebooks, where versioned data and code matter just as much.","Deterministic execution is the key stress point; containerized or pinned computing environments would be a direct way to make the reproducibility guarantee reliable across machines.","This approach could enable 'living figures' in preprints, where a linked repository automatically regenerates the displayed figure whenever the data are updated.","A simple quantitative test of the claimed benefit would compare the time and error rate of updating a manuscript with and without embedded execution."],"forward_implications":["Authors can stop manually regenerating figures and copying statistics, because each compile produces figures and numbers directly from source data.","Version control becomes the backbone of writing: every change to text, data, or code leaves a traceable Git history.","Reviewers and readers can regenerate the PDF from the Markdown source, making the artifact fully reproducible.","The workflow lowers the barrier to applying software-engineering discipline to academic writing, including versioning and automation.","Updates to a preprint after new data arrive reduce to recompiling, keeping the posted document current without extra manual steps."],"supporting_citations":[],"fun_headline_variants":["Rxiv-Maker: self-updating PDFs from Markdown","Markdown compiles to PDFs that update themselves","Code in Markdown, figures auto-regenerate in PDF","Reproducible PDFs via Markdown with live code","Rxiv-Maker: manuscripts that regenerate from data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The workflow's promise of always-current, reproducible PDFs depends on code execution during compilation being deterministic, so re-running the build in the same environment reproduces identical figures and numbers, and it assumes the source data and code remain accessible and unchanged.","fun_headline_variants_meta":{"raw":{"variants":["Rxiv-Maker: self-updating PDFs from Markdown","Markdown compiles to PDFs that update themselves","Code in Markdown, figures auto-regenerate in PDF","Reproducible PDFs via Markdown with live code","Rxiv-Maker: manuscripts that regenerate from data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00031,"raw_usage":{"total_tokens":1692,"prompt_tokens":792,"completion_tokens":900,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":408,"tokens_out":900,"duration_ms":9208,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:35:27.466603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompile the same Rxiv-Maker manuscript twice in a clean environment, including after a fresh clone from its Git repository, and compare the generated figures and numerical values; if the two PDFs differ without any intentional change to data or code, the self-updating and reproducibility claims collapse.","supporting_citations":[],"review_version":1}