{"id":"a7b1decf-5119-4436-9ea5-73102feff503","arxiv_id":"2509.00038","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Declarative prompt optimization is adapted into a four-step, code-backed framework (goal, standard, compile, package) for reproducible LLM-assisted systematic review screening.","lead":"A researcher proposes replacing hand-crafted prompts for AI-assisted systematic literature reviews with a four-step 'compiler' workflow that searches for good instructions and packages them as verifiable artifacts. The paper ships working code and notebook links so review teams can build reproducible, auditable AI pipelines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The artifact's `metrics.json` is computed on the optimization split, not a held-out test set; this undermines the advertised 'verifiable and recomputable' quality certificate.","rationale":"The reader's weakest assumption focused on hosted-LLM nondeterminism. That is a legitimate external risk and is well documented. My concern is more immediate and more internal: even before considering nondeterminism, the code in Appendix A conflates validation and test sets. The optimizer evaluates on the same examples it tunes, so the bundled metric cannot support the claim that the artifact meets the predefined quality standard on unseen abstracts. This is load-bearing because the paper's promised advantage over hand-crafted prompts is a verifiable, auditable bundle that demonstrates a quality standard; if the bundled metric is computed on the optimization split, the certificate does not certify what it claims. The issue is readily fixable by adding a held-out evaluation and changing wording. The paper explicitly refrains from empirical claims and calls for future testing, which supports a conditional rather than a rejected reading. Because the reader's verdict was already CONDITIONAL and my concern reinforces that condition, I keep the verdict unchanged.","tokens_in":7599,"tokens_out":4318,"duration_ms":56975,"concrete_test":"Use the Appendix A example with N≥20 gold-standard abstracts. Split into disjoint train/validation/test sets (e.g., 8/2/10), compile with MIPROv2 using only train/validation, then load the saved artifact and compute metrics on the untouched test set. If `metrics.json` lacks this held-out score, or if the test score differs materially from the optimization score, the 'test-set results' label is unsupported and the protocol needs either a real held-out evaluation or substantially softened wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that declarative prompt tuning can package SLR screening modules as verifiable digital artifacts satisfying a predefined quality standard. The most load-bearing defect is internal: Box 1 Step 4 promises 'metrics.json with the test-set results,' but Appendix A never creates a held-out test set. The code at lines 99-107 sets `valset = gold_standard[:10]` and `trainset = gold_standard[10:]`; with the three examples shown, the optimizer evaluates on the same gold-standard examples used to tune prompts. Thus `metrics.json` reports an optimization-set score, not a test-set score. Box 1 Step 2 says the gold-standard examples are 'not part of the study,' but that only means they are external to the reviewed corpus; it does not make them independent of the compile step. If metrics are computed on the validation split used for prompt selection, the quality certificate is partially circular and cannot establish that the compiled prompt meets the predefined standard on unseen abstracts. This is independent of the nondeterminism issue: even with perfect bit-reproducible inference, the reported numbers do not measure held-out performance. The paper honestly states that no empirical testing was done (Sections 3-4), so this is not a falsified claim; it is a mismatch between the advertised artifact and the supplied implementation. A separate, secondary issue is the asserted first-application novelty (Section 4), which is not verified by a prior-art search; but the evaluation leak is more load-bearing because it affects the core reproducibility promise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that LLM-assisted systematic literature review (SLR) is hampered by fragile, manually crafted prompts, and proposes replacing this 'prompt alchemy' with a declarative, four-step 'prompt compilation' workflow: (1) define the task and context, (2) codify a quality standard with gold-standard examples and a metric, (3) use an automated optimizer (DSPy MIPROv2 or GEPA) to search over instructions and few-shot exemplars, and (4) package the compiled program as a verifiable, auditable artifact containing config, prompt, exemplars, metrics, and run log. The proposal is illustrated through an abstract-screening blueprint (Box 1) and a DSPy code example (Appendix A). The paper explicitly positions itself as an exploratory/prototyping contribution and calls for future empirical testing.","tokens_in":7817,"tokens_out":6971,"duration_ms":76852,"significance":"The paper identifies a genuine problem—prompt fragility in SLR automation—and makes a plausible case that declarative prompt optimization frameworks from general LLM applications could be adapted to SLR. Its strengths include a concrete, code-level implementation, a clear four-component structure, and a useful Table 1 cataloging prompt-induced performance swings. If validated empirically, the framework would be a valuable methodological contribution, moving the field toward programmatic, auditable prompt construction. However, as written the significance is limited: no empirical evaluation is reported, and several load-bearing claims about verifiability, reproducibility, and LLM-agnosticism are not supported by the supplied implementation or evidence.","major_comments":[{"comment":"The blueprint promises that the packaged artifact contains 'metrics.json with the test-set results,' but the code never creates a held-out test set. The split `valset = gold_standard[:10]`, `trainset = gold_standard[10:]` means the validation split used for prompt selection is what would be reported. With the three gold-standard examples printed, `valset` contains all examples and `trainset` is empty, so the example as written is degenerate. Even with more than ten examples, metrics computed on this valset cannot certify held-out performance; they measure the optimization split. This is a load-bearing mismatch between the advertised 'verifiable quality standard' and the implementation.","section":"Box 1, Step 4; Appendix A, lines 99-107"},{"comment":"The claims that 'a collaborator can load the artefact and achieve identical performance' and that results are 'verifiable and recomputable under a pinned environment' are asserted, not demonstrated. Fixing temperature=0.0, top_p=1.0, and seed=42 does not guarantee bit-reproducible outputs from hosted LLM APIs, and model updates can silently change behavior. The code also does not log model IDs, data hashes, or environment snapshots, despite Box 1 saying 'all runs log hashes.' The paper's own Table 1 and reference [4] document cross-model and cross-prompt divergence, making the determinism assumption especially fragile. Without a concrete reproducibility protocol and evidence, the packaged artifact is documentation rather than verification.","section":"Appendix A, lines 6-11, 114-117; Box 1, Step 4"},{"comment":"The framework is described as searching for 'an LLM-agnostic prompt configuration,' but the compilation is performed under a pinned model build (e.g., `openai/gpt-4o-mini` in Appendix A) with fixed decoding parameters. The optimized prompt is therefore likely model-specific and prompt-transfer across models is not established. This is an internal tension in the central claim: the method pins a model to compile a prompt, yet declares the output LLM-agnostic. The claim should be either removed or replaced with a qualified statement about portability.","section":"Section 3; Box 1, Step 3"},{"comment":"The abstract and conclusion say the framework 'demonstrates' applicability and utility, and the conclusion calls the code a 'proof-of-concept, reproducible implementation.' However, no empirical evaluation is reported: there are no measured accuracy values, no comparison to manual prompting, and no demonstration that the compiled prompt satisfies any predefined quality standard. The paper is honest that future work should test the approach, but the wording overstates what is shown. This is acceptable if the contribution is framed as a methodological blueprint; the claims should be softened to 'proposes' and 'illustrates with a code example,' or an evaluation should be added.","section":"Sections 3-4"}],"minor_comments":[{"comment":"Typo in comment: 'determninistic' should be 'deterministic'; also the dict keys in lines 7-10 have stray spaces around the quoted strings in the printed listing.","section":"Appendix A, line 5"},{"comment":"The paragraph after the MIPROv2 example contains a duplicated and grammatically broken sentence: 'This positions the method ... This process is can be seen as analogous to...' The same analogy to hyperparameter tuning is already stated earlier in the section; the text should be consolidated.","section":"Section 3"},{"comment":"The claim that 'this represents the first application of such declarative techniques to evidence synthesis workflows' is asserted without a prior-art search. Given the rapid growth of LLM/SLR literature, this novelty claim should be softened or supported with a systematic literature search.","section":"Section 4"},{"comment":"The text groups GRPO [20] with DSPy and GEPA as 'prompt optimisation' frameworks. GRPO is a reinforcement-learning training method, not a declarative prompt-tuning framework. The citation should be recontextualized or replaced with a more directly relevant reference.","section":"References [19]-[21]; Section 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is better characterized as a position/blueprint paper than an empirical methods paper. The central idea is plausible and the code is a useful starting point, but the gap between the advertised reproducibility guarantees and the supplied implementation is large. A revision should add a genuine held-out evaluation, correct the code split, and temper the determinism and novelty claims. Whether this journal accepts methods proposals without validation depends on scope; if it expects empirical evidence, the paper would need a substantial evaluation component."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's genuine contribution is narrow but real: it adapts existing declarative prompt optimization frameworks (DSPy/MIPROv2, GEPA) to the SLR abstract-screening task and packages the idea as a four-step blueprint with working Python. That combination is new in this subfield, and the appendix code is a plausible DSPy implementation. The paper is also refreshingly honest that no empirical evaluation was run and positions itself as a probing study. For a researcher in SLR automation, this is a useful starting point.\n\nThe soft spots are where the advertised guarantees outrun the evidence. The most load-bearing is internal: Box 1 promises metrics.json with 'test-set results,' but the appendix never creates a held-out test set. The code splits gold_standard into valset and trainset, and the optimizer evaluates on valset — the same examples used for prompt selection. So the reported metrics are optimization-set scores, not held-out performance, and the quality certificate is partly circular. That would be a quick fix, but it needs to be stated clearly. The second issue is the determinism claim: 'identical performance' on reload assumes hosted LLMs are bit-reproducible at temperature zero, which they are not. The paper's own Table 1 documents cross-model and cross-prompt divergence. So the artifact should be described as reproducible in the sense of pinned code and logging, not bit-identical outputs. Third, the 'first application' novelty claim would be stronger with a prior-art search; it's asserted, not demonstrated.\n\nNone of these are fatal to the central proposal. The idea of treating prompt compilation as audit-friendly hyperparameter tuning is sound and worth pursuing. The paper just needs a held-out evaluation, softened wording on determinism, and a more careful novelty claim. I'd send it to peer review because it gives the SLR community a concrete, testable methodology and ships code that others can build on. The reader's conditional verdict is about right.\n\nWho is this for? Methodologists working on LLM-assisted evidence synthesis, and people who want a cheap entry point into DSPy-style prompt optimization. I would not cite it yet, but I'd keep an eye on the follow-up.","headline":"A honest methods proposal for declarative prompt tuning in SLRs, but the supplied code's 'test-set' metrics are computed on the optimization split and the reproducibility claim overreaches; still worth a serious referee.","tokens_in":8448,"tokens_out":1062,"would_cite":false,"duration_ms":13705,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes replacing hand-crafted prompts in AI-assisted systematic reviews with four-step “prompt compilation”: declare the task, codify the quality bar with labeled examples, search for the best prompt automatically, and package","keywords":["Systematic Literature Review Automation","Evidence Synthesis","Large Language Models","Reproducibility","Prompt Engineering","Context Engineering","Prompt Optimisation","Prompt Compilation"],"falsifier":"Run the saved screening artifact on the same gold-standard abstracts twice—once at packaging time and once after a vendor-side model update—with the same model ID and fixed generation settings, and compare every decision. Any change in decisions or accuracy falsifies the strong claim that the artifact is verifiable and recomputable under a pinned environment; even repeated identical calls that yield different outputs would settle the determinism question.","tokens_in":7357,"feed_emoji":"⚙️","tokens_out":7986,"duration_ms":83381,"temperature":0.7,"pith_summary":"Systematic literature reviews are slow and labor-intensive, and large language models could accelerate them—but LLM outputs swing wildly with small prompt changes, which is a serious problem for evidence synthesis. This paper argues that the remedy is to stop treating prompts as hand-crafted text and instead compile them: a researcher declares the review task, codifies the quality standard with gold-standard examples and a metric, runs an automated budgeted search over instruction templates and few-shot exemplars, and saves the tuned prompt as a versioned bundle others can reload. It translates this into a four-component blueprint and a working code example for abstract screening, and frames the work as the first application of declarative prompt tuning to SLR automation. The contribution is a methodological proposal with a proof-of-concept implementation, not a report of measured accuracy gains.","feed_headline":"Compile, don't craft: reproducible LLM prompts for systematic reviews","feed_subtitle":"Declarative prompt tuning bundles task specs, labeled tests, and tuned prompts into auditable review artifacts.","key_machinery":"The moving part is declarative LM-program compilation: pair a fixed language model with a structured task declaration, a small gold-standard validation set, and a metric, then let an automated optimizer search over instruction templates and few-shot exemplars to maximize the metric. This converts prompt writing from an act of crafting into an act of tuning, and the tuned result becomes an object that can be saved, loaded, and audited. The four-step workflow—define the goal, codify the standard, compile the program, package the artifact—is the practical chassis that carries the argument.","core_discovery":"The central claim is that declarative prompt tuning—an approach in which a fixed pre-trained language model is treated as a compiler and a prompt as a program to be optimized against a validation metric—can and should be applied to systematic-review automation. The paper operationalizes the idea through four components: a task declaration plus versioned review context (define the goal), a gold-standard dataset with an accuracy metric (codify the standard), a controlled search over prompts and few-shot exemplars under pinned decode settings (compile the program), and a shareable bundle containing config, prompt, exemplars, metrics, and run logs (package the artifact). The contribution is the","pith_inferences":["A natural stress test not reported in the paper: run the same saved artifact after a vendor-side model update and measure how much screening decisions drift; large drift would undercut the 'verifiable and recomputable' promise even if the workflow itself is sound.","The compiled prompt inherits any bias or gaps in the gold-standard examples and the choice of metric, so an auditable artifact is only as trustworthy as the labeled data it was tuned against—a curation standard would be a natural companion to this framework.","The compile step could itself be benchmarked against careful manual prompt engineering on the same gold-standard sets; the paper proposes the workflow but does not yet quantify when compilation wins, ties, or loses.","If packaged artifacts are shared as community components, the same tuning loop could be re-run per model generation, turning model updates from a reproducibility hazard into a routine recompilation step."],"forward_implications":["SLR tasks such as abstract screening can be expressed as declarative workflows whose prompts are produced by automated search rather than manual trial and error.","Reviewers' scientific intent (the 'what') is separated from model-specific implementation (the 'how'), so pipelines should become less sensitive to which LLM happens to be used.","Prompt optimization becomes a data-driven process analogous to hyperparameter tuning, with validation metrics, budgets, and run logs that support audit.","Saved artifacts can be reloaded by collaborators to reproduce classifications, making AI-assisted screening steps transparent and recomputable under a pinned environment.","The framework positions AI-assisted reviews to satisfy transparency and traceability expectations in evidence synthesis rather than treating prompts as hidden or arbitrary."],"supporting_citations":[{"why":"supplies the scoping review documenting that LLMs can automate many SLR phases but are not yet dependable, motivating the need for a more systematic approach.","marker":"[1]"},{"why":"quantifies how prompt-format changes can move task accuracy by up to 76 points, the core fragility the framework targets.","marker":"[3]"},{"why":"benchmarks extraction recall varying 64–92% across LLMs and prompting strategies, direct evidence of prompt/model sensitivity inside SLR tasks.","marker":"[4]"},{"why":"provides a reproducibility and generalizability study of LLM query generation, evidence of brittle behavior in an SLR-adjacent task.","marker":"[7]"},{"why":"introduces the declarative prompt-optimization approach that the paper adapts into its SLR compilation workflow.","marker":"[19]"},{"why":"supplies a prompt-optimizing reinforcement-learning method the paper names as part of the emerging toolbox it imports.","marker":"[20]"},{"why":"supplies a reflective prompt-evolution method the paper lists alongside the other optimizers as inspiration for automated prompt search.","marker":"[21]"}],"fun_headline_variants":["Treat prompts as programs to compile AI-assisted reviews","Declarative prompt tuning: a reproducible blueprint for reviews","Automate evidence synthesis with compilable LLM prompts","From handcrafted to compiled: LLM prompts for rigorous reviews"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The promise that a saved prompt bundle can be reloaded later with identical results rests on the assumption that a fixed model with fixed decoding settings produces the same output every time, including after the vendor updates the model.","fun_headline_variants_meta":{"raw":{"variants":["Treat prompts as programs to compile AI-assisted reviews","Declarative prompt tuning: a reproducible blueprint for reviews","Automate evidence synthesis with compilable LLM prompts","From handcrafted to compiled: LLM prompts for rigorous reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1397,"prompt_tokens":664,"completion_tokens":733,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":408,"completion_tokens_details":{"reasoning_tokens":666}},"tokens_in":408,"tokens_out":733,"duration_ms":8806,"temperature":1.0,"reasoning_tokens":666,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:08:55.558101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the saved screening artifact on the same gold-standard abstracts twice—once at packaging time and once after a vendor-side model update—with the same model ID and fixed generation settings, and compare every decision. Any change in decisions or accuracy falsifies the strong claim that the artifact is verifiable and recomputable under a pinned environment; even repeated identical calls that yield different outputs would settle the determinism question.","supporting_citations":[{"cited_title":"What Level of Automation is \"Good Enough\"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction","cited_arxiv_id":"2507.15152","evidence_quote":"benchmarks extraction recall varying 64–92% across LLMs and prompting strategies, direct evidence of prompt/model sensitivity inside SLR tasks."},{"cited_title":"A reproducibility and generalizability study of large language models for query generation","cited_arxiv_id":null,"evidence_quote":"provides a reproducibility and generalizability study of LLM query generation, evidence of brittle behavior in an SLR-adjacent task."},{"cited_title":"Optimizinginstructionsanddemonstrationsformulti-stagelanguagemodelprograms","cited_arxiv_id":null,"evidence_quote":"introduces the declarative prompt-optimization approach that the paper adapts into its SLR compilation workflow."}],"review_version":1}