Pith. sign in

REVIEW 4 major objections 4 minor 21 references

Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes replacing hand-crafted prompts in AI-assisted systematic reviews with four-step “prompt compilation”: declare the task, codify the quality bar with labeled examples, search for the best prompt automatically, and package

desk verdict A honest methods proposal for declarative prompt tuning in SLRs, but the supplied code's 'test-set' metrics are computed on the optimization split and the reproducibility claim overreaches; still worth a serious referee. read the letter →

arxiv 2509.00038 v1 pith:5JP57L6M submitted 2025-08-22 cs.CL cs.AI

Teo Susnjak This is my paper
classification cs.CLcs.AI
keywords SystematicLiteratureReviewAutomationEvidenceSynthesisLargeLanguageModelsReproducibilityPromptEngineeringContextOptimisationCompilation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Systematic literature reviews are slow and labor-intensive, and large language models could accelerate them—but LLM outputs swing wildly with small prompt changes, which is a serious problem for evidence synthesis. This paper argues that the remedy is to stop treating prompts as hand-crafted text and instead compile them: a researcher declares the review task, codifies the quality standard with gold-standard examples and a metric, runs an automated budgeted search over instruction templates and few-shot exemplars, and saves the tuned prompt as a versioned bundle others can reload. It translates this into a four-component blueprint and a working code example for abstract screening, and frames the work as the first application of declarative prompt tuning to SLR automation. The contribution is a methodological proposal with a proof-of-concept implementation, not a report of measured accuracy gains.

What carries the argument

The moving part is declarative LM-program compilation: pair a fixed language model with a structured task declaration, a small gold-standard validation set, and a metric, then let an automated optimizer search over instruction templates and few-shot exemplars to maximize the metric. This converts prompt writing from an act of crafting into an act of tuning, and the tuned result becomes an object that can be saved, loaded, and audited. The four-step workflow—define the goal, codify the standard, compile the program, package the artifact—is the practical chassis that carries the argument.

What would settle it

Run the saved screening artifact on the same gold-standard abstracts twice—once at packaging time and once after a vendor-side model update—with the same model ID and fixed generation settings, and compare every decision. Any change in decisions or accuracy falsifies the strong claim that the artifact is verifiable and recomputable under a pinned environment; even repeated identical calls that yield different outputs would settle the determinism question.

Watch

Extended reading notes

Core claim

The central claim is that declarative prompt tuning—an approach in which a fixed pre-trained language model is treated as a compiler and a prompt as a program to be optimized against a validation metric—can and should be applied to systematic-review automation. The paper operationalizes the idea through four components: a task declaration plus versioned review context (define the goal), a gold-standard dataset with an accuracy metric (codify the standard), a controlled search over prompts and few-shot exemplars under pinned decode settings (compile the program), and a shareable bundle containing config, prompt, exemplars, metrics, and run logs (package the artifact). The contribution is the

Load-bearing premise

The promise that a saved prompt bundle can be reloaded later with identical results rests on the assumption that a fixed model with fixed decoding settings produces the same output every time, including after the vendor updates the model.

Editorial extensions

If this is right

  • SLR tasks such as abstract screening can be expressed as declarative workflows whose prompts are produced by automated search rather than manual trial and error.
  • Reviewers' scientific intent (the 'what') is separated from model-specific implementation (the 'how'), so pipelines should become less sensitive to which LLM happens to be used.
  • Prompt optimization becomes a data-driven process analogous to hyperparameter tuning, with validation metrics, budgets, and run logs that support audit.
  • Saved artifacts can be reloaded by collaborators to reproduce classifications, making AI-assisted screening steps transparent and recomputable under a pinned environment.
  • The framework positions AI-assisted reviews to satisfy transparency and traceability expectations in evidence synthesis rather than treating prompts as hidden or arbitrary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test not reported in the paper: run the same saved artifact after a vendor-side model update and measure how much screening decisions drift; large drift would undercut the 'verifiable and recomputable' promise even if the workflow itself is sound.
  • The compiled prompt inherits any bias or gaps in the gold-standard examples and the choice of metric, so an auditable artifact is only as trustworthy as the labeled data it was tuned against—a curation standard would be a natural companion to this framework.
  • The compile step could itself be benchmarked against careful manual prompt engineering on the same gold-standard sets; the paper proposes the workflow but does not yet quantify when compilation wins, ties, or loses.
  • If packaged artifacts are shared as community components, the same tuning loop could be re-run per model generation, turning model updates from a reproducibility hazard into a routine recompilation step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper argues that LLM-assisted systematic literature review (SLR) is hampered by fragile, manually crafted prompts, and proposes replacing this 'prompt alchemy' with a declarative, four-step 'prompt compilation' workflow: (1) define the task and context, (2) codify a quality standard with gold-standard examples and a metric, (3) use an automated optimizer (DSPy MIPROv2 or GEPA) to search over instructions and few-shot exemplars, and (4) package the compiled program as a verifiable, auditable artifact containing config, prompt, exemplars, metrics, and run log. The proposal is illustrated through an abstract-screening blueprint (Box 1) and a DSPy code example (Appendix A). The paper explicitly positions itself as an exploratory/prototyping contribution and calls for future empirical testing.

Significance. The paper identifies a genuine problem—prompt fragility in SLR automation—and makes a plausible case that declarative prompt optimization frameworks from general LLM applications could be adapted to SLR. Its strengths include a concrete, code-level implementation, a clear four-component structure, and a useful Table 1 cataloging prompt-induced performance swings. If validated empirically, the framework would be a valuable methodological contribution, moving the field toward programmatic, auditable prompt construction. However, as written the significance is limited: no empirical evaluation is reported, and several load-bearing claims about verifiability, reproducibility, and LLM-agnosticism are not supported by the supplied implementation or evidence.

major comments (4)
  1. [Box 1, Step 4; Appendix A, lines 99-107] The blueprint promises that the packaged artifact contains 'metrics.json with the test-set results,' but the code never creates a held-out test set. The split `valset = gold_standard[:10]`, `trainset = gold_standard[10:]` means the validation split used for prompt selection is what would be reported. With the three gold-standard examples printed, `valset` contains all examples and `trainset` is empty, so the example as written is degenerate. Even with more than ten examples, metrics computed on this valset cannot certify held-out performance; they measure the optimization split. This is a load-bearing mismatch between the advertised 'verifiable quality standard' and the implementation.
  2. [Appendix A, lines 6-11, 114-117; Box 1, Step 4] The claims that 'a collaborator can load the artefact and achieve identical performance' and that results are 'verifiable and recomputable under a pinned environment' are asserted, not demonstrated. Fixing temperature=0.0, top_p=1.0, and seed=42 does not guarantee bit-reproducible outputs from hosted LLM APIs, and model updates can silently change behavior. The code also does not log model IDs, data hashes, or environment snapshots, despite Box 1 saying 'all runs log hashes.' The paper's own Table 1 and reference [4] document cross-model and cross-prompt divergence, making the determinism assumption especially fragile. Without a concrete reproducibility protocol and evidence, the packaged artifact is documentation rather than verification.
  3. [Section 3; Box 1, Step 3] The framework is described as searching for 'an LLM-agnostic prompt configuration,' but the compilation is performed under a pinned model build (e.g., `openai/gpt-4o-mini` in Appendix A) with fixed decoding parameters. The optimized prompt is therefore likely model-specific and prompt-transfer across models is not established. This is an internal tension in the central claim: the method pins a model to compile a prompt, yet declares the output LLM-agnostic. The claim should be either removed or replaced with a qualified statement about portability.
  4. [Sections 3-4] The abstract and conclusion say the framework 'demonstrates' applicability and utility, and the conclusion calls the code a 'proof-of-concept, reproducible implementation.' However, no empirical evaluation is reported: there are no measured accuracy values, no comparison to manual prompting, and no demonstration that the compiled prompt satisfies any predefined quality standard. The paper is honest that future work should test the approach, but the wording overstates what is shown. This is acceptable if the contribution is framed as a methodological blueprint; the claims should be softened to 'proposes' and 'illustrates with a code example,' or an evaluation should be added.
minor comments (4)
  1. [Appendix A, line 5] Typo in comment: 'determninistic' should be 'deterministic'; also the dict keys in lines 7-10 have stray spaces around the quoted strings in the printed listing.
  2. [Section 3] The paragraph after the MIPROv2 example contains a duplicated and grammatically broken sentence: 'This positions the method ... This process is can be seen as analogous to...' The same analogy to hyperparameter tuning is already stated earlier in the section; the text should be consolidated.
  3. [Section 4] The claim that 'this represents the first application of such declarative techniques to evidence synthesis workflows' is asserted without a prior-art search. Given the rapid growth of LLM/SLR literature, this novelty claim should be softened or supported with a systematic literature search.
  4. [References [19]-[21]; Section 3] The text groups GRPO [20] with DSPy and GEPA as 'prompt optimisation' frameworks. GRPO is a reinforcement-learning training method, not a declarative prompt-tuning framework. The citation should be recontextualized or replaced with a more directly relevant reference.

Circularity Check

1 steps flagged · score 4.0 of 10

Quality certificate is computed on the tuning split, not an independent held-out test set.

  1. fitted input called prediction [Box 1, Step 4; Appendix A, lines 99-107]
    "The bundle contains config.yaml (task spec and run controls), prompt.txt and exemplars.json, metrics.json with the test-set results, and a run log. ... Results are verifiable and recomputable under a pinned environment. | valset = gold_standard[:10], # validation set with a simple approach assuming one has 10 samples / trainset = gold_standard[10:], # training set assuming one has more than 10 samples"

    The gold-standard examples are the only task data introduced, and the compiler is run on exactly those examples: valset = gold_standard[:10] and trainset = gold_standard[10:]. No held-out test set is ever constructed. Box 1 nevertheless packages 'metrics.json with the test-set results' as part of the verifiable artefact. If metrics.json is produced from the same split that the optimizer used for selection, the reported quality certificate is the optimization objective evaluated on the optimization data. Thus the advertised independent verification of the predefined quality standard reduces, in the supplied implementation, to the fitting inputs themselves.

full rationale

The central proposal — adapting declarative prompt-optimization frameworks (DSPy, GEPA) to SLR automation — is not circular: the gold-standard labels are external expert judgments, the optimizers are external tools, and the paper does not claim measured accuracy gains. The self-citations ([2], [4]) document LLM sensitivity but are corroborated by multiple independent references and are not load-bearing for the framework. The one concrete circularity is in the artifact certi?cation: Box 1 promises 'test-set results' in metrics.json, while Appendix A only splits gold_standard into validation and training sets and never creates a held-out test set, so a user following the code would certify quality on the same examples used to tune the prompt. The novelty claim ('first application') is unsupported by a prior-art search but is a correctness risk, not circularity. Overall, the core methodology is independent, but this verification step partially reduces to its own inputs, giving a moderate score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The 'compiler', 'LM program', and 'verifiable digital artefact' are organizational metaphors for existing DSPy/GEPA components (optimizer.compile, module.save/load) and have no independent falsifiable handle. The load-bearing assumptions are about generalization and determinism, both untested.

free parameters (3)
  • Decoding controls (temperature, top_p, seed, max_tokens) = 0.0, 1.0, 42, 1024
    Hand-chosen in Appendix A (lines 6-11) to 'maximise determnistic responses'; the paper's reproducibility claims rest on these, but hosted LLM APIs do not guarantee bit-identical output at these settings.
  • Gold-standard set size N = 10-50 examples (3 shown)
    Box 1 step 2 and the code comment (Appendix A line 84) prescribe 10-50 expert-labeled abstracts; whether this is sufficient for the optimizer to reach the 'predefined quality standard' is untested.
  • Tuning budget B and few-shot exemplar count k = unspecified
    Box 1 step 3 defines a budget B of evaluations and up to k few-shot exemplars without concrete values; these control what the compiler finds and are left to the researcher.
assumptions (3)
  • domain assumption A prompt configuration optimized on a small gold-standard validation set generalizes to unseen abstracts.
    This is the framework's core hypothesis (Section 3, Box 1 steps 2-3). The paper reports no experiments testing generalization.
  • domain assumption Pinning the model build and decoding parameters makes LLM outputs deterministic and the compiled artifact recomputable.
    Invoked in Box 1 steps 3-4 and Appendix A lines 6-11 and 115. Contradicted in practice by API non-determinism and model updates; the paper's own Table 1 and reference [4] document cross-model and cross-prompt divergence.
  • domain assumption Gold-standard labels described as 'not part of the study' are sufficient to prevent contamination of the packaged quality metrics.
    Box 1 step 2 versus step 4: the code splits one gold_standard list into valset/trainset and packages metrics.json; no separate held-out test evaluation exists in the shipped example, so the packaged 'test-set results' are optimization-set metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis." pith.science (2026). https://pith.science/paper/5JP57L6M

@misc{pith2026250900038,
  author       = {Pith},
  title        = {Pith review of: Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JP57L6M}},
  note         = {Machine review of arXiv:2509.00038}
}
read the original abstract

Large language models (LLMs) offer significant potential to accelerate systematic literature reviews (SLRs), yet current approaches often rely on brittle, manually crafted prompts that compromise reliability and reproducibility. This fragility undermines scientific confidence in LLM-assisted evidence synthesis. In response, this work adapts recent advances in declarative prompt optimisation, developed for general-purpose LLM applications, and demonstrates their applicability to the domain of SLR automation. This research proposes a structured, domain-specific framework that embeds task declarations, test suites, and automated prompt tuning into a reproducible SLR workflow. These emerging methods are translated into a concrete blueprint with working code examples, enabling researchers to construct verifiable LLM pipelines that align with established principles of transparency and rigour in evidence synthesis. This is a novel application of such approaches to SLR pipelines.

Figures

Figures reproduced from arXiv: 2509.00038 by the authors.

Figure 1
Figure 1. Four Components of the Declarative Language Model-program Tuning Framework [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

  1. [4]

    What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction

    Lingbo Li, Anuradha Mathrani, and Teo Susnjak. What level of automation is "good enough"? a benchmark of large language models for meta-analysis data extraction, 2025. URLhttps://arxiv.org/abs/2507.15152

    work page Pith review arXiv 2025

  2. [1]

    Meerpohl, and Angelika Eisele-Metzger

    Judith-Lisa Lieberum, Markus Töws, Maria-Inti Metzendorf, Felix Heilmeyer, Waldemar Siemens, Christian Haverkamp, Daniel Böhringer, JoergJ. Meerpohl, and Angelika Eisele-Metzger. Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review.Journal of Clinical Epidemiology, 181:111746, 2025. doi:10.1016/j...

    arXiv 2025

  3. [2]

    Teo Susnjak, Peter Hwang, Napoleon Reyes, Andre L. C. Barczak, Timothy McIntosh, and Surangika Ranathunga. Automating research synthesis with domain-specific large language model fine-tuning.ACM Trans. Knowl. Discov. Data,19(3),March2025. ISSN1556-4681. doi:10.1145/3715964. URL https://doi.org/10.1145/3715964

    doi:10.1145/3715964

  4. [3]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324, 2023

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting.arXiv preprint arXiv:2310.11324, 2023

    arXiv 2023

  5. [5]

    Responsible ai in the generative era, May 2023

    Michael Kearns. Responsible ai in the generative era, May 2023. URLhttps://www.amazon.science/blog/ responsible-ai-in-the-generative-era . Accessed: 2025-08-01

    work page 2023

  6. [6]

    Benchmarking prompt sensitivity in large language models

    Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. Benchmarking prompt sensitivity in large language models. InEuropean Conference on Information Retrieval, pages 303–313. Springer, 2025

    2025

  7. [7]

    A reproducibility and generalizability study of large language models for query generation

    Moritz Staudinger, Wojciech Kusa, Florina Piroi, Aldo Lipani, and Allan Hanbury. A reproducibility and generalizability study of large language models for query generation. InProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, pages 186–196, 2024

    work page 2024

  8. [8]

    Efficacy of large language models for systematic reviews

    Aaditya Shah, Shridhar Mehendale, and Siddha Kanthi. Efficacy of large language models for systematic reviews. In2024 2nd International Conference on Foundation and Large Language Models (FLLM), pages 29–35. IEEE, 2024

    work page 2024

Show all 21 references
  1. [9]

    Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

    Fabio Dennstädt, Johannes Zink, Paul Martin Putora, Janna Hastings, and Nikola Cihoric. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Systematic reviews, 13(1):158, 2024

    2024

  2. [10]

    Prompting is all you need: Llms for systematic review screening.medRxiv, pages 2024–06, 2024

    Christian Cao, Jason Sang, Rohit Arora, Robbie Kloosterman, Matt Cecere, Jaswanth Gorla, Richard Saleh, David Chen, Ian Drennan, Bijan Teja, et al. Prompting is all you need: Llms for systematic review screening.medRxiv, pages 2024–06, 2024

    2024

  3. [11]

    Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation.BMC medical research methodology, 25(1):130, 2025

    Fouad Trad, Ryan Yammine, Jana Charafeddine, Marlene Chakhtoura, Maya Rahme, Ghada El-Hajj Fuleihan, and Ali Chehab. Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation.BMC medical research methodology, 25(1):130, 2025

    2025

  4. [12]

    Development of prompt templates for large language model-driven screening in systematic reviews.Annals of Internal Medicine, 178(3):389–401, 2025

    Christian Cao, Jason Sang, Rohit Arora, David Chen, Robert Kloosterman, Matthew Cecere, Jaswanth Gorla, Richard Saleh, Ian Drennan, Bijan Teja, et al. Development of prompt templates for large language model-driven screening in systematic reviews.Annals of Internal Medicine, 17...

    2025

  5. [13]

    Automation of systematic reviews with large language models

    Christian Cao, Rohit Arora, Paul Cento, Katherine Manta, Elina Farahani, Matthew Cecere, Anabel Selemon, Jason Sang, Ling Xi Gong, Robert Kloosterman, et al. Automation of systematic reviews with large language models. medRxiv, pages 2025–06, 2025

    2025

  6. [14]

    Language models for data extraction and risk of bias assessment in complementary medicine

    Honghao Lai, Jiayi Liu, Chunyang Bai, Hui Liu, Bei Pan, Xufei Luo, Liangying Hou, Weilong Zhao, Danni Xia, Jinhui Tian, et al. Language models for data extraction and risk of bias assessment in complementary medicine. npj Digital Medicine, 8(1):74, 2025

    2025

  7. [15]

    Qusai Khraisha, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. Can large language models replace humans in systematic reviews? evaluating gpt-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages.Resea...

    2024

  8. [16]

    Prompt engineering in consistency and reliability with the evidence-based guideline for llms.NPJ digital medicine, 7(1):41, 2024

    Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. Prompt engineering in consistency and reliability with the evidence-based guideline for llms.NPJ digital medicine, 7(1):41, 2024. 5 Compiling Prompts, Not Crafting Them A Preprint

    2024

  9. [17]

    Assessing the risk of bias in randomized clinical trials with large language models.JAMA Network Open, 7(5):e2412687–e2412687, 05 2024

    Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang, Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao, Xiaoman Wang, Ming Liu, Jhalok Ronjan Talukdar, Jinhui Tian, Kehu Yang, and Janne Estill. Assessing the risk of bias in randomized c...

    2024

  10. [18]

    Angelika Eisele-Metzger, Judith-Lisa Lieberum, Markus Toews, Waldemar Siemens, Felix Heilmeyer, Christian Haverkamp, Daniel Boehringer, and Joerg J Meerpohl. Exploring the potential of claude 2 for risk of bias assessment: Using a large language model to assess randomized cont...

    2024

  11. [19]

    Optimizinginstructionsanddemonstrationsformulti-stagelanguagemodelprograms

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. Optimizinginstructionsanddemonstrationsformulti-stagelanguagemodelprograms. In 2024Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Hyb...

    2024

  12. [20]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  13. [21]

    temperature

    Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. Gepa: Reflecti...

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.