REVIEW 4 major objections 5 minor 22 references
Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that DSPy's programmatic prompt optimization can beat handcrafted prompts in several LLM applications, with the largest gain taking contradiction-evaluation accuracy from 46.2% to 64.0%, but the benefit is task-dependent.
desk verdict The headline accuracy gains are much noisier than the abstract implies, but this is an honest, clearly written DSPy case study that deserves a rigorous peer review round, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is DSPy's declarative programming model: each task is defined by a signature (typed input/output fields with descriptions) and modules, and an optimizer searches over candidate instructions and bootstrapped few-shot examples to maximize a user-supplied metric. The key optimizers are MIPROv2, a Bayesian optimizer that proposes and selects the best combination of instructions and demonstrations, and BootstrapFewShotWithRandomSearch, which searches over example sets. The paper also introduces CustomMIPROv2, a two-stage variant that first extracts constraints and edge cases from demonstrations, then uses them to guide instruction generation; this constraint-guided search produced the best prompt-evaluator result.
What would settle it
Re-run the contradiction-evaluation experiment with a held-out test set of at least 100 examples and compute a confidence interval on the accuracy gain; if the 46.2% to 64.0% improvement (about 2.3 correct examples out of 13) does not reproduce beyond sampling error, the paper's central claim that DSPy optimization improves this task would be unsupported.
Extended reading notes
Core claim
The central discovery is that a systematic, programmatic prompt optimizer like DSPy can match or beat carefully handcrafted prompts in real-world LLM applications, especially when the optimizer tunes instructions and few-shot demonstrations together. In the paper's experiments, zero-shot MIPROv2 with a signature that explicitly names the three contradiction types raised prompt-evaluation accuracy from 46.2% to 64.0%; adding custom constraints and tips in CustomMIPROv2 pushed it to 76.9%. For routing agents, CustomMIPROv2 improved accuracy from 85.7% to 90.5%, and for code generation a simple-format optimized instruction improved 87.5% to 90.0%, though the DSPy-formatted version and InferRules did not beat the baseline. The paper argues this is evidence that prompts can be managed as code—structured inputs, outputs, and optimizable parameters—while cautioning that the same approach does not guarantee gains on every task.
Load-bearing premise
The main load-bearing premise is that the measured accuracy differences reflect genuine optimization effects rather than noise from very small test sets and from picking the best of many optimization trials on small validation mini-batches.
Editorial extensions
If this is right
- If DSPy's optimization is as effective as these cases suggest, developers can replace manual prompt trial-and-error with typed signatures and a metric, letting the compiler propose the prompt.
- Jointly optimizing instructions and few-shot examples should be the default strategy, since the paper's best results came from combining the two, consistent with the cited MIPROv2 work.
- Production teams should verify gains per task rather than assume a universal lift: the same optimizer improved some use cases and left others at baseline.
- Extracting an optimized prompt out of DSPy's runtime can degrade quality, so treating prompts as code works best when the program, not the string, is what gets deployed.
- A weaker model with an optimized prompt did not close the gap to a stronger model in the routing use case, so prompt optimization complements model selection rather than replacing it.
Reading between the lines
- The 17.8-point gain on a 13-example test set is about 2.3 examples; a reasonable next step would be to re-run the contradiction task with a larger held-out set and confidence intervals, since the current numbers may not separate a real effect from sampling noise.
- The success of naming contradiction types in the signature suggests that the search space the optimizer sees matters as much as the search algorithm; this points to a general recipe of enriching signatures with explicit failure categories before optimizing.
- CustomMIPROv2's constraint-extraction stage could be developed into a reusable pattern for other domains, where the optimizer distills demonstrations into rules before proposing instructions, reducing the need for human tips.
- If small-sample effects are confirmed, the paper's 'prompts as code' thesis still stands but needs stronger empirical grounding: reporting effect sizes and uncertainty would make multi-case optimization studies comparable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a multi-use-case study of DSPy-based prompt optimization across five production-oriented tasks: jailbreak detection, hallucination detection in Pandas code, Pandas code generation, a routing agent, and a prompt evaluator. The central claim is that DSPy's systematic prompt optimization can improve LLM performance, especially when instructions and few-shot examples are optimized together, while gains vary by task. The strongest reported result is a prompt-evaluation accuracy improvement from 46.2% to 64.0% using zero-shot MIPROv2, and the paper also explores a custom variant (CustomMIPROv2) that incorporates user-provided constraints and tips. The paper is transparent about negative results (e.g., InferRules showing no gain; a cheaper model not matching a stronger one) and acknowledges that extracted optimized prompts may degrade outside DSPy.
Significance. If the results were statistically robust, the paper would provide a useful practical demonstration that prompt optimization can be treated as a programmatic, repeatable activity rather than manual trial-and-error, with concrete evidence from real-world scenarios. The study's honest reporting of failures and its attempt to evaluate prompts outside the DSPy architecture are strengths, as is the comparison of several optimizers. However, the central quantitative evidence is weakened by very small test sets (13–50 examples), selection of the best instruction over multiple optimization trials without multiple-comparison correction, and the absence of confidence intervals or significance tests. As presented, the headline gains are consistent with sampling noise, so the strength of the central claim exceeds what the evidence supports. The paper's main value lies in its catalog of use cases and qualitative lessons rather than in the specific accuracy deltas.
major comments (4)
- [§3.5, 'Program and Results'] The headline improvement from 46.2% to 64.0% in the contradiction-evaluation task is measured on a test set of only 13 examples. This corresponds to a shift from roughly 6/13 to 8/13 correct, i.e., two or three examples. A binomial 95% confidence interval for 13 items spans approximately ±25 percentage points, so the observed difference cannot be distinguished from sampling noise. The claim of a 'substantial performance increase' is not supported without per-item counts or a paired significance test.
- [§3.4, 'Program and Results'] The routing-agent improvement from 85.71% to 90.47% is reported on a 21-conversation test set, which corresponds to 18/21 versus 19/21 correct, a one-example difference. The research question asks whether an under-performing prompt can be optimized to achieve 'significantly better results,' but no confidence interval, significance test, or effect-size measure is provided. This result does not by itself establish a meaningful improvement.
- [§3.4 and §3.5, 'Optimization'] The optimization procedure selects the best instruction after 15 trials (Section 3.4) or 10 trials (Section 3.5), each scored on a small validation mini-batch (15 examples in Section 3.4; the full validation set of 45 in Section 3.5). Selecting the maximum over many noisy candidate evaluations introduces an upward bias, and no multiple-comparison correction or holdout-based significance test is applied. The reported test-set numbers therefore conflate true optimization gains with selection effects.
- [§3.2, 'Results' and §3.4, 'Program and Results'] The manuscript repeatedly states that tests were repeated three times, but only averaged percentages are reported, with no per-run counts, standard deviations, or raw predictions. Neither code nor data is released. This makes it impossible for readers to recompute binomial confidence intervals or to assess the stability of the reported accuracies, directly undermining the reproducibility of the central claim.
minor comments (5)
- [Figure 1 caption] The caption refers to 'RuleInfer' while the text consistently uses 'InferRules' (Section 3.3); please unify the terminology.
- [Table 1] The 'Not answered' column is reported but it is unclear how blocked responses were treated in the accuracy, precision, recall, and F1 calculations; please specify the handling rule.
- [§3.4, 'Optimization'] The text says 'allowing CustomMIPRO to explore the 6 fixed provided tips twice' but the optimizer is named 'CustomMIPROv2' elsewhere; please correct the name for consistency.
- [§3.5, 'Program and Results'] The narrative reports that the first optimized instruction reached 56.3%, then states 'The new prompt achieved 53.80%' for the updated signature; please clarify which signature corresponds to which baseline and confirm that all are evaluated on the same 13-example test set.
- [§4, 'Discussion'] The assertion that optimized prompts rely on DSPy's internal behavior and can 'hurt' quality when extracted is stated after a discussion with the DSPy team; this claim should be substantiated with an empirical comparison or a citation, since it directly concerns the generalization of the reported gains.
Circularity Check
No circularity: the paper is an empirical evaluation with no derivation chain that reduces to its own inputs.
full rationale
This paper is an empirical, multi-use-case evaluation of the DSPy prompt-optimization framework. It does not present a mathematical derivation, a fitted model, or a theoretical claim whose conclusion is equivalent to an input by construction. The reported results are before/after accuracy measurements on held-out test sets, and the optimization objective (e.g., exact match, recall, or a hand-defined weighted score) is not identical to the reported headline metric in any reductionist sense. The authors do not fit a parameter and then rename it a prediction; they run optimizers that propose instructions and examples, select candidates using a validation mini-batch, and then measure accuracy on separate test examples. The custom tips and constraints added to CustomMIPROv2 guide instruction generation, but the subsequent evaluation is performed against ground-truth labels (e.g., contradiction labels in Section 3.5), not against the constraints themselves, so the result is not forced by definition. The citations to Tan et al. and Opsahl-Ong et al. are to external prior work and benchmarks, not to the present authors' own unverified results, so there is no load-bearing self-citation chain. The manuscript's own discussion explicitly notes that extracted optimized prompts can degrade outside DSPy, which is a limitation on practical generalization rather than a circular step. The main weakness of the paper is statistical: the 13-example and 21-example test sets, combined with best-of-N selection over multiple optimization trials and no confidence intervals, mean the headline gains may be consistent with sampling noise. That is a validity or correctness concern, not a circularity concern, and the review instructions require concrete reduction-by-definition evidence before flagging circularity. No such evidence is present here, so the appropriate score is 0.
Assumptions & free parameters
free parameters (6)
- Contradiction score threshold =
0.6
- Number of optimization trials =
15 (router), 10 (prompt evaluator)
- Validation mini-batch size =
15 (20% of validation)
- Number of few-shot examples =
0, 4, 6, or 8 depending on task
- Panel of Experts criteria weights =
not reported
- Temperature for prompt-generating model =
0.5
assumptions (5)
- domain assumption LLM-as-a-Judge provides reliable evaluation of generated code
- domain assumption DSPy optimizers generalize from small validation sets
- domain assumption Custom datasets are representative of real-world production scenarios
- standard math Repeated trials provide consistent estimates
- ad hoc to paper The custom routing constraint is a correct policy
Cite this review
Pith. "Pith review of Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy." pith.science (2026). https://pith.science/paper/OO2Q3X2Z
@misc{pith2026250703620,
author = {Pith},
title = {Pith review of: Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy},
year = {2026},
howpublished = {\url{https://pith.science/paper/OO2Q3X2Z}},
note = {Machine review of arXiv:2507.03620}
}
read the original abstract
Although prompt engineering is central to unlocking the full potential of Large Language Models (LLMs), crafting effective prompts remains a time-consuming trial-and-error process that relies on human intuition. This study investigates Declarative Self-improving Python (DSPy), an optimization framework that programmatically creates and refines prompts, applied to five use cases: guardrail enforcement, hallucination detection in code, code generation, routing agents, and prompt evaluation. Each use case explores how prompt optimization via DSPy influences performance. While some cases demonstrated modest improvements - such as minor gains in the guardrails use case and selective enhancements in hallucination detection - others showed notable benefits. The prompt evaluation criterion task demonstrated a substantial performance increase, rising accuracy from 46.2% to 64.0%. In the router agent case, the possibility of improving a poorly performing prompt and of a smaller model matching a stronger one through optimized prompting was explored. Although prompt refinement increased accuracy from 85.0% to 90.0%, using the optimized prompt with a cheaper model did not improve performance. Overall, this study's findings suggest that DSPy's systematic prompt optimization can enhance LLM performance, particularly when instruction tuning and example selection are optimized together. However, the impact varies by task, highlighting the importance of evaluating specific use cases in prompt optimization research.
Figures
Reference graph
Works this paper leans on
-
[1]
Chen, B., Zhang, Z., Langren´ e, N., Zhu, S.: Unleashing the potential of prompt engineering for large language models. Patterns 6, 101260 (2025). https://doi.org/10.1016/j.patter.2025.101260
-
[2]
Weights & Biases Blog (2025-02-12)
Davies, D.: LLM evaluation: Metrics, frameworks, and best practices. Weights & Biases Blog (2025-02-12). https: //wandb.ai/onlineinference/genai-research/reports/ LLM-evaluation-Metrics-frameworks-and-best-practices--VmlldzoxMTMxNjQ4NA
work page 2025
-
[3]
et al.: Building Guardrails for Large Language Models
Dong, Y. et al.: Building Guardrails for Large Language Models. arXiv preprint arXiv:2402.01822 (2024). https://doi.org/10.48550/arXiv. 2402.01822
-
[4]
et al.: A Survey on LLM-as-a-Judge
Gu, J. et al.: A Survey on LLM-as-a-Judge. arXiv preprint arXiv:2411.15594 (2025). https://doi.org/10.48550/arXiv.2411.15594
-
[5]
et al.: EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers
Guo, Q. et al.: EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers. arXiv preprint arXiv:2309.08532 (2025) https://doi.org/10.48550/arXiv.2309.08532
-
[6]
arXiv preprint arXiv:2311.10117 (2023)
Hsieh, C.-J., Si, S., Yu, F.X., Dhillon, I.S.: Automatic Engineering of Long Prompts. arXiv preprint arXiv:2311.10117 (2023). https://doi.org/10. 48550/arXiv.2311.10117
-
[7]
arXiv preprint arXiv:2307.10169 (2023)
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., McHardy, R.: Challenges and Applications of Large Language Models. arXiv preprint arXiv:2307.10169 (2023). https://doi.org/10.48550/arXiv.2307.10169
-
[8]
Kiseleva, J., Arabzadeh, N.: How to Assess Utility of LLM-powered Applica- tions? Microsoft Autogen Blog (2023-11-20). https://microsoft.github. io/autogen/0.2/blog/2023/11/20/AgentEval/
work page 2023
Show all 22 references
-
[9]
et al.: From Generation to Judgment: Opportunities and Challenges of LLM-as-a-Judge
Li, D. et al.: From Generation to Judgment: Opportunities and Challenges of LLM-as-a-Judge. arXiv preprint arXiv:2411.16594 (2025). https://doi. org/10.48550/arXiv.2411.16594
2025 doi
- [10]
- [11]
-
[12]
Microsoft Learn (2025-05.28)
Microsoft: Content filtering overview. Microsoft Learn (2025-05.28). https://learn.microsoft.com/en-us/azure/ai-services/openai/ concepts/content-filter
2025
- [13]
-
[14]
arXiv preprint arXiv:2402.07927 (2025)
Sahoo, P., Singh, A.K., Saha, S., Jain, V., Mondal, S., Chadha, A.: A Sys- tematic Survey of Prompt Engineering in Large Language Models: Tech- niques and Applications. arXiv preprint arXiv:2402.07927 (2025). https: //doi.org/10.48550/arXiv.2402.07927
-
[15]
In: 2023 10th International Conference 20 F
Su, H., Ai, J., Yu, D., Zhang, H.: An Evaluation Method for Large Language Models’ Code Generation Capability. In: 2023 10th International Conference 20 F. Lemos et al. on Dependable Systems and Their Applications (DSA), pp. 831–838. IEEE (2023). https://doi.org/10.1109/DSA593...
2023
- [16]
-
[17]
Sourcery.ai Blog (2024- 05-13)
Thapen, N.: Better LLM Prompting using the Panel-of-Experts: How role- playing a panel discussion can improve LLM results. Sourcery.ai Blog (2024- 05-13). https://sourcery.ai/blog/panel-of-experts/
2024
- [18]
- [19]
- [20]
- [21]
- [22]
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.