{"id":"395e9ad5-032a-44ab-80bf-7a2082c31db6","arxiv_id":"2508.03678","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Prompt specificity substantially raises LLM pass@1 on code benchmarks, with the largest gains on parallel-programming tasks.","lead":"PartialOrderEval gives every code-generation problem a ladder of prompts from a bare function signature to a heavily detailed spec, and measures how pass@1 climbs as prompts get more specific. On the parallel-programming benchmark ParEval, adding detail lifted pass@1 by up to 30 points, so part of the reported gap is a prompting gap, not just missing domain knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are anchored by ptop, which is generated from the reference solution and edited until Qwen2.5-Coder-14B passes, so 'prompt specificity' is confounded with leaking the solution into the prompt; the 0.30 gain may not measure specificity.","rationale":"The central empirical direction is credible: pass@1 does rise with more detailed prompts, and the paper's use of multiple augmentation families and model families is a genuine strength. However, the strongest claim that 'a substantial portion of the ParEval performance gap is attributable to prompt detail rather than missing domain knowledge' rests on the construction of ptop. The reader's weakest-assumption identifies the test-based selection of ptop, which is correct and related. My concern goes one step further: ptop is derived from the reference solution itself, so the 'maximally detailed prompt' contains the algorithm's steps. The ifft example in Appendix E makes this concrete: the prompt literally enumerates the four steps of the inverse FFT implementation, which is the answer. The paper's own qualitative analysis then identifies 'Implementation Sketch or Step Breakdown' as one of the key drivers, effectively acknowledging that the gain comes from supplying the solution structure. This does not make the experimental measurements wrong, but it does mean the 0.30 gain conflates prompt specificity with answer leakage. The right endpoint of every curve is therefore not a neutral measurement of how much detail helps; it is a measurement of how much of the reference solution can be inferred from a summary of ptop. The threshold τ = 0.8 and the use of Qwen2.5-Coder-14B as the tuning model further ensure that the right endpoint is optimized for the very model whose improvement is reported. A solution-blind, specification-only ptop would settle whether the claimed gains survive when the prompt is detailed without giving away the algorithm. Because this is a concrete, addressable experimental condition rather than a fundamental flaw, the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed, but the stated condition should be made explicit.","tokens_in":22889,"tokens_out":5321,"duration_ms":59055,"concrete_test":"Regenerate ptop for all 60 ParEval-OMP problems with GPT-4.1 (or an expert annotator) using only the original problem statement, explicitly forbidding implementation steps, algorithm names, or pseudocode, and allowing only specification-level detail such as I/O formats, edge cases, constraints, and examples; do not perform any pass@1-based editing. Re-run greedy pass@1 for Qwen2.5-Coder-14B and compare with the original ParEval prompt. If the absolute gain drops materially below the reported 0.30 (e.g., below 0.10-0.15), the headline effect is largely solution leakage rather than prompt specificity. As a corroborating check, also generate ptop tuned to pass@1 >= 0.8 on Llama-3.1-8B instead of Qwen2.5-Coder-14B and evaluate Qwen2.5-Coder-14B; if its ptop score falls, the reported ceiling is selection-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 builds ptop by giving GPT-4.1 the original prompt plus the reference solution and asking for a step-by-step description (Appendix B), then manually editing or regenerating until Qwen2.5-Coder-14B reaches pass@1 ≥ 0.8. The published example in Appendix E shows ptop for ifft states exactly: conjugate, apply forward FFT, conjugate again, scale by 1/N. That is the algorithm from the reference solution, not merely a more precise task specification. Since ptop is the right endpoint of every performance curve and the source of all three augmentation families, the monotone rise in pass@1 and the headline 0.30 absolute ParEval-OMP gain quantify how much of the gold solution can be recovered from summaries of ptop. Section 5 confirms this reading: the fastest-growing taxonomy themes include 3.4 Implementation Sketch or Step Breakdown and 1.5 Core Behaviour, meaning the answer structure is being supplied in the prompt. The paper therefore does not separate 'prompt specificity' from 'domain knowledge placed in the prompt'; a model that can follow a step-by-step solution sketch may still lack the domain knowledge needed to produce that sketch. The test-based threshold τ = 0.8 compounds this: ptop is selected on the same Qwen2.5-Coder-14B model whose gains are reported, so the ceiling is model-optimized. Intermediate summary levels were not individually test-selected and still show a monotone trend, so the direction of the result is plausible; but the magnitude and the 'detail versus knowledge' interpretation are not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PARTIALORDEREVAL, a framework that augments code-generation benchmarks with a partially ordered set of prompts spanning from minimal to maximally detailed descriptions, and applies it to HumanEval, ParEval-Serial, and ParEval-OpenMP. Maximally detailed prompts ptop are generated by GPT-4.1 from the original prompt plus the reference solution and manually edited or regenerated until Qwen2.5-Coder-14B reaches pass@1 ≥ 0.8; three families of less detailed variants (LLM summarization at word limits, paragraph sampling, and sentence-block masking) are derived from ptop. The authors report pass@1 curves for four Qwen and four Llama models, find that performance generally rises with prompt detail, with up to a 0.30 absolute gain on ParEval-OMP, and use LLM-based taxonomy annotation to identify promising prompt-detail themes.","tokens_in":23166,"tokens_out":5172,"duration_ms":56979,"significance":"If the result holds, the paper is significant because it complicates the interpretation that ParEval underperformance reflects missing domain knowledge alone, and it contributes a reusable methodology and dataset for prompt-sensitivity evaluation. The paper is also unusually transparent about experimental budgets, and the partial-order prompt generation is a concrete artifact that others can build on. However, the significance is substantially tempered by the construction of ptop from the reference solution and its selection on the same model whose gains are reported; this makes the magnitude of the reported gains not a clean measure of prompt specificity. The qualitative theme analysis is correlational, and the performance curves lack uncertainty quantification.","major_comments":[{"comment":"The right endpoint of every performance curve is selected on the outcome being measured. In §3.3, ptop is generated by GPT-4.1 from the original prompt plus the reference solution (Appendix B), then manually edited or regenerated until Qwen2.5-Coder-14B reaches pass@1 ≥ 0.8, with τ = 0.8 imposed by the formal condition in §3.2. Since ptop is the anchor for all three augmentation families and the source of the 0.30 absolute ParEval-OMP gain reported in §4.3, the rise from pbot to ptop is partly by construction. Appendix E makes the confound concrete: the ptop example for ifft states the exact algorithm from the reference solution (conjugate, forward FFT, conjugate, scale by 1/N), which is a solution sketch rather than a more precise task specification. The manuscript should either bound the confound (e.g., construct ptop without access to the reference solution, or ablate prompts that contain step-by-step solution structure), or reframe the claims as measuring 'how much of the gold solution can be recovered from summaries of ptop.' As written, the central attribution of the gains to prompt specificity rather than to domain knowledge inserted into the prompt is not established.","section":"§3.2–3.3 and §4.3"},{"comment":"The performance curves are computed from a single greedy decoding per prompt (Appendix A.1, temperature = 0), and no error bars, bootstrap intervals, or significance tests are reported. With 60 problems per ParEval subset, a 0.05 absolute difference in pass@1 is within the noise of a binomial proportion (standard error ≈ 0.06), so several intermediate-level differences in Figures 3 and 4 may not be reliable. More importantly, the paper's own Figure 3 shows a HumanEval decline beyond L=200 (e.g., Qwen2.5-Coder-14B at 0.921 for L=200 vs 0.860 at ptop), which contradicts the conclusion's statement that increased prompt specificity 'consistently improves pass@1.' The monotonicity claim should be replaced by a qualified statement, and confidence intervals or per-problem breakdowns should be provided.","section":"§4.1–4.3 and Figure 3"},{"comment":"The detail metric D is defined only syntactically (word limit, retention ratio, masking ratio), yet the partial order pbot <D p <D ptop is used to interpret pass@1 as a function of 'specificity.' The manuscript does not validate that these proxies are monotonic in information content. Appendix E shows that paragraph sampling at rp=0.2 can retain the later solution steps for ifft while dropping the earlier ones, while rp=0.8 retains all four steps; a 200-word LLM summary can omit a crucial constraint that a shorter summary includes. Section 3.2 states 'as long as the metric D is reasonably defined, this empirically generally holds' but gives no evidence for the reasonableness of the ordering. The authors should either add a validation of the ordering (e.g., human or model-based informativeness judgments on a sample) or explicitly define D as a syntactic proxy and not claim that the partial order orders information content.","section":"§3.2 and Appendix E"},{"comment":"The qualitative analysis is correlational, and the causal language should be softened. Section 5 reports that themes 1.5, 1.3, 1.4, 2.3, and 3.4 grow in frequency with word limit and concludes they 'might play particularly influential roles' and recommends prioritizing them in prompt design. Because the annotation is performed by an LLM (Claude Sonnet 4) using a taxonomy generated by another LLM (o3), and because theme frequency is confounded with summary length, these observations do not establish that adding those themes improves pass@1. The Limitations section acknowledges potential taxonomy bias but does not address the correlational nature of the theme-frequency analysis, and the abstract's phrase 'key drivers' overstates the evidence. A targeted ablation (adding or removing individual themes at fixed length), or at least explicit acknowledgment that these are hypotheses rather than drivers, is needed.","section":"§5"}],"minor_comments":[{"comment":"There are several grammar and typographical errors ('Rather then', 'thrends', 'serval limitations', 'a increase', 'generatd') that should be corrected before publication.","section":"Abstract and §1"},{"comment":"The radar chart in Figure 5 and the appendix plots in Figure 6 are difficult to read; specify the number of prompts per length (284) in the caption and clarify that ptop corresponds to 'full words' in Figure 7.","section":"Figure 5 and Figure 6"},{"comment":"The definition of pass@1 does not state whether the hidden tests are exactly the original benchmark tests and whether partial credit is possible; please clarify the evaluation protocol.","section":"§4.2"},{"comment":"The Limitations section should also mention the lack of statistical uncertainty and the reference-solution confound in ptop construction, rather than only augmentation strategies, qualitative-analysis bias, and low-resource languages.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a natural-language-processing or software-engineering venue and the partial-order prompt dataset is a potentially useful artifact. I do not see grounds for rejection: the direction of the result is plausible, and independent support comes from the intermediate summary levels that were not individually test-selected. However, the main claims currently rest on the ptop anchor, which is constructed from the reference solution and selected on the same model whose gains are reported, and the curves have no uncertainty quantification. These are fixable with additional experiments or a careful reframing, so a major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the partial-order prompt ladder is a genuinely useful evaluation idea, and the paper shows convincingly that ParEval is far more prompt-sensitive than HumanEval. The headline number — up to 0.30 absolute pass@1 gain on ParEval-OMP from maximal prompt detail — is real but not a clean measure of specificity. ptop is built from the reference solution and edited until Qwen2.5-Coder-14B passes at ≥0.8; Appendix E's ifft example spells out the algorithm (conjugate, FFT, conjugate, scale by 1/N). That leaks the solution structure into the prompt, so the 0.30 gain mixes specificity with giving away the answer. The paper's own “no complete solution verbatim” check does not address this; the stress-test note has it right.\n\nWhat is genuinely new: three augmentation families form a coherent ladder, and the qualitative taxonomy of which prompt themes grow with length (step breakdowns, I/O specs, edge cases) is a reasonable first map. The direction is credible: intermediate summaries below ptop also show monotone improvement on ParEval, so the claim that detail helps is not just an artifact of the endpoint. Clear writing and honest reporting of compute costs.\n\nSoft spots: no error bars (single greedy run per prompt); the HumanEval curve declines beyond L=200, undercutting “consistently improves”; the detail metric is word count / retention ratio, not validated as information content; no data or code release, so the 11,000 prompts are not reproducible. These are fixable. The ptop selection design is the load-bearing flaw: either use a held-out tuning model, or present curves excluding ptop, or construct the ceiling without feeding the reference solution to the prompt generator.\n\nWho it's for: benchmark builders, people interpreting ParEval gaps, and prompt-engineering researchers. It warrants a serious referee; my own verdict would be major revision, conditional on releasing the data, de-confounding the ceiling, and reporting variance. The framework deserves to be discussed.","headline":"Useful prompt-ladder framework; headline gain is anchored by a solution-leaking ceiling prompt, so the specificity claim needs rework before the numbers are trusted.","tokens_in":23772,"tokens_out":2622,"would_cite":true,"duration_ms":30049,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Increasing prompt specificity consistently improves LLM code-generation pass@1, with up to a 0.30 absolute gain on ParEval-OpenMP — evidence that a large part of the benchmark gap is prompt leanness rather than missing domain knowledge.","keywords":["prompt specificity","code generation","pass@1","LLM evaluation","PartialOrderEval","HumanEval","ParEval","parallel programming"],"falsifier":"Run the same PartialOrderEval suite with ptop produced by a fixed, untuned prompt-generation procedure (no pass@1 feedback loop) on ParEval-OpenMP, and check whether the maximally detailed prompt still yields roughly the reported 0.30 absolute gain over the original prompt; if the gain largely disappears, the headline effect is an artifact of threshold tuning rather than prompt specificity. A complementary check replaces the word-count detail metric with an information-theoretic measure of prompt content and tests whether pass@1 remains monotone in that ordering.","tokens_in":22618,"feed_emoji":"📈","tokens_out":11257,"duration_ms":102431,"temperature":0.7,"pith_summary":"State-of-the-art LLMs ace general coding benchmarks like HumanEval but score far lower on specialized suites such as ParEval, and it has been unclear whether the shortfall is missing domain knowledge or insufficient prompt detail. This paper argues for the second explanation and builds PartialOrderEval, a framework that replaces each benchmark prompt with a partial order of prompts from a bare signature to a maximally detailed, step-by-step specification. Across HumanEval, ParEval-Serial, and ParEval-OpenMP, with Llama-3.x and Qwen2.5-Coder models, pass@1 rises steadily as prompt detail grows, with the largest gains — up to 0.30 absolute — on the hardest parallel subset. If the paper is right, a substantial portion of the measured ParEval gap is prompt leanness, not missing knowledge, and single-prompt pass@1 scores should be supplemented with prompt-specificity curves.","feed_headline":"Adding prompt detail lifts LLM code scores by up to 0.30","feed_subtitle":"A partial-order benchmark shows the ParEval gap is prompt leanness, not missing domain knowledge.","key_machinery":"The load-bearing mechanism is the partial order of prompts generated by PartialOrderEval, anchored by a maximally detailed prompt ptop and a signature-only minimal prompt pbot. From ptop, three augmentation strategies produce intermediate prompts: LLM summarization at word limits of 10, 25, 50, 75, 100, 150, and 200 words; paragraph sampling at retention ratios of 0.2–0.8; and sentence-block masking at mask ratios of 0.2–0.8, each with four random placements. The detail metric D — word count, retention ratio, or inverse mask ratio — orders these prompts, and the evaluation records pass@1 as a function of D, producing performance curves. Construction of ptop is itself a key step: GPT-4.1 drafts a stepwise description of the reference solution, and the authors manually edit or regenerate it until Qwen2.5-Coder-14B reaches pass@1 ≥ 0.8, which fixes the right endpoint of every curve.","core_discovery":"The paper's central claim, stated sympathetically, is that LLM code-generation performance is a systematic function of prompt specificity, and that this function can be measured by ordering prompts from minimal to maximally detailed. Using its PartialOrderEval framework, the paper reports that pass@1 monotonically increases with prompt detail across all three benchmark variants and all eight instruction-tuned models tested, that the increase is steeper for ParEval than for HumanEval, and that at the maximally detailed prompt models exceed their original ParEval scores by up to 0.30 (e.g., Qwen2.5-Coder-14B rises from 0.667 to 0.967 on ParEval-OpenMP). The paper interprets this as evidence that ParEval tasks are more challenging and more sensitive to prompt specificity than HumanEval, that larger models need less prompt detail to reach a given accuracy, and that prompt-specificity curves differentiate models more sharply than a single pass@1 number.","pith_inferences":["If the effect generalizes, benchmark difficulty is partly a property of prompt style, so cross-benchmark model comparisons are only meaningful once prompt specificity is controlled; this could change how leaderboards are read.","The 0.30 figure is a ceiling set by tuning ptop against one model (Qwen2.5-Coder-14B) at a pass@1 threshold of 0.8; other models or an untuned ptop would likely show smaller or different gains, so the number should be treated as a model-specific bound until reproduced.","A direct extension would construct ptop by tuning a different model (e.g., Llama-3.3-70B) and compare the resulting curves; divergence would reveal how much of the specificity effect is a property of the prompt rather than the model.","The partial-order construction is language-agnostic, so applying it to low-resource languages, SQL, or shell tasks — where benchmark prompts are typically sparse — is a natural test of whether the specificity effect is universal."],"forward_implications":["A large share of the ParEval gap is prompt-detail rather than domain-knowledge-limited: with maximally detailed prompts, Qwen2.5-Coder-14B reaches 0.983 on ParEval-Serial and 0.967 on ParEval-OpenMP, versus 0.800 and 0.667 on the original prompts.","Benchmarks that aim to compare model capability should report performance across prompt specificity, since a single-prompt pass@1 conflates model ability with prompt quality.","Specialized suites like ParEval are stronger discriminators of model capability than HumanEval, because the large-versus-small model gap widens with task difficulty and detail sensitivity.","Prompt engineers should prioritize explicit input/output specifications, edge-case handling, and stepwise implementation breakdowns, the themes that grow most as prompts lengthen and correlate with accuracy gains.","Diminishing returns are real: HumanEval curves plateau around 100 words of detail, and over-verbose prompts can slightly reduce accuracy."],"supporting_citations":[{"why":"Supplies the HumanEval benchmark of 164 Python problems whose saturated scores motivate the framework.","marker":"Chen et al., 2021"},{"why":"Supplies the ParEval serial and OpenMP benchmark suites whose performance gap the paper probes.","marker":"Nichols et al., 2024"},{"why":"Describes the Qwen2.5-Coder models used for evaluation and as the pass@1 threshold target when tuning ptop.","marker":"Hui et al., 2024"},{"why":"Describes the Llama 3.x model family used as the second evaluation series.","marker":"Grattafiori et al., 2024"},{"why":"GPT-4.1 drafts the maximally detailed prompts ptop, and o3 assists the taxonomy generation.","marker":"OpenAI, 2025"},{"why":"Qwen3-8B produces the LLM summarization variants at the specified word limits.","marker":"Qwen Team, 2025"},{"why":"Claude Sonnet 4 annotates prompts with taxonomy themes for the qualitative analysis.","marker":"Anthropic, 2025"}],"fun_headline_variants":["Prompt detail lifts LLM code pass@1 by up to 0.30","ParEval gap stems from prompt leanness, not knowledge gaps","PartialOrderEval measures how prompt specificity scales scores","Richer prompts add up to 0.30 to LLM code pass@1","LLM code scores climb with prompt detail, up to 0.30"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the maximally detailed prompt ptop — the right endpoint of every performance curve — is a fair ceiling for prompt detail, but it is generated by GPT-4.1 and manually edited until Qwen2.5-Coder-14B passes at least 80 percent of tests, so that threshold and tuning model directly set the measured gains; the paper also assumes its detail metric (word count, retention ratio, mask ratio) orders prompts by genuine information content without validating that monotonicity.","fun_headline_variants_meta":{"raw":{"variants":["Prompt detail lifts LLM code pass@1 by up to 0.30","ParEval gap stems from prompt leanness, not knowledge gaps","PartialOrderEval measures how prompt specificity scales scores","Richer prompts add up to 0.30 to LLM code pass@1","LLM code scores climb with prompt detail, up to 0.30"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1388,"prompt_tokens":889,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":401}},"tokens_in":505,"tokens_out":499,"duration_ms":5593,"temperature":1.0,"reasoning_tokens":401,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:11:56.747426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same PartialOrderEval suite with ptop produced by a fixed, untuned prompt-generation procedure (no pass@1 feedback loop) on ParEval-OpenMP, and check whether the maximally detailed prompt still yields roughly the reported 0.30 absolute gain over the original prompt; if the gain largely disappears, the headline effect is an artifact of threshold tuning rather than prompt specificity. A complementary check replaces the word-count detail metric with an information-theoretic measure of prompt content and tests whether pass@1 remains monotone in that ordering.","supporting_citations":[],"review_version":1}