{"id":"b6861eb5-231a-403d-8b75-1bfd7a4c3028","arxiv_id":"2506.05614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Across ten software engineering tasks and four LLMs, no prompting technique wins consistently; ES-KNN is best on many tasks, some techniques underperform the baseline, and USC is best for code QA and code generation.","lead":"This study tests 14 prompt-writing styles across 10 software engineering tasks using four large language models, and finds that no single style is best everywhere. It also measures token and time costs, so developers can pick a prompt style that balances accuracy and expense.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central rankings and Observation 1 rest on single runs with no significance testing; reported winners and \"negative impact\" claims may be sampling noise.","rationale":"I focused on the absence of inferential statistics rather than the reader's chosen weakest assumption (paraphrase fidelity) because it is the less secure premise for the central claim. The prompt-validation process (Section III-C), despite moderate kappa=0.45, at least attempts to control wording; single-run rankings have no analogous mitigation. Every conclusion that depends on comparing techniques—Observation 1, Tables II–IV, and the practical per-task/model recommendations—requires the observed differences to be reliable. The paper provides no variance estimates, no confidence intervals, and no hypothesis tests, so the differences could be chance. The paraphrase concern, while real, would affect the interpretation of even a statistically significant result; but if the rankings are within noise, the central claim fails regardless of paraphrase fidelity. The paper's broad benchmark and public replication package are real strengths, and the descriptive tables are useful, but the inferential gap is exactly why the reader's CONDITIONAL verdict is appropriate. Since my concern is one the reader already identified in the rationale (though not as the weakest assumption), no change to the verdict is needed.","tokens_in":19176,"tokens_out":8943,"duration_ms":96170,"concrete_test":"Using the per-instance outputs in the replication package (github.com/prompt-study/prompt-tasks-study), compute for each (technique, task, model) cell a 95% bootstrap confidence interval over the metric, and run paired significance tests between each technique and the control baseline, and between the top and bottom ranked techniques, with Bonferroni correction across 14 techniques. If the majority of winner-vs-control and best-vs-worst differences fail to reach significance (e.g., p >= 0.05), the rankings in Tables II–IV and Observation 1 should be reframed as descriptive rather than evidence of technique-level differences, and the conditional acceptance should require this inferential analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The primary evidence for Observation 1 and the task/model-specific recommendations is the aggregate ranking in Tables II–IV, but the methodology (Section III-E) reports a single run per technique-task-model cell, and Section IV-A presents no confidence intervals, effect sizes, or significance tests. With task sample sizes around 380–391 instances, a difference of a few percentage points is within plausible sampling error: for example, in Code QA the top technique USC scores 55.67 vs. control 50.99 (Table IV), a 4.68-point gap that is only ~1.3 standard errors if each mean has ~380 independent samples. Similar close calls appear in Code Summarization (SG-ICL 4.15 vs. control 4.16). The z-scores in Table IV are descriptive standardizations, not inferential statistics. Consequently, the claim that \"certain prompting techniques can negatively impact their performance\" (Observation 1) is not statistically supported: the apparent worst techniques may be within noise of the baseline, and the \"consistent\" winners such as ES-KNN may not differ reliably across tasks or models. Without variance information, the central practical guidance—select techniques by task and model—is built on rankings that could reorder under repeated sampling.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical comparison of 14 prompting techniques applied to 10 software-engineering tasks using four LLMs (DeepSeek-V3, Qwen2.5-Coder-32B-Instruct, Llama-3.3-70B-Instruct, and OpenAI o3-mini). For each technique-task-model cell the authors run a single pass over a sampled dataset, compute task-specific metrics (accuracy, F1, BLEU, CodeBLEU), and rank the techniques. They also compute correlations between five linguistic prompt features and performance (RQ2), solicit contrastive explanations from the same LLMs to explain why the best technique outperformed the worst (RQ3), and report token and time savings (RQ4). The headline finding is Observation 1: no single prompting technique consistently outperforms all others, and some techniques perform below a baseline; ES-KNN is the most frequent aggregate winner, while USC is best for Code QA and Code Generation. The authors release their data and scripts.","tokens_in":19345,"tokens_out":5903,"duration_ms":57535,"significance":"If its central claims held, the paper would give practitioners a useful evidence-based map of prompt-technique performance across common SE tasks and models, and the cost analysis would address a real operational concern. The study is comparatively broad (14 techniques × 10 tasks × 4 models) and the public artifact release is a strength. However, the current statistical reporting is not sufficient to support the ranking-based conclusions: single runs without variance information cannot establish which differences are real, and the RQ2/RQ4 analyses omit key definitions and independence assumptions. The empirical core is plausible but needs substantial strengthening before the practical recommendations can be accepted.","major_comments":[{"comment":"The central ranking results and Observation 1 rest on a single run per technique-task-model cell, with no confidence intervals, effect sizes, or significance tests. In Code QA, the top technique USC scores 55.67 versus control 50.99 on roughly 382 instances; the approximate standard error of a proportion near 0.55 is about 2.5 percentage points, so the 4.68-point gap is on the order of 1.3 standard errors and cannot be distinguished from sampling noise. In Code Summarization, the reported best technique SG-ICL (BLEU 4.15) is actually below the control (4.16), so the 'best' label is not even directionally consistent with the baseline. The z-scores in Table IV are descriptive standardizations, not inferential statistics. I request multi-run or bootstrap/permutation analyses and reporting of variance to support Observation 1, the worst-versus-baseline claims, and the task/model-specific recommendations.","section":"Section IV-A / Section III-E"},{"comment":"The 'tokens saved' and 'time saved' metrics are never defined. The text in Section IV-D explains that efficiency is computed by dividing performance by tokens and response time, but Tables V and VI report mean numbers of tokens and seconds 'saved per prompt' without specifying the reference (presumably the baseline prompt) or the unit over which the mean is taken. The reported value of 19,485 tokens saved for Code QA with Llama is not interpretable without this definition, and no variance or sample size is reported. This undermines RQ4 and Observation 4.","section":"Section IV-D / Tables V–VI"},{"comment":"The Spearman correlations in RQ2 do not state their unit of analysis. Section III-F says linguistic metrics are averaged per prompting technique for each SE task before correlating with performance, but the N used for the p-values is not reported. If the unit is technique-task combinations, the observations are not independent because each technique appears in every task and each task in every technique, so the reported p-values (e.g., r=0.4440, p<0.001) are unverifiable and likely anti-conservative. Please report N, degrees of freedom, and a multilevel analysis that accounts for technique and task as random effects.","section":"Section III-F / Section IV-B"},{"comment":"Observation 3 claims that prompting techniques show greater effectiveness when they include structured guidance and in-context examples, but this conclusion is based on contrastive explanations generated by the same LLMs that produced the task outputs, followed by manual categorization. The manuscript does not validate that these self-reports reflect actual causes; for instance, no intervention test adds or removes the alleged 'structured guidance' to confirm its effect. RQ3 is framed as 'according to LLMs,' and Observation 3 should be qualified accordingly or supported by a controlled prompt manipulation study.","section":"Section III-G / Section IV-C"}],"minor_comments":[{"comment":"The sentence describing embedding-model selection is incomplete: 'We selected the most freqency used and best performing' ends without an object or period.","section":"Section III-B, item 1"},{"comment":"The 'Understanding Tasks' and 'Generation Tasks' headers appear merged with the first task rows in the rendered tables, making it difficult to associate each technique with the correct task.","section":"Tables II and III"},{"comment":"The table columns are not aligned with the header: for example, the Code Summarization row appears to list SG-ICL 4.15 versus control 4.16, which is inconsistent with Table II, where Control is the aggregate best for that task. Please reconcile the tables and clarify what the z-scores and raw values represent.","section":"Table IV"},{"comment":"'Response time' is not defined: it is unclear whether this is end-to-end API latency, generation time only, or something else, and how network variability was handled.","section":"Section IV-D"},{"comment":"The claimed 'more than 2k prompts' is ambiguous: 14 techniques × 10 variations × 10 tasks × 4 models gives 5,600 prompt-template instances, while 14 × 10 × 4 = 560 prompt runs. Please clarify what counts as a 'prompt' in this number.","section":"Abstract / Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical footprint and a useful artifact release, but the missing statistical analyses and the undefined savings metrics are load-bearing for the central recommendations. The internal inconsistency between Table II and Table IV for Code Summarization should also be resolved. I do not see grounds for rejection, as the raw benchmark can be re-analyzed with additional significance testing and clearer definitions within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a genuinely broad empirical benchmark—14 prompting techniques, 10 SE tasks, 4 LLMs, with token and time costs—but the headline rankings are not backed by any significance testing. Treat the 'winners' list as directional until the authors add variance information.\n\nWhat is genuinely useful: the cross-product is new. Prior work surveyed prompting techniques or compared pretrained models, but nobody has crossed techniques × tasks × models with cost data. The paper is clearly structured, uses standard benchmarks, samples datasets at 95% confidence, and promises a public replication package. The qualitative takeaway—no single technique dominates, and some techniques fall below a simple baseline—is plausible and consistent with the wider literature.\n\nThe soft spots are real. The stress-test note is on target: rankings come from single runs per cell, and the paper gives no confidence intervals or significance tests. For Code QA, the top technique scores 55.67 vs. control 50.99; with ~380 samples that gap is roughly 1.3 standard errors, so it could easily be noise. Several other close calls look similar. That means Observation 1's second sentence—'certain prompting techniques can negatively impact their performance'—is not statistically supported as written. The token/time 'saved' metrics also never define their reference baseline. The RQ3 contrastive explanation analysis asks the same model that produced the outputs to explain why a technique won, then categorizes those self-reports; that's descriptive at best, and the introduction promises composite-prompt results that never appear in the results section. The kappa=0.45 for prompt template agreement is moderate, so the 'faithful instances' assumption is shaky.\n\nThe paper is not circular: it uses external benchmarks and has no fitted parameters beyond standard ones. The central 'no universal best' claim is robust even if exact rankings shift. Who benefits: practitioners wanting a starting point for prompt selection, and researchers building prompt benchmarks. It deserves a serious referee—the flaws are fixable, not fatal. I would send it to peer review with a request for significance testing, defined baselines, and either removal or reframing of RQ3. If those are addressed, this becomes a solid citable resource.","headline":"A broad but statistically fragile benchmark of prompting techniques for SE; worth a serious referee if the authors add significance testing.","tokens_in":19943,"tokens_out":2593,"would_cite":true,"duration_ms":25062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single prompt-engineering technique consistently outperforms others across software engineering tasks, according to a four-model, ten-task benchmark.","keywords":["prompt engineering","large language models","software engineering tasks","code generation","code understanding","prompt selection","resource cost","empirical benchmarking"],"falsifier":"Run the identical benchmark with a fresh set of paraphrases per technique produced independently and validated with inter-rater agreement above 0.8; if one technique then wins all ten tasks across the four models, the paper's Observation 1 is overturned. If the rankings instead reshuffle, the no-universal-winner result survives but technique-level attributions would need to be treated with caution.","tokens_in":18927,"feed_emoji":"🤖","tokens_out":6526,"duration_ms":61864,"temperature":0.7,"pith_summary":"The paper establishes that the effectiveness of prompt engineering techniques for software engineering tasks is task- and model-dependent: no single technique consistently wins across all ten tasks and four LLMs. It reports that ES-KNN, which selects similar in-context examples by nearest-neighbor search, is the most frequent aggregate winner, while USC (universal self-consistency) leads on code question answering and code generation, and several techniques perform below a simple baseline. The study also links prompt linguistic features, such as lexical diversity and token count, to performance, and uses contrastive explanations to identify structured guidance and in-context examples as the factors most often behind a technique's success. Resource accounting shows that top performers can be token- or time-heavy, so choosing a technique involves a cost-performance trade-off.","feed_headline":"No single prompt technique wins every coding task","feed_subtitle":"A 14-technique, 10-task, 4-model benchmark finds ES-KNN leads overall; some prompts hurt","key_machinery":"The load-bearing instrument is a controlled comparison corpus: 14 prompting techniques drawn from six prompt-engineering dimensions, applied to 10 SE tasks sampled from a prior benchmark, with each technique instantiated by ten paraphrase templates produced by a commercial LLM and filtered by pairwise researcher review (Cohen's kappa = 0.45). Every technique-task pair is run on four LLMs, and performance per task is aggregated across models and z-scored to compare metrics of different scales. Linguistic metrics (lexical diversity, token count, readability indices) are correlated with performance via Spearman correlation, and contrastive explanations, where the LLM is asked why the best technique beats the worst, are hand-coded into factor categories such as Structured Guidance and In-Context Examples. Token counts and response times are collected per prompt to rank resource efficiency.","core_discovery":"On the paper's own terms, the central discovery is Observation 1: no prompting technique consistently outperforms all others across SE tasks, and some techniques can hurt performance. Aggregating four LLMs per task, ES-KNN is best for Clone Detection, Code Translation, and Assert Generation; USC is best for Code QA and Code Generation; ToT (Tree of Thought) is best for Defect Detection; and for some tasks the winning technique is the control baseline itself. The worst techniques often score below the simple baseline, and the best techniques differ for the o3-mini model, with Role Prompting winning most tasks there. Linguistic analysis finds lexical diversity positively correlated with performance and token count negatively correlated, with readability correlations reversing between code understanding and code generation tasks.","pith_inferences":["Editorial inference: because the ten paraphrases per technique were filtered with only moderate inter-rater agreement (kappa = 0.45), the observed rankings may partly reflect wording artifacts; a replication with stricter validation could shrink or shift the gaps between techniques.","Editorial inference: the linguistic correlations suggest a testable design rule—reduce token count while increasing lexical diversity—that could be applied to prompts outside the 14 techniques studied.","Editorial inference: the task-model specificity of the winners implies that an automated prompt selector trained on task embeddings and model identity could outperform any fixed technique; the released dataset would support such a meta-model."],"forward_implications":["Practitioners should select prompting techniques per task and per model rather than adopting a single prompt recipe, since no technique won across all ten tasks.","ES-KNN is a strong default for clone detection, code translation, and assert generation, while USC is a strong default for code question answering and code generation.","Some published prompting techniques can underperform a simple baseline, so prompt engineering is not automatically beneficial and should be validated before adoption.","Resource-aware selection matters: ES-KNN is fast but token-hungry, USC and Self-Refine add latency, and Role Prompting is the most token-efficient option.","The winning techniques shift for o3-mini, suggesting that prompt effectiveness does not transfer across model scales or families."],"supporting_citations":[{"why":"Supplies the 46-technique prompt-engineering catalog from which the 14 studied techniques were filtered.","marker":"[9]"},{"why":"Provides the 10 SE tasks, datasets, and understanding/generation grouping used as the benchmark.","marker":"[12]"},{"why":"Defines k-nearest-neighbor exemplar selection (ES-KNN), the most frequent aggregate winner.","marker":"[43]"},{"why":"Defines universal self-consistency (USC), the winner for Code QA and Code Generation.","marker":"[51]"},{"why":"Provides the code-oriented embedding model used to retrieve ES-KNN exemplars.","marker":"[47]"},{"why":"Supports the prompt-validation protocol that produced ten paraphrases per technique.","marker":"[61]"},{"why":"Supplies the contrastive-explanation method used to attribute success to structured guidance and examples.","marker":"[13]"}],"fun_headline_variants":["No prompt technique wins all coding tasks — some even hurt","Best prompt technique? Depends on task, model, and even the baseline","Study: 14 prompt techniques, 10 tasks — no winner takes all","Some prompt tricks backfire: choose carefully per coding task","Prompting for code: best technique varies, some hurt (4-model study)"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings hold only if the ten paraphrases per technique, filtered with researcher agreement of Cohen's kappa 0.45, are faithful instances of the technique rather than incidental wording differences.","fun_headline_variants_meta":{"raw":{"variants":["No prompt technique wins all coding tasks — some even hurt","Best prompt technique? Depends on task, model, and even the baseline","Study: 14 prompt techniques, 10 tasks — no winner takes all","Some prompt tricks backfire: choose carefully per coding task","Prompting for code: best technique varies, some hurt (4-model study)"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000757,"raw_usage":{"total_tokens":3339,"prompt_tokens":896,"completion_tokens":2443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":2350}},"tokens_in":512,"tokens_out":2443,"duration_ms":17129,"temperature":1.0,"reasoning_tokens":2350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:40.331599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical benchmark with a fresh set of paraphrases per technique produced independently and validated with inter-rater agreement above 0.8; if one technique then wins all ten tasks across the four models, the paper's Observation 1 is overturned. If the rankings instead reshuffle, the no-universal-winner result survives but technique-level attributions would need to be treated with caution.","supporting_citations":[{"cited_title":"jina-embeddings-v2-base-code,","cited_arxiv_id":null,"evidence_quote":"Defines universal self-consistency (USC), the winner for Code QA and Code Generation."},{"cited_title":"What makes good in-context examples for GPT-3?","cited_arxiv_id":null,"evidence_quote":"Provides the code-oriented embedding model used to retrieve ES-KNN exemplars."},{"cited_title":"Prompt engineering: How prompt vocabulary af- fects domain knowledge,","cited_arxiv_id":null,"evidence_quote":"Supports the prompt-validation protocol that produced ten paraphrases per technique."}],"review_version":1}