{"id":"3c7c27c6-5e59-4f5d-acc6-cee2a571722b","arxiv_id":"2608.11513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pressure-style prompt framings were associated with lower functional correctness and more security warnings than neutral prompts in LiveCodeBench, while most other influence tactics had minimal effects.","lead":"This paper tests whether prompts written like human influence tactics, such as pressure, flattery, or exchange, change the code that large language models produce. It finds that pressure-style wording lowers correctness and security on algorithmic coding tasks, while model choice matters far more than framing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main correctness claim rests on Eq. (2), a negative-binomial GLMM applied to binary pass/fail outcomes; that distributional choice is invalid, so the reported p-values do not support the claim as stated.","rationale":"The reader identified the exact load-bearing point: the main correctness comparison is computed with a negative-binomial GLMM applied to binary outcome data. I read the full manuscript and found no other flaw that is more damaging. The study is well-designed in many respects: large sample, reproducible templates, official harnesses, Bonferroni corrections, a published replication package, and a qualitative analysis with strong inter-rater reliability. The cross-run variability for LiveCodeBench is small (0.059% mean difference), so using the first trial for inference is a lesser concern. The SWE-bench metric analyses exclude unsuccessful patches, but the main claim is about LiveCodeBench correctness and security. The pressure-vs-neutral lexical confound is acknowledged and is inherent to operationalizing a tactic; it does not undercut the narrow claim that these particular framings differ. The statistical model, however, is not a minor caveat: a negative binomial distribution cannot be the correct likelihood for a dichotomous outcome, and no diagnostics are provided to show the fit is harmless. The reported p-values and post hoc contrasts therefore cannot be taken at face value. The appropriate remedy is to refit with a binary GLMM; if the contrasts survive, the conditional acceptance should stand, otherwise the main claim needs to be weakened or re-analyzed. Recommendation: UNCHANGED relative to the reader's CONDITIONAL verdict, with the condition made explicit: correctness significance must be re-established with a valid binary-outcome model.","tokens_in":28516,"tokens_out":4228,"duration_ms":39296,"concrete_test":"Re-fit Eq. (2) for LiveCodeBench functional correctness using a Bernoulli/logit GLMM (family=binomial, link=logit) with the same fixed effects (Tactic x LLM x Difficulty) and ProblemID random intercept, using the same first-trial data; then perform the Bonferroni-adjusted contrasts Neutral vs Pressure and Neutral vs PressureAlternative. If both contrasts remain significant at alpha=0.05, the central claim survives; if not, the paper's main conclusion is unsupported. As a secondary check, fit the same model with all three runs and a run-level random effect to confirm stability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.6 states that for 'binary outcomes' the authors fitted 'a GLMM from the negative binomial family using a log-link function' (Eq. 2), and all LiveCodeBench functional-correctness p-values in Table 5 and Section 4.1 come from this model. Functional correctness is defined in Section 3.4 as C_i in {0,1} (all tests pass or not). A negative binomial model is a count model with a support of non-negative integers and an overdispersion parameter; it is not a Bernoulli/binomial model. Fitting it to 0/1 data imposes a variance-mean relationship that cannot hold for Bernoulli responses (variance = mu(1-mu)), and the resulting standard errors and p-values are not trustworthy. The paper also says models were fit using REML and validated with residual diagnostics, but no residuals, dispersion checks, or model-comparison diagnostics are shown, and REML is not standard for GLMMs. Because the central claim (Neutral > Pressure, p=0.002; Neutral > PressureAlternative, p=0.03) rests on these p-values, the statistical support for the headline result is currently unverified. The security-warning result uses count outcomes and is not affected by this particular objection; the correctness claim is the weak point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether prompt framings based on Yukl and Falbe's influence-tactic taxonomy affect the code generated by five open-weight LLMs. Eight influence tactics plus a neutral baseline are operationalized as prompt templates and evaluated on LiveCodeBench (1,055 problems) and SWE-bench Verified (485 problems), with approximately 123,000 and 57,000 generations respectively. Outcomes are functional correctness, code-quality metrics (cyclomatic complexity, maintainability index, PyLint, SLOC, comment percentage), and Bandit security warnings, complemented by a qualitative codebook analysis. The headline findings are that, on LiveCodeBench, Pressure and Pressure Alternative prompts reduce functional correctness and increase security warnings relative to Neutral prompts, while effects on SWE-bench Verified are mostly absent. The paper concludes that model choice matters more than prompt framing, but that coercive wording is a small, non-negligible risk factor.","tokens_in":28870,"tokens_out":4459,"duration_ms":43692,"significance":"If the results hold, this is a useful empirical contribution to the prompt-engineering and LLM4SE literature. The study is carefully scoped, uses two established external benchmarks, includes five models and multiple runs for non-reasoning models, applies Bonferroni corrections, and ships a replication package. The qualitative phase is conducted with blinded coders and inter-rater reliability reporting, which is a strength. The main limitation is the statistical model used for the binary correctness outcome; because the headline claim depends on that model, the current manuscript does not provide verified statistical support for its central contribution. The security warning finding, based on count outcomes, is more defensible, and the qualitative evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The functional correctness outcome is binary (Section 3.4 defines C_i in {0,1}), but Eq. (2) specifies a negative binomial GLMM with a log link. A negative binomial distribution has support on nonnegative integers and variance mu + mu^2/theta, which cannot represent a Bernoulli response whose variance is mu(1-mu); the resulting standard errors and p-values are therefore not trustworthy. All LiveCodeBench correctness p-values in Table 5 and Section 4.1, including the central claim that Neutral outperforms Pressure (p=0.002) and Pressure Alternative (p=0.03), come from this model. The authors should refit the correctness analyses with a Bernoulli/binomial GLMM using a logit link (or an equivalent appropriate model) and report the resulting estimates. The Bandit security warning analyses concern count outcomes and are not affected by this particular objection.","section":"Section 3.6, Eq. (2)"},{"comment":"The paper states that 'model fit was validated using residual diagnostics' and that 'all models were fit using Restricted Maximum Likelihood (REML),' but no residual plots, dispersion checks, or model-comparison diagnostics are presented. REML is not a standard estimation criterion for GLMMs, so the reader cannot verify the distributional adequacy of the fitted models. The authors should either provide the diagnostics in an appendix or explicitly describe the estimation method (e.g., Laplace approximation or adaptive Gauss-Hermite quadrature) and justify its use.","section":"Section 3.6"},{"comment":"All statistical inferences are based on the first trial per condition for non-reasoning models, rather than on all three runs, and run-level variability is only reported descriptively in Table 3. For SWE-bench Verified the between-run variability is substantial (mean 10.31% difference, absolute change 26.56%), so inferences based on a single run may not be stable. The authors should either include run as a random effect or perform a sensitivity analysis across all three runs to confirm that the reported conclusions, especially the null results for most tactics on SWE-bench, are not artifacts of selecting the first trial.","section":"Section 3.6 and Section 4"}],"minor_comments":[{"comment":"The phrase 'which established the distinctiveness and utility of the expanded autonomy' appears to be a typo for 'expanded taxonomy'.","section":"Section 2.1"},{"comment":"The sentence 'responses to SWE-bench must be written in diffformat' should read 'in diff format'.","section":"Section 3.1.2"},{"comment":"The qualitative analysis says 1,600 samples were selected but only 350 prompt completions were ultimately coded; the relationship between the sampled 1,600 and the coded 350 should be clarified, including whether the reported frequencies are based on the 350 or the full 1,600.","section":"Section 4.3"},{"comment":"The table reports eta-squared values (such as eta^2_p(Tactic)=0.015) alongside results from GLMMs; it should be clarified how partial eta-squared is computed for generalized mixed models, or the effect-size measure should be changed to one that is standard for the fitted model family.","section":"Table 5"},{"comment":"The two-response example is illustrative but anecdotal; it should be labeled as such and not presented as independent evidence for the quantitative conclusions.","section":"Section 4.1, representative example"}],"recommendation":"major_revision","confidential_remarks":"The manuscript header states that it has been accepted for publication and includes a DOI (10.1007/s10664-026-10934-z). If the paper is simultaneously under review elsewhere, this raises a dual-submission or prior-publication concern that the editor may wish to verify. I did not factor this into the technical recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know about: it is the first large-scale attempt to test whether Yukl and Falbe's influence tactics, operationalized via the IBQ-G, change LLM code generation. Two benchmarks, five models, roughly 123k generations, reproducible prompts on OSF, plus a qualitative codebook analysis. The main finding — pressure phrasing lowers correctness and raises Bandit warnings on LiveCodeBench — is plausible and practically useful, and the security result is supported by count models where the distributional choice is appropriate.\n\nWhat it does well: the tactics are operationalized from a validated instrument, the tone is controlled, the authors explicitly distinguish their work from politeness and emotion prompting, and they are honest about threats to validity. The replication package is a real asset. The result that model choice swamps framing is consistent with prior work and a useful reassurance for practitioners.\n\nSoft spots, in proportion: the load-bearing correctness claim rests on Eq. (2), a negative-binomial GLMM fitted to binary pass/fail outcomes. That is a count model; Bernoulli responses have variance mu(1-mu), and the negative binomial's variance function does not hold for 0/1 data. No dispersion or residual diagnostics are shown, and REML for a GLMM is non-standard. So the reported p=0.002 and p=0.03 values do not cleanly support the claim as stated. This is fixable: a logistic mixed model would be the natural re-analysis, and the authors should be asked to show it. The security finding and the qualitative example survive this concern. Two smaller issues: the SWE-bench metric analyses drop unsuccessful patches, which the authors acknowledge but which does bias those comparisons, and the reasoning model got only one run, making its results harder to trust.\n\nWho this is for: empirical software engineering researchers, prompt engineering folks, and developers who want reliable codegen. The takeaway is narrow but real: avoid urgency and coercion wording when you need correct, secure code, and choose the model carefully.\n\nRecommendation: send it to peer review — a serious referee should ask for the corrected binary analysis and a check on robustness. The paper is worth engaging with; the statistical fix is not costly.","headline":"A large, transparent empirical study of influence-tactic framing in LLM code generation; the headline pressure-framing result is plausible but the binary-outcome model is misspecified, so the correctness claim needs a re-analysis before I would repeat it.","tokens_in":29255,"tokens_out":2284,"would_cite":false,"duration_ms":22930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coercive prompt wording reduces the correctness and security of LLM-generated code.","keywords":["influence tactics","prompt framing","large language models","code generation","software quality","security","prompt engineering","empirical software engineering"],"falsifier":"Re-run the LiveCodeBench comparison of Neutral versus Pressure using a model appropriate for binary outcomes, such as a logistic or binomial GLMM with the same fixed and random effects, and check whether the $p=0.002$ and $p=0.03$ contrasts survive; if they do not, the paper's headline claim fails. A complementary check is to paraphrase the Pressure prompt into different coercive wordings and see whether the correctness drop is reproducible.","tokens_in":28335,"feed_emoji":"🤖","tokens_out":7643,"duration_ms":67605,"temperature":0.7,"pith_summary":"This paper tests whether psychologically inspired ways of phrasing a request, borrowed from a management-research taxonomy of influence tactics, change the code that LLMs produce when the task content is held fixed. Across roughly 123,000 LiveCodeBench generations and 57,000 SWE-bench Verified generations from five open-weight LLMs, prompts framed with pressure or coercive urgency produced functionally correct code less often and code with more static-analysis security warnings than a plain neutral prompt. The effect was strongest on structured algorithmic problems; on real-world GitHub maintenance tasks, the only consistent framing effect was that pressure made output more verbose. The paper's practical point is that prompt wording is a real but secondary factor in code generation, and that coercive phrasing is the one to avoid when correctness and security matter.","feed_headline":"Pressure wording cuts LLM code correctness and security","feed_subtitle":"Neutral prompts beat urgent, threatening wording on correctness and security across five LLMs and 123,000 generations.","key_machinery":"The argument is carried by reproducible prompt templates that translate eight influence tactics from Yukl and Falbe's taxonomy, a classification of persuasion behaviours from organizational psychology, into concrete wording, with each template built from the item requirements of the validated Influence Behavior Questionnaire-General. The key contrast is Neutral, a straightforward request, versus Pressure, which adds demands, surveillance, and threatened consequences. The quantitative engine is a mixed-effects model treating ProblemID as a random intercept, with tactic, LLM, and difficulty as fixed effects; binary correctness is modelled with a generalized linear mixed model described as negative-binomial with log link, and continuous quality metrics with a linear mixed model. The evaluation stack, using LiveCodeBench and SWE-bench Verified harnesses for correctness, Radon for complexity and maintainability, PyLint for style, and Bandit for security, turns a linguistic manipulation into measurable software-quality signals.","core_discovery":"The central claim is that influence-tactic prompt framings, particularly pressure, shift LLM code-generation outcomes even though the underlying task is identical. On LiveCodeBench, neutral prompts significantly outperformed both Pressure ($p=0.002$) and Pressure Alternative ($p=0.03$) on functional correctness, and both pressure variants were associated with significantly more Bandit-flagged security warnings than Neutral ($p<0.001$ and $p=0.0004$). By contrast, no significant tactic effects appeared for maintainability, complexity, PyLint scores, or lines of code on LiveCodeBench, and on SWE-bench Verified the only significant pairwise framing effect was that Pressure produced more code than Neutral ($p=0.0025$); model identity and task difficulty accounted for most of the variance. The authors interpret the effects as distributional cues learned from human communication rather than as evidence that LLMs experience pressure, and they frame the result as the first large-scale demonstration that psychologically grounded prompt framing is a measurable, if modest, factor in code generation.","pith_inferences":["If the distributional-cue explanation is right, the effect size should track how strongly a model's training corpus associates coercive phrasing with truncated or non-deliberative responses; this could be tested by holding the task constant and varying only the frequency of words like 'must', 'urgent', and 'watching'.","Replacing the negative-binomial GLMM with a logistic GLMM for the binary correctness outcome would be the first robustness check a reader should run, because the paper's reported p-values for the main correctness claim depend on that distributional choice.","The authors' contrast with adversarial persuasion results, where psychologically framed prompts can achieve very high success rates in jailbreak-style settings, suggests that pressure framing may matter far more in open-ended or security-sensitive interactions than in benchmarked code tasks; a direct test would combine the Pressure template with a prompt-injection payload and measure success rates","Because the reasoning model had only a single run and SWE-bench cross-run absolute change was 26.56 percent, the null results on maintenance tasks should be treated as provisional until multi-seed replicates are available."],"forward_implications":["Developers should avoid coercive or urgency-laden phrasing in code-generation prompts when correctness and security are priorities, because neutral wording performed best in this study.","Prompt framing is a minor but non-negligible factor: model choice and task difficulty explain far more variance, so selecting a capable model matters more than polishing prompt tone.","Framing effects are task-dependent: they showed up in structured algorithmic problems on LiveCodeBench but mostly disappeared in realistic maintenance tasks on SWE-bench Verified, where only verbosity shifted.","Influence-tactic framings can alter surface features of generated code, such as comment density, explanation length, error handling, and hallucination patterns, but these changes are too modest to be a reliable style-control technique.","The released prompt templates give other researchers a reusable, taxonomy-aligned instrument for studying pragmatic framing in LLM code generation."],"supporting_citations":[{"why":"Supplies the taxonomy of influence tactics from organizational psychology that the prompt templates operationalize.","marker":"[70]"},{"why":"Provides the validated IBQ-G questionnaire whose per-tactic items were turned into wording requirements for each prompt.","marker":"[72]"},{"why":"Defines the LiveCodeBench benchmark, including the 1,055 Python problems and test harness used for correctness evaluation.","marker":"[28]"},{"why":"Defines the SWE-bench Verified subset of real-world GitHub issue-fix pairs used for maintenance-task evaluation.","marker":"[11]"},{"why":"Meta-analysis showing that pressure tactics are counterproductive in human work settings, which motivates why pressure might degrade LLM output.","marker":"[35]"},{"why":"Documents high success rates for psychologically framed prompt-injection attacks, used in the discussion to contextualize the modest effects found here.","marker":"[13]"},{"why":"Contributes the pre- and post-patch delta metric approach adopted for measuring code quality and security changes on SWE-bench Verified.","marker":"[10]"}],"fun_headline_variants":["Urgent prompts weaken LLM code security","Threatening prompts hurt code correctness","Persuasive prompts backfire in code generation","Pressure tactics reduce LLM code quality","Influence framing: a modest but real effect"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the statistical model in Eq. (2), described as a negative-binomial GLMM with a log link, is a valid way to analyze the binary pass/fail correctness outcome, and the paper gives no diagnostic check showing that this count-style model fits binary data better than a standard logistic GLMM.","fun_headline_variants_meta":{"raw":{"variants":["Urgent prompts weaken LLM code security","Threatening prompts hurt code correctness","Persuasive prompts backfire in code generation","Pressure tactics reduce LLM code quality","Influence framing: a modest but real effect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1246,"prompt_tokens":980,"completion_tokens":266,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":201}},"tokens_in":596,"tokens_out":266,"duration_ms":3088,"temperature":1.0,"reasoning_tokens":201,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T13:56:51.812213+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the LiveCodeBench comparison of Neutral versus Pressure using a model appropriate for binary outcomes, such as a logistic or binomial GLMM with the same fixed and random effects, and check whether the $p=0.002$ and $p=0.03$ contrasts survive; if they do not, the paper's headline claim fails. A complementary check is to paraphrase the Pressure prompt into different coercive wordings and see whether the correctness drop is reproducible.","supporting_citations":[{"cited_title":"Journal of applied psychology75(2), 132 (1990)","cited_arxiv_id":null,"evidence_quote":"Supplies the taxonomy of influence tactics from organizational psychology that the prompt templates operationalize."},{"cited_title":"The Leadership Quarterly19(5), 609–621 (2008)","cited_arxiv_id":null,"evidence_quote":"Provides the validated IBQ-G questionnaire whose per-tactic items were turned into wording requirements for each prompt."},{"cited_title":"URLhttps://openai.com/index/ introducing-swe-bench-verified/","cited_arxiv_id":null,"evidence_quote":"Defines the SWE-bench Verified subset of real-world GitHub issue-fix pairs used for maintenance-task evaluation."},{"cited_title":"The Leadership Quarterly28(1), 210–228 (2017)","cited_arxiv_id":null,"evidence_quote":"Meta-analysis showing that pressure tactics are counterproductive in human work settings, which motivates why pressure might degrade LLM output."},{"cited_title":"In: European Conference on Cyber Warfare and Security, pp","cited_arxiv_id":null,"evidence_quote":"Documents high success rates for psychologically framed prompt-injection attacks, used in the discussion to contextualize the modest effects found here."},{"cited_title":"In: 2025 IEEE International Confer- ence on Software Analysis, Evolution and Reengineering (SANER), pp","cited_arxiv_id":null,"evidence_quote":"Contributes the pre- and post-patch delta metric approach adopted for measuring code quality and security changes on SWE-bench Verified."}],"review_version":1}