{"id":"fe05c013-1cee-45ee-95df-46797064c7ad","arxiv_id":"2608.07776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A single SOC 2 sentence moved LLM-generated code conformance from 47 to 88 percent up to 86 to 100 percent across 24 generations, while MFA hooks, cookie flags, and account lifecycle controls still required explicit naming.","lead":"This paper tested three frontier AI models writing four production-like code artifacts, each with and without one sentence saying the code is subject to SOC 2, and scored all 24 outputs against compliance rubrics. One sentence raised conformance from a 47 to 88 percent range to 86 to 100 percent and removed the real vulnerabilities found, but MFA, cookie flags, and account lifecycle still had to be named individually.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOC 2 directive bundles 'data is sensitive' with 'subject to SOC 2,' so the claimed causal role of naming the standard is not identified; the 23-50 point effect could be a generic sensitivity response.","rationale":"The reader's A4 (one generation per cell) is real, correctly flagged, and addressed by the authors' pre-committed repetition; it threatens the exact percentages and fine model rankings. I do not dispute it. But the most load-bearing weakness for the central claim as quoted is upstream of sampling: the treatment itself is a compound. The paper's own wording in Section 3.1 makes the confound explicit, and the abstract's causal phrasing ('one sentence naming the standard') is stronger than the design can support. The paper deserves credit for publishing prompts, frozen rubrics, artifacts, and acknowledging the commission-list expansion and asymmetric audit; those make the confound easy to see and fix. The suggested ablation is cheap and decisive: it distinguishes 'sensitivity priming' from 'standard-name priming.' Until that runs, the verdict stays CONDITIONAL: the direction of the prompt effect is robust and the artifacts are credible, but the headline causal attribution should not be accepted as stated.","tokens_in":15423,"tokens_out":8378,"duration_ms":77461,"concrete_test":"Run all four use cases under four prompt conditions, with at least three generations per cell: (1) original neutral prompt; (2) original full directive; (3) neutral plus 'The data handled by this system is sensitive'; (4) neutral plus 'This artifact is subject to SOC 2.' Score with the same frozen rubrics and complete hand adjudication of passes and failures. If condition (3) reproduces most of the (2)-(1) gap in conformance or in commission findings, the paper must narrow its claim from 'naming the standard' to 'declaring data sensitivity, with or without SOC 2'; if condition (4) alone reproduces the gap, the original attribution survives.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.1 defines the directed condition as the identical task 'plus one sentence declaring the data sensitive and the artifact subject to SOC 2,' but the Abstract and the reader's summary translate this into 'one sentence naming the standard' and 'unless the standard is named.' The design never varies the two components of the sentence, so the observed gain in every cell is compatible with 'data sensitive' alone eliciting generic secure-coding behavior (encryption, access control, retention), with the label 'SOC 2' contributing little or nothing. That alternative would invalidate the paper's central explanatory claim that compliance knowledge is latent and needs the standard itself named; it would leave the practical sentence effective but the mechanism misattributed. Section 7 acknowledges only single-phrasing sensitivity and does not list a sensitive-only or SOC2-only ablation, and the pre-committed paraphrase run does not separate the components. Repeating cells without adding such conditions cannot resolve the confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled, factorial evaluation of three Claude models (Fable 5, Opus 4.8, Opus 5) across four code-generation use cases (S3 CLI, authentication service, RDS Terraform, file-upload handler). Each cell was generated once from a neutral task statement and once from the same statement plus a single added sentence declaring the data sensitive and the artifact subject to SOC 2, producing 24 artifacts scored against binary rubrics mapped to Trust Services Criteria. The authors find unprompted conformance of 47-88% varying by whether controls are idiomatically part of the code, that the added sentence raises every cell to 86-100% (worth 23-50 points) and removes every insecure construction found, that model choice has little detectable effect, and that a regular-expression scorer is unreliable, under-crediting well-engineered code 21 of 27 times and passing a real defect expressed through a conditional. The paper is unusually transparent: rubrics frozen before generation, predictions registered for two use cases and for the later Opus 5 arm, all outputs published, and limitations stated at length, including the admission that one generation per cell does not support fine differences.","tokens_in":15538,"tokens_out":4796,"duration_ms":44270,"significance":"If the findings hold, the paper provides concrete evidence for a practically important phenomenon: frontier models can apply compliance controls when a generic security/compliance signal is present, but omit many of the same controls when it is absent, and the omission can coexist with real vulnerabilities. The main contribution is not a new benchmark but a carefully described measurement protocol with published rubrics, verbatim prompts, raw artifacts, and per-item hand-verified scores, which makes the study unusually easy to reproduce or rescore. The consistency of the prompt effect across all eight cells is a genuine strength, and the paper's treatment of the commission list as an instrument whose scope changes the answer is honest. The two most significant limitations are that the directive bundles two cues ('data sensitive' and 'subject to SOC 2'), so the specific causal role of naming the standard is not identified, and that every cell is a single non-deterministic generation, so the exact percentages and especially the cross-model comparisons sit at the resolution floor.","major_comments":[{"comment":"The directed condition is defined as the identical task 'plus one sentence declaring the data sensitive and the artifact subject to SOC 2' (§3.1), but the Abstract and §8 attribute the effect to 'one sentence naming the standard' and state that the default behavior omits controls 'unless the standard is named.' Because the single sentence varies two properties at once, the observed 23-50 point gains and removal of insecure constructions are compatible with a generic response to 'data sensitive' alone, with the SOC 2 label contributing little or nothing. The design never varies the two components, and the pre-committed paraphrase run does not separate them either. Please either add a sensitivity-only and a standard-only ablation, or recast the mechanistic claims so that they attribute the effect to the two-part directive rather than to naming the standard. This is load-bearing for the central explanatory claim in the Abstract and §8.","section":"§3.1, Abstract, §8"},{"comment":"All 24 cells are single non-deterministic generations, so the exact percentages, the stated size of the directive effect, and the cross-model conclusions in §5.5 rest on one draw per cell. The paper discloses this and pre-commits to three generations per cell, and the direction of the prompt effect is consistent across all eight cells, which buffers the main practical finding. However, the abstract's statement that 'model choice mattered least' and the specific claim that same-generation models are 'within one rubric item across all eight cells' are descriptions of single samples, not of model behavior; the paper itself correctly notes the resolution floor. A repetition run with three or more generations per cell is needed before the model comparison can be stated as more than a descriptive observation about these particular outputs. I would ask that the abstract's wording be brought in line with this limitation (the body already is).","section":"§4 (A4), §5.5, §7"}],"minor_comments":[{"comment":"There is a spacing typo in 'F astAPI' that should read 'FastAPI'.","section":"§3.3"},{"comment":"The text says declarative Terraform came in '30 points above' the boto3 script, but the subsequent AWS-defaults correction reduces the gap to 17 points; the sentence should acknowledge the corrected figure in the same paragraph to avoid an apparent inconsistency.","section":"§5.1, Table 2"},{"comment":"The abstract states that Opus 5 improved on the two weakest cases 'by less than the sentence was worth,' but for the upload case the improvement (25 points) equals the directive's effect for Opus 5 on that case, not less; the wording should be adjusted to 'at most as much as.'","section":"§5.5, Abstract"},{"comment":"The phrase 'the −0.75 deficit therefore bounds total error from below' is confusing, since the deficit is only with respect to the scorer's negative errors and the paper explains that over-credits are unmeasured; consider rewording to 'the −0.75 mean signed error quantifies the known under-credit direction but does not bound the unknown over-credit rate.'","section":"§6"},{"comment":"The discussion of the Opus 5 arm's deviations from the original conditions is thorough, but the phrase 'its fresh-session requirement was met by isolated clean-context sessions in place of separate interactive ones' would benefit from one concrete example of how the sessions differed (e.g., new process versus new conversation), since the contamination risk is otherwise hard to assess.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely transparent empirical paper. Rubrics frozen before generation, predictions registered before the RDS/upload and Opus 5 arms, prompts and all 24 artifacts published, failures hand-verified, limitations stated plainly. That is real and rare, and it makes the measurement usable even where I disagree with the framing. Second, the headline causal claim over-reaches. The directed condition is the neutral prompt plus one sentence that declares the data sensitive and the artifact subject to SOC 2. The abstract and conclusion say \"one sentence naming the standard,\" but there is no condition that varies \"data sensitive\" alone or \"SOC 2\" alone. The 23-50 point gain in every cell is compatible with the sensitivity language alone triggering generic secure-coding behavior; naming the standard could be doing little or none of the work. Section 7 flags single-phrasing sensitivity but does not list a sensitive-only or SOC 2-only ablation, so the pre-committed paraphrase run won't fix the confound either. This does not kill the practical finding — one sentence containing both sensitivity and SOC 2 moves every case, and the consistency across eight cells is convincing on direction. But the mechanism claim, that compliance knowledge is latent and needs the standard itself named, is not identified by this design.\n\nWhat is new and worth credit: scoring whole artifacts against a rubric mapped to specific Trust Services Criteria, rather than CWE-level vulnerability presence; the neutral-versus-directed contrast (confounded as above); and the careful documentation that a regex scorer under-credits semantically compliant code and passes a real defect expressed through a conditional (X4). The vulnerability findings X1–X3 are file:line citations in unchanged artifacts, hand-verified, and the paper correctly reports them as existence claims, not rates.\n\nThe soft spots are the confound, the n=1 per cell design, and the partially asymmetric human audit. The paper discloses the latter two and pre-commits to more generations and a full audit, which is the right response. The n=1 issue makes exact percentages and the model-choice null one draw, but the prompt effect is 3-7 items per cell, so its direction and rough size are probably stable. The R3 pass asymmetry means some over-credits may persist; they say this too. Those are disclosed limitations, not hidden defects.\n\nThis paper deserves a serious referee. I would send it, but with a request for a sensitivity-only/SOC2-only ablation and repeated generations before the causal language is accepted. For a reading group it is a good case study in pre-registration and honest limitation reporting.","headline":"Honest, transparent measurement of SOC 2 prompt effects, but the \"naming the standard\" mechanism is confounded with data sensitivity and needs an ablation before the causal story holds.","tokens_in":16088,"tokens_out":3285,"would_cite":true,"duration_ms":29139,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When no one mentions security, frontier models omit SOC 2 controls; one sentence naming the standard moved every case from 47–88% to 86–100%.","keywords":["SOC 2 compliance","AI code generation","prompt engineering","secure code generation","compliance controls","Trust Services Criteria","static analysis","Terraform"],"falsifier":"Run each of the four use cases at least three times per cell, neutral and directed, and compare the score distributions. If any directed draw lands inside the neutral range, or any neutral draw reaches the directed range, the claimed 23-to-50-point directive effect is not stable; a replication that finds neutral code matching directed conformance would also falsify the claim that the standard has to be named.","tokens_in":15179,"feed_emoji":"🔒","tokens_out":7730,"duration_ms":64675,"temperature":0.7,"pith_summary":"This paper asks whether code generated by frontier language models carries the controls a SOC 2 program would check, and how much one sentence naming the standard changes the answer. In a factorial test of three models across four use cases, code written from a neutral task statement satisfied 47–88% of the rubric items, with the missing controls clustering where security is not part of the normal coding idiom, such as S3 hardening calls, retention, and MFA hooks. The same task plus a single directive sentence declaring the data sensitive and subject to SOC 2 moved every cell to 86–100%, worth 23 to 50 points, and removed every insecure construction the paper found. The paper argues this places the compliance knowledge inside the model while its default behavior leaves the controls outside the prompt, and it reports that model choice mattered least among the variables measured.","feed_headline":"One SOC 2 sentence lifts compliant AI code to 86-100%","feed_subtitle":"Adding the standard-naming sentence gained 23-50 points on every case; model choice mattered least.","key_machinery":"The carrying mechanism is the SOC 2 directive: one appended sentence declaring the data sensitive and the artifact subject to SOC 2, held in context on first receipt, with no individual control named. Around it, the method builds 12-to-15-item binary rubrics per use case, each item mapped to a stated Trust Services Criterion and frozen before any generation, plus a three-rater scoring protocol — a regular-expression pattern scorer, a blinded semantic model grader, and human hand-verification — that separates what the prompt changes from what the measurement instruments miss. The rubrics and all prompts are published verbatim so readers can rescore the unchanged artifacts.","core_discovery":"The central claim is that compliance knowledge is latent, not automatic: when the prompt says nothing about security, these models produce code that runs, passes ordinary tests, and still fails a compliance review, because an omitted control leaves no wrong line of code behind. Conformance without prompting ranged from 47% to 88% and tracked whether a control is embedded in the code idiom — password hashing and storage_encrypted appear unasked, while four S3 hardening calls, retention, and MFA hooks do not. One sentence naming SOC 2 recovered most of the gap: directed conformance ran 86–100% in every cell, and three real vulnerabilities in the neutral code (a reachable debugger giving remote code execution, an unauthenticated download, and an endpoint returning all stored names and emails) all disappeared. A stable remainder survived the sentence, including MFA hooks, cookie flags, and account lifecycle items, which have to be named individually. The paper also establishes that a regular-expression compliance scorer is directionally biased, under-crediting well-engineered code in 21 of 27 disagreements with semantic grading and passing a defect whose non-compliant value was computed by a conditional, so plan-time or execution-time resolution and auditing of scanner passes are necessary.","pith_inferences":["If the prompt-shaped gap generalizes across vendors, the cheapest reliable fix is a persistent instruction file re-read each session rather than prompt text alone; the direct test the paper registered but did not run is placing such a file in the repository and rerunning the neutral prompts.","The idiom-account hypothesis — that conformance tracks whether the control is part of normal code — was generated from four data points after seeing them; a discriminating test would vary repository context and task wording to see whether defaults shift when surrounding code already carries compliance signal.","The application-layer defects (debugger, unauthenticated endpoints) sit beyond what plan-time policy gates can block, so even a well-gated infrastructure pipeline still needs semantic control verification for application code.","Because each cell is one non-deterministic generation, the exact percentages are single draws; the first replication with three generations per cell should report per-cell variance and paraphrase variants before fine rankings are taken seriously."],"forward_implications":["Teams auditing model-generated code with literal pattern matching will under-credit well-engineered code and can pass real defects; configuration values must be resolved semantically (plan output or AST) and a scanner's passes checked as carefully as its failures.","A generic compliance instruction is a ceiling, not a cure: controls outside the model's conception of the task, such as MFA hooks, cookie flags, and account lifecycle, must be named explicitly in the instruction set.","Model selection is the wrong lever for compliance: same-generation models finished within one rubric item in all eight comparisons, and the largest cross-generation difference (25 points) was smaller than the 23-to-50-point directive effect.","Environment is a stronger lever than instruction: declarative Terraform, where security attributes sit in the resource schema, beat an imperative boto3 script doing comparable work by 30 points, which points to secure-by-default wrappers as the most direct remediation.","Null commission results should be read as 'none of the enumerated acts': expanding the search list turned a null into three real vulnerabilities in unchanged files, so reported commission counts are lower bounds."],"supporting_citations":[{"why":"Supplies the baseline of roughly 40% of AI completions containing weaknesses, which the paper's control-omission framing extends to missing controls rather than wrong lines.","marker":"[1]"},{"why":"Shows security-oriented prompting reduces weaknesses, positioning the single SOC 2 sentence as the lightest intervention in that family.","marker":"[10]"},{"why":"Reports that static analyzers overestimate security of defended generations under adversarial prompting, motivating the paper's three-rater measurement design.","marker":"[16]"},{"why":"Provides the kappa agreement statistic used to compare the pattern scorer with the semantic grader.","marker":"[19]"},{"why":"Documents security weaknesses in AI-generated code shipped to real software projects, contextualizing why whole-artifact control scoring is needed.","marker":"[25]"}],"fun_headline_variants":["One SOC 2 sentence lifts AI code compliance to 86-100%","Unprompted AI code fails SOC 2; a single sentence fixes most","Saying 'SOC 2' once adds 23-50 points to AI code compliance","AI knows SOC 2 but needs a nudge: one sentence suffices","Unprompted AI code ships vulnerabilities; one sentence removes them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that one non-deterministic code generation per cell represents the model's behavior on that task; the paper states this is not defensible for fine differences and pre-commits to three generations per cell.","fun_headline_variants_meta":{"raw":{"variants":["One SOC 2 sentence lifts AI code compliance to 86-100%","Unprompted AI code fails SOC 2; a single sentence fixes most","Saying 'SOC 2' once adds 23-50 points to AI code compliance","AI knows SOC 2 but needs a nudge: one sentence suffices","Unprompted AI code ships vulnerabilities; one sentence removes them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2584,"prompt_tokens":1196,"completion_tokens":1388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":812,"completion_tokens_details":{"reasoning_tokens":1285}},"tokens_in":812,"tokens_out":1388,"duration_ms":11036,"temperature":1.0,"reasoning_tokens":1285,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:12:52.385461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the four use cases at least three times per cell, neutral and directed, and compare the score distributions. If any directed draw lands inside the neutral range, or any neutral draw reaches the directed range, the claimed 23-to-50-point directive effect is not stable; a replication that finds neutral code matching directed conformance would also falsify the claim that the standard has to be named.","supporting_citations":[{"cited_title":"Asleep at the Keyboard? Assessing the Secu- rity of GitHub Copilot’s Code Contributions,","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline of roughly 40% of AI completions containing weaknesses, which the paper's control-omission framing extends to missing controls rather than wrong lines."},{"cited_title":"How Secure is Secure Code Generation? Ad- versarial Prompts Put LLM Defenses to the Test,","cited_arxiv_id":null,"evidence_quote":"Reports that static analyzers overestimate security of defended generations under adversarial prompting, motivating the paper's three-rater measurement design."},{"cited_title":"A Coefficient of Agreement for Nominal Scales,","cited_arxiv_id":null,"evidence_quote":"Provides the kappa agreement statistic used to compare the pattern scorer with the semantic grader."}],"review_version":1}