{"id":"ccf5462c-1372-4773-b530-4043cea3d48f","arxiv_id":"2506.08320","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM-generated pwquality.conf password policies are frequently inconsistent, hallucinated, and incorrect, so they require validation before deployment in Linux PAM systems.","lead":"Seven popular large language models were asked to turn plain-English password rules into a Linux password-policy file, and the resulting files were often inconsistent, incomplete, or filled with invented settings. The study matters because organizations are starting to trust AI for security configuration, and these results show that such output still needs careful validation before use.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2's comparison is undefined for hallucinated parameters that appear in only one of the compared files, so the consistency and hallucination metrics as published are not reproducible.","rationale":"The reader's weakest-assumption analysis pointed to the manually constructed benchmark, which is a legitimate reproducibility concern but leaves open the possibility that the benchmark, even if unreleased, is correct. My stress-test identifies a more immediate and internal problem: the published comparison algorithm (Algorithm 2) is undefined for hallucinated parameters that appear in only one of the two files being compared. Since hallucination is by definition the production of such parameters, the algorithm cannot process the central phenomenon of the paper. This is not merely a missing artifact; it is an inconsistency in the method as described. The qualitative warning that LLMs produce inconsistent and non-deployable pwquality.conf files is still credible—Section V-A gives concrete, plausible examples of blank files, missing '=' signs, and bracketed section headers. Those examples alone support the paper's high-level conclusion. However, the quantitative comparisons (which model is most consistent, which hallucinates most, whether documentation helps) are not verifiable from the manuscript. Because the flaw is repairable by publishing the implementation or a corrected, unambiguous algorithm, the appropriate verdict is CONDITIONAL rather than REJECT. The paper should be accepted only after the algorithm is clarified, the implementation is released, and the headline numbers are recomputed.","tokens_in":13435,"tokens_out":5938,"duration_ms":64063,"concrete_test":"Re-implement Algorithm 2 exactly as written and run it on two synthetic files: File A contains `minlen=8` and a hallucinated key `check_userpass=1`; File B contains only `minlen=8`. The union of keys includes `check_userpass`, but resps2 has no such key, so the pseudocode fails with a KeyError. Then either (a) obtain the authors' actual implementation and recompute Figures 3, 4, and 6 with the missing-key rule explicitly documented, or (b) supply a corrected algorithm and verify that the per-model rankings in Figure 6, including the claim that Cohere hallucinates the most, remain unchanged. If any model's average hallucination count shifts by more than 10%, the published conclusions are not robust to the specification of this undefined step.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Algorithm 2 (Response Comparison) initializes resps1 and resps2 as copies of the default parameter dictionary, then overlays each response's parameters. For every key in the union of both dictionaries, it evaluates resps1[k] == resps2[k]. A hallucinated parameter, by definition, is not in the default dictionary. When such a parameter appears in one response but not the other (e.g., Cohere's `check_userpass` appears in one generation but not in a paired generation), the key is in the union but missing from the other dictionary. The pseudocode provides no fallback, so this comparison raises a KeyError. To avoid the crash, an implementation must deviate from the pseudocode, e.g., by treating the missing key as absent or as a sentinel value. That choice materially changes the metric: treating missing as unequal makes the hallucinated parameter increment `num_hal` and affects `num_same`, while excluding it entirely changes the denominator `len`. The paper does not state which behavior was used. The same issue propagates into Algorithm 3, since correctness reuses Response Comparison with the benchmark as Response2, and any hallucinated parameter in a generated file is absent from the benchmark. Consequently, Figures 3, 4, 5, and 6 are not reproducible from the published algorithms. This is not a minor prose gap: the central quantitative claims about which models hallucinate most and which are most consistent rest on an algorithm whose behavior is undefined for the very phenomenon (hallucination) it claims to measure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks how well LLMs translate natural-language password policies into Linux pwquality.conf files. It defines four properties — consistency, correctness, hallucination, and incompleteness — and aggregates them into a notion of \"soundness.\" The experiments prompt seven LLMs (GPT-4o-mini, GPT-o3-mini, Cohere, Gemini, DeepSeek, Llama-3, Bloom) five times per policy prompt, both with and without the official pwquality.conf documentation, and compare outputs pairwise and against manually constructed gold-standard files using three algorithms. The reported results indicate that models such as Bloom return blank files, Cohere hallucinates parameters such as check_userpass, and only the OpenAI models approach high consistency and correctness; the paper concludes that LLM-generated policies are not yet sound for direct deployment. The qualitative observation that LLMs are inconsistent and hallucinate on this task is credible from the examples, but the quantitative pipeline as published is not reproducible because key algorithmic operations are undefined and no raw data or benchmark files are provided.","tokens_in":13660,"tokens_out":5218,"duration_ms":59386,"significance":"If the results are made reproducible, this is a useful and timely contribution: it studies a concrete, security-relevant configuration task rather than abstract consistency, and the two-framework design (with and without official documentation) provides a practical estimate of whether supplying documentation helps or hurts. The paper's strengths include concrete failure examples (missing '=' signs, bracketed headers, blank responses), explicit consideration of PAM failsafe behavior, and the attempt to define soundness for generated configurations. However, the paper currently ships no code, no benchmark files, and no raw outputs; the central metric algorithms have undefined cases; and the statistical support consists of point averages over five generations without variance, confidence intervals, or significance tests. These issues must be fixed before the quantitative claims can be relied upon.","major_comments":[{"comment":"The comparison `resps1[k] == resps2[k]` is undefined for hallucinated parameters that occur in only one of the two responses, because the dictionaries are initialized from the default parameter list and contain no entry for the missing key. A hallucinated parameter such as Cohere's `check_userpass` (Section V-A and V-D) will be present in one response but absent from the other, so this line raises a KeyError unless the implementation silently deviates from the pseudocode. The paper does not state whether missing keys are treated as absent, as a sentinel, or as unequal, and each choice changes `num_hal`, `num_same`, and the denominator `len`. Since the same procedure is reused by Algorithm 3 with the benchmark as Response2, this issue propagates to the correctness and incompleteness metrics in Figures 3–6. The published algorithms are therefore not reproducible for the central quantitative claims.","section":"Algorithm 2, lines 12–14"},{"comment":"The notation `total += ResponseComparison(...)` is ambiguous. If `total` is a Python list, `+=` concatenates, and then `return total[2]/iterations` in Algorithm 3 returns the third component of the first iteration only, not an average over iterations. If element-wise vector addition is intended, the pseudocode must say so explicitly. The same ambiguity affects Algorithm 1, where `total[0]/sum`, `total[1]/sum`, and `total[2]/sum` are returned as averages. This is load-bearing because every reported consistency, hallucination, and correctness score is computed from these accumulations.","section":"Algorithms 1 and 3, accumulation and return statements"},{"comment":"All quantitative claims in Section V are based on five generations per model and prompt, yet the paper reports only point averages without error bars, confidence intervals, significance tests, or raw output data. With n = 5, statements such as \"o3-mini outscoring 4o-mini\" in Section V-C and \"Cohere is actually the stronger performer\" in Section V-B may reflect sampling noise. At minimum, the paper should show per-generation distributions or bootstrapped intervals and release the raw generated files for independent analysis. This is not a presentation nicety: the ordering of models by consistency and correctness is the paper's main quantitative result.","section":"Section V-B and V-C, statistical reporting"},{"comment":"The gold-standard benchmark files are manually constructed by the authors, are not released, and are not independently validated. Correctness is therefore a measure of agreement with one team's translation of natural-language policies into pwquality.conf parameters; if that translation is wrong or idiosyncratic, the correctness numbers in Figure 5 do not measure what they claim. The paper should release the benchmark files, the mapping from each policy sentence to the chosen parameters, and ideally inter-annotator agreement on the translation step.","section":"Section III-A.2 and Figure 5, benchmark validity"},{"comment":"Section III-C defines an LLM-generated policy as sound \"if it generates functionally equivalent files,\" while Section IV-A defines a configuration as sound if it meets all four criteria (consistency, correctness, absence of hallucination, and completeness). These are different definitions, and the Conclusion uses the four-criteria version. The paper should explicitly reconcile the two definitions or state that they refer to different objects (policy soundness versus configuration soundness); otherwise the aggregate soundness claim is ambiguous.","section":"Section III-C vs Section IV-A, definition of soundness"}],"minor_comments":[{"comment":"The expression `resps1.keys() + resps2.keys()` is not valid Python and is unclear as pseudocode; write it as the union of the two key sets.","section":"Algorithm 2, line 12"},{"comment":"The text and Figures 7–9 refer to \"Prompt 4 vs Prompt 5,\" but Table II labels the two outlier policies as P1 and P2. Please align the numbering.","section":"Section V-E and Table II"},{"comment":"Reference [4] is cited as \"A. L. et. al.\" and reference [35] as \"B. Workshop, :, and T. L. S. et. al.\"; both author lists are corrupted and need to be completed.","section":"References"},{"comment":"The text notes that Bloom returns blank files and that this \"still doesn't score badly on accuracy,\" but Figure 5 shows Bloom at or near 100% for the outlier prompts. Please clarify that blank files are counted as fully correct by construction because missing parameters are mapped to defaults, and discuss whether this conflates incompleteness with correctness.","section":"Section V-A and Figure 5"},{"comment":"Write the number of pairs as `iterations * (iterations - 1) / 2` to avoid any ambiguity about operator precedence.","section":"Algorithm 1, line 9"},{"comment":"Model names are spelled inconsistently: \"Deepseek\" vs \"DeepSeek\" and \"GPT o3-mini\" vs \"o3-mini.\" Standardize the naming.","section":"Throughout Section V"}],"recommendation":"major_revision","confidential_remarks":"The qualitative finding — that LLM-generated pwquality.conf files contain hallucinations and inconsistencies — is plausible and worth publishing once the quantitative machinery is repaired. The undefined behavior in Algorithm 2, the ambiguous accumulation in Algorithms 1 and 3, the missing statistical support, and the unreleased benchmark make the current quantitative claims unsuitable in their present form. These are fixable within the manuscript's scope: clarify the algorithm semantics, release the benchmark and raw outputs, and add error bars or significance tests. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core message is believable and practically relevant: modern LLMs do not reliably translate natural-language password policies into correct pwquality.conf files. The concrete failure modes they document—Cohere dropping the equals sign, Llama3 adding bracketed section headers, Bloom returning blank files—are exactly the kind of thing a busy sysadmin would hit. Second, the quantitative evaluation as published is not reproducible. The stress-test note is right: Algorithm 2 initializes two copies of the default parameter dictionary, overlays each response, then iterates over the union of keys. If a hallucinated parameter appears in one response but not the other, the missing key produces a KeyError. The paper never states how this case was handled, and the choice (treat missing as unequal, or exclude it) materially changes the consistency and hallucination scores. The same problem spills into Algorithm 3, since the benchmark file won't contain hallucinated parameters. So Figures 3–6 rest on undefined behavior.\n\nWhat's genuinely new: applying consistency and hallucination evaluation to a specific security-critical configuration file, and the documentation-augmentation comparison is a clean idea. The paper is honest about some limitations in Section V-F. The impact table is a bit hand-wavy but harmless.\n\nSoft spots in proportion: the missing data and code are a bigger deal than the sample size. Five generations per cell is small but not crazy for a first look; the lack of error bars is annoying but not fatal if raw numbers were available. The benchmark files are not released, so correctness percentages are unverifiable. Also, their own definition of soundness via functional equivalence (Section III-C) is not what Algorithm 2 measures: exact parameter equality will flag defaults-as-explicit versus omitted as different in some cases. That's a modeling gap, not a fatal one.\n\nWho this is for: people building or auditing AI-assisted sysadmin tools, and researchers working on LLM consistency for structured outputs. It deserves a serious referee, but only with the promise that the authors will release prompts, benchmark files, raw outputs, and a corrected algorithm. If they do, this could be a solid empirical data point. As it stands, I'd treat the qualitative warning as credible and the specific model rankings as not yet established.\n\nRecommendation: send to peer review with a request for major revision and mandatory artifact release.","headline":"A useful warning buried under an unreproducible evaluation: the qualitative finding is credible, but the published algorithms crash on hallucinated parameters and no data or code are released.","tokens_in":14210,"tokens_out":2317,"would_cite":false,"duration_ms":26695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper uses parameter-by-parameter comparison to show that LLM-generated pwquality.conf password policies are frequently inconsistent, hallucinated, incomplete, or incorrect, and therefore not sound for direct deployment in Linux PAM.","keywords":["LLM-generated configuration files","pwquality.conf","Linux PAM","password policy","consistency evaluation","hallucination","correctness benchmark","access control"],"falsifier":"Take the same natural-language policy prompts, have independent Linux administrators write their own pwquality.conf files by hand, and compare those files to the paper's benchmarks; if the independent translations disagree substantially, the reported correctness numbers are not a stable measure of LLM accuracy. A complementary runtime check is to feed each generated file through an actual pam_pwquality.so test stack and compare the enforced password rules to the intended policy, since malformed assignments can make PAM fall back to weaker defaults without any error being shown.","tokens_in":13172,"feed_emoji":"🔐","tokens_out":7571,"duration_ms":84175,"temperature":0.7,"pith_summary":"This paper asks whether a large language model can be trusted to translate a natural-language password policy into a pwquality.conf file that Linux's PAM password-quality module will actually enforce. The paper defines a generated configuration as sound only if repeated generations are functionally equivalent, every parameter matches a human-built gold-standard file, no fabricated parameters appear, and no required parameter is missing. In tests of several LLMs, with and without the official pwquality.conf documentation supplied in the prompt, output quality varied widely: some models were largely consistent, while others hallucinated parameters, omitted assignments, or emitted files that PAM would reject or silently override. The paper concludes that LLM-generated password policies are not ready for direct deployment and that a validation step is needed before the files are placed into system configuration.","feed_headline":"LLM-written Linux password policies often fail consistency checks","feed_subtitle":"A multi-model test of pwquality.conf generation shows that even the better outputs need validation before PAM enforces them.","key_machinery":"The load-bearing object is pwquality.conf, the configuration file read by the pam_pwquality.so PAM module, whose parameters such as minlen and difok govern password complexity and which silently falls back to built-in defaults when the file is absent, malformed, or contains an invalid assignment. The argument is carried by three comparison algorithms: one generates several responses per prompt and averages pairwise comparisons; a second parses each generated file into a parameter-value hashmap, overlays documented defaults, and counts matching, differing, and hallucinated parameters; a third compares each generated file against a manually constructed gold-standard benchmark to score correctness and incompleteness. These algorithms turn the question of whether an LLM output is trustworthy into countable differences in parameter assignments.","core_discovery":"The paper's central claim is that current LLMs cannot yet be relied on to produce sound password-policy configuration files, and that their failures take specific, measurable forms. Consistency is defined as functional equivalence: two generated files are consistent when they enforce the same effective parameter values, even if one writes minlen=8 and the other omits it because PAM supplies the same default. Correctness is scored parameter-by-parameter against a manually written benchmark pwquality.conf for each natural-language prompt, hallucination is the presence of parameters the real file does not define, and incompleteness is any missing parameter that silently falls back to a default. Across the models tested, the two best-scoring models generally scored highest on accuracy and lowest on hallucination, one model that hallucinated the most was nonetheless the most functionally consistent, and one model that refused to generate content scored well only because blank files inherit the defaults. Supplying the official documentation did not reliably help, and for some models it increased hallucination. The paper concludes that some LLMs show high consistency while others show discrepancies and hallucinations that could lead to inconsistent access control enforcement across a network.","pith_inferences":["A runtime validation harness that actually calls pam_pwquality.so would be a stronger check than parameter comparison alone, because the paper's own outlier examples show that a file can look correct yet be rejected or overridden by the PAM module.","The blank-file model scoring near-perfect accuracy exposes a scoring hazard: prompts whose intended settings are close to PAM defaults reward empty output, so future benchmarks should force at least one parameter away from its default to distinguish real generation from non-generation.","If LLM-generated configuration files become common, prompt injection becomes a plausible attack: a poisoned prompt could make a model emit a hallucinated parameter that causes PAM to fall back to weak defaults, a direction the paper names as future work but does not test.","Because the documentation effect varies by model, a single recipe for adding reference documentation to prompts is unlikely to work for every model; per-model grounding strategies would be a natural next step."],"forward_implications":["An administrator who copies an LLM-generated pwquality.conf into a Linux system without validation risks a password policy that is weaker than the one requested, because missing or invalid parameters make PAM fall back to built-in defaults.","Consistency evaluations should compare effective enforced behavior rather than raw text, since hallucinated parameters PAM ignores do not change enforcement, while a real parameter set to an invalid value can disable the whole file.","Supplying official documentation is not a reliable corrective: the paper finds the effect is model-dependent, sometimes reducing hallucination and sometimes increasing it.","The same parameter-by-parameter comparison method can be carried over to other security-sensitive configuration files, such as login.defs or sshd_config, where a misconfiguration could open a vulnerability.","Wide variance between models means an organization cannot adopt 'LLM-generated configs' as a category; each model needs individual validation before use."],"supporting_citations":[{"why":"Defines the valid parameters, defaults, and assignment syntax that ground both the augmentation prompts and the correctness comparison.","marker":"[10]"},{"why":"Prior study of LLM consistency that the paper extends from plain-text answers to configuration-file parameter assignments.","marker":"[22]"},{"why":"Supplies one of the natural-language password policy descriptions used to build prompts and benchmark files.","marker":"[23]"},{"why":"Supplies another natural-language policy source used for prompts and gold-standard files.","marker":"[13]"},{"why":"Supplies a further natural-language password policy source used to construct prompts and benchmarks.","marker":"[25]"},{"why":"Supplies the government-issued password guideline text used as a prompt, including the verbose red-herring prompt examined in the paper.","marker":"[17]"},{"why":"Supplies an industry password policy guide used in the prompt set.","marker":"[3]"},{"why":"Supplies the Linux default password policy text used as the default-setting prompt.","marker":"[11]"},{"why":"Provides the notion of hallucination that the paper adapts to count fabricated configuration parameters.","marker":"[14]"}],"fun_headline_variants":["LLMs fail real-world test for password policy configs","AI password policies: inconsistent, hallucinated, incomplete","LLM password configs break consistency checks","Even with docs, LLMs hallucinate password rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The manual translation of each natural-language password policy into a gold-standard pwquality.conf file is correct; if that translation is wrong, the correctness and soundness scores do not measure what they claim to measure.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail real-world test for password policy configs","AI password policies: inconsistent, hallucinated, incomplete","LLM password configs break consistency checks","Even with docs, LLMs hallucinate password rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2877,"prompt_tokens":972,"completion_tokens":1905,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1843}},"tokens_in":588,"tokens_out":1905,"duration_ms":14995,"temperature":1.0,"reasoning_tokens":1843,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:13:22.562401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same natural-language policy prompts, have independent Linux administrators write their own pwquality.conf files by hand, and compare those files to the paper's benchmarks; if the independent translations disagree substantially, the reported correctness numbers are not a stable measure of LLM accuracy. A complementary runtime check is to feed each generated file through an actual pam_pwquality.so test stack and compare the enforced password rules to the intended policy, since malformed assignments can make PAM fall back to weaker defaults without any error being shown.","supporting_citations":[{"cited_title":"Linux, pwquality.conf(5) — Arch Manual Pages, n.d., accessed: 2025- 03-28","cited_arxiv_id":null,"evidence_quote":"Defines the valid parameters, defaults, and assignment syntax that ground both the augmentation prompts and the correctness comparison."},{"cited_title":"Automated consistency analysis of llms,","cited_arxiv_id":null,"evidence_quote":"Prior study of LLM consistency that the paper extends from plain-text answers to configuration-file parameter assignments."},{"cited_title":"Cyber security policy templates: Password security,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the natural-language password policy descriptions used to build prompts and benchmark files."},{"cited_title":"Successful password policies for organizations,","cited_arxiv_id":null,"evidence_quote":"Supplies another natural-language policy source used for prompts and gold-standard files."},{"cited_title":"Top 10 password policy best practices,","cited_arxiv_id":null,"evidence_quote":"Supplies a further natural-language password policy source used to construct prompts and benchmarks."},{"cited_title":"Digital identity guidelines: Authentication and lifecycle management (sp 800-63b),","cited_arxiv_id":null,"evidence_quote":"Supplies the government-issued password guideline text used as a prompt, including the verbose red-herring prompt examined in the paper."},{"cited_title":"Cis password policy guide: Passphrases, monitoring, and more,","cited_arxiv_id":null,"evidence_quote":"Supplies an industry password policy guide used in the prompt set."},{"cited_title":"Default password policy in linux systems (e.g., pam, /etc/login.defs),","cited_arxiv_id":null,"evidence_quote":"Supplies the Linux default password policy text used as the default-setting prompt."}],"review_version":1}