{"id":"74e7a9c0-708c-4497-a792-9f4539b23ce3","arxiv_id":"1908.03296","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 2019 evaluation of thirteen password managers found generation is mostly strong, but browser-based managers still suffer from insecure autofill and unencrypted metadata.","lead":"This paper tested thirteen password managers to see how securely they generate, store, and auto-fill passwords. It found that app and extension managers have improved, while browser-based managers still expose unencrypted metadata and are vulnerable to autofill attacks, so users should choose carefully.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Guessability findings rely on estimators not calibrated for random strings; the neural-net model was retrained on the generator's own outputs, so the 'generated passwords vulnerable to guessing' claim and the 10/18-character safe-length recommendations are not secure.","rationale":"The reader's weakest-assumption pick was in the same area: the guessability estimates from zxcvbn and the Melicher neural network are treated as ground truth for random generated passwords. I agree that this is the least secure part of the paper's central argument. However, the specific problem is slightly different and sharper: the paper does not just use a pre-trained human-password model; it retrains the Melicher LSTM on 80% of each manager's own generated corpus. That makes the model a distribution-aware performer rather than a realistic attacker, and the reported guesses are not directly comparable to an offline cracking attack in the wild. The zxcvbn calibration issue remains, and Table 5's /dev/random example suggests the estimator is far from the uniform brute-force bound. I do not think this invalidates the paper's broader conclusions about storage and autofill, which are based on direct observation and replicated attacks; those findings support the recommendation that browser-based password managers lag behind app- and extension-based managers. But the generation pillar, including the 'vulnerable to guessing' finding and the specific safe-length recommendations, is not established with the same rigor. That warrants keeping the paper's CONDITIONAL verdict rather than accepting the generation claims at face value. Since the reader already reached CONDITIONAL for overlapping reasons, my read does not move the verdict; I therefore mark it UNCHANGED rather than proposing a new disposition.","tokens_in":21159,"tokens_out":17716,"duration_ms":202088,"concrete_test":"Extract the weak examples from Table 5 and a random sample of the released corpus. For each example, compute the exact brute-force cost |A|^L/2 under the manager's declared alphabet, length, and composition constraint, and compare it with zxcvbn's guesses. Then re-run the RNN evaluation using a model trained only on a public human-password corpus (e.g., RockYou) instead of on 80% of the manager's own generated passwords, scoring the same held-out examples. If both the keyspace bound and the human-trained model place the length-12 'weak' passwords above 10^14 guesses while the paper's retrained model places them below, the vulnerability claim is an artifact of the training/estimation procedure. If the human-trained model also gives sub-threshold estimates for genuine random baselines such as /dev/random, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is the claim in Section 4 that some generated passwords are vulnerable to online and offline guessing attacks, together with the derived safe-length recommendations of 10 and 18 characters. Two estimators are used: zxcvbn and a Melicher LSTM. For the LSTM, Section 4.2 states: 'For each password corpus, we used 80% of the passwords to train the neural network and tested against 20% of the passwords' (see Listing 1). This is not an attacker model trained on human-chosen passwords; it is a model trained on the same generator's output distribution. A real offline attacker would not possess 80% of the manager's generated corpus before attacking a user's vault, so the resulting guess counts conflate predictability to a generator-aware modeler with vulnerability to a realistic adversary. zxcvbn, meanwhile, is designed to estimate guessing difficulty for human-chosen passwords; applying its per-password 'guesses' to uniformly random strings is not calibrated. Table 5 even lists a '/dev/rand' 12-character letter password at 10^9 guesses, while the uniform keyspace for a 12-character mixed-case letter password is over 10^20, indicating the heuristic is in the wrong regime by roughly 11 orders of magnitude. If the true cost of guessing these passwords is close to the uniform keyspace bound, the 'random but weak' passwords are not a security vulnerability and the recommended lengths are arbitrary; if the estimators are instead optimistic in the other direction, the recommended lengths could be unsafe. Since the abstract advertises instances of generated passwords vulnerable to guessing as one of the paper's headline findings, this calibration issue directly bears on the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a security evaluation of thirteen password managers across the full lifecycle of password generation, storage, and autofill. The authors generated 147 million passwords through available interfaces, source-code modifications, and Selenium automation, then analyzed this corpus with Shannon entropy, chi-square tests, zxcvbn, and a Melicher-style LSTM guesser. They also manually examined local vault encryption and metadata handling, and replayed and extended prior autofill attacks including clickjacking, cross-origin iframe filling, and insecure form filling. The paper reports that generated passwords are generally strong at length 12 and above, but that some short generated passwords are vulnerable to online and offline guessing attacks, and it recommends length 10 for online resistance and length 18 for offline resistance. For storage and autofill, the authors find improvements over prior work but also remaining problems, especially in browser-based password managers, including unencrypted metadata, insecure defaults, and clickjacking-vulnerable autofill behavior.","tokens_in":21455,"tokens_out":12876,"duration_ms":126936,"significance":"If the results hold, this is the most comprehensive lifecycle-oriented evaluation of password managers to date and a timely update to five-year-old prior studies. The study has clear strengths: a very large generated-password corpus, published data and analysis scripts, replication of prior attack definitions, and explicit handling of external benchmarks rather than self-referential validation. The autofill and storage findings are concrete, and the paper's recommendation structure (which managers to avoid, which settings matter) is actionable. The main source of uncertainty is the password-generation guessability analysis, which currently rests on estimators that are not calibrated for random strings; that issue affects the paper's headline length recommendations and the more general claim that generated passwords are vulnerable to guessing attacks.","major_comments":[{"comment":"The generation-strength results and the safe-length recommendations of 10 and 18 characters are derived from guess counts produced by zxcvbn, but the authors' own /dev/random baseline contradicts the calibration of this tool for random strings. The baseline password 'MrKNxQNDAViS' is a 12-character mixed-case random password with a uniform keyspace of 52^12 ≈ 10^20.6, yet Table 5 reports only 10^9 guesses. A tool that underestimates a uniform-random baseline by roughly eleven orders of magnitude is not a sound basis for classifying manager-generated passwords as 'vulnerable to online and offline guessing attacks' or for setting length thresholds. Please reanalyze the corpus using a method appropriate for uniformly random strings, report how many passwords fall below the 10^6 and 10^14 thresholds under that method, and recompute the recommended lengths accordingly.","section":"Section 4.2, Table 5, Section 4.3"},{"comment":"The neural-network guess estimates are obtained by training on 80% of each manager's own generated corpus and testing on the remaining 20% (Listing 1). This is a defensible approximation of a generator-aware adversary, but the paper does not state that this is the intended adversary model or compare it with a uniform-baseline model, so the reader cannot distinguish genuine generator bias from model artifacts. Moreover, the memory failure means the RNN results cover only lengths 8 and 12; the statements in Section 4.3 about length 20 and the 18-character offline threshold, as well as the interpolated 10-character online threshold, rest entirely on zxcvbn. Please supply RNN estimates for length 20, or explicitly restrict the generation-strength conclusions to the tested lengths.","section":"Section 4.2, Listing 1, Section 4.3"},{"comment":"The storage evaluation examines only local vault files; Section 5 states that 'the cloud databases are not available to us for direct evaluation.' Because most extension-based and browser-based password managers synchronize vaults through cloud backends by default, the contribution claim of evaluating 'password storage' for thirteen managers is broader than the evidence presented. The Section 6.6 web-vault checks (CSP header inspection and a CSRF test that changes a session timeout parameter) do not substitute for a security evaluation of cloud-side storage. Please either narrow the storage claims in the abstract and contribution list, or add direct evaluation of the cloud backends.","section":"Section 5, Table 6"}],"minor_comments":[{"comment":"The tool name 'zxcvbn' is misspelled as 'zxcbvn' in multiple places in Section 4.2; please correct this throughout.","section":"Section 4.2"},{"comment":"The symbols 'G' and '#' are used in the tables without a legend, making it impossible for the reader to determine which symbol denotes a supported feature; please add an explicit legend.","section":"Table 1 and Table 2"},{"comment":"The captions for Figure 1 do not define the axes, the meaning of 'ld' and 'all', or the units of the log10 scale; please expand the caption so the plots are interpretable without referring to the body text.","section":"Figure 1"},{"comment":"The sentence 'These password managers are all similar in high-level functionality' is too vague; please give a concrete list of the shared and distinguishing features of Chrome, Edge, Firefox, Internet Explorer, and Opera.","section":"Section 3.3"},{"comment":"The authors state that Safari could not be scripted for password generation and that only 100 passwords were manually analyzed; this limitation should be restated in the contributions where the generation corpus is described, since Safari is otherwise included in the thirteen-manager evaluation.","section":"Section 4.2"},{"comment":"The claim that 'all extension-based password managers leak the email address used to log in to the password manager' would be easier to verify if the table or text identified the specific file or field where the email address appears.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the final version of a USENIX Security 2020 paper, since the title page carries the footnote 'This paper will appear at USENIX Security 2020.' If this is a journal submission, the editor should verify the relationship to the conference version and ask the authors to update that footnote and clarify the new material added for the journal. The main technical risk is the calibration of the guessability analysis in Section 4; with that reanalyzed, I would be supportive of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is the paper that finally looks at password generation in password managers, not just storage and autofill. The core finding—that some generated passwords are weak enough to be guessed—is real but overstated. The empirical work is careful and extensive: 147 million generated passwords, multiple statistical tests, and a replication of prior storage and autofill attacks. The authors publish their artifacts and disclose limitations, which deserves credit.\n\nThe real soft spot is the guessability analysis. The neural network is trained on 80% of each generator's own output and tested on the remaining 20%. That can detect patterns in the generator's output, but it is not an attacker model. A real attacker does not have 80% of a password manager's generated corpus. The paper uses the resulting guess counts to claim 'vulnerable to offline guessing' and to recommend 10/18-character minimum lengths. That inference is not justified.\n\nzxcvbn is a second issue. It is designed for human-chosen passwords, and the paper's own Table 5 shows a /dev/random 12-character mixed-case password estimated at 10^9 guesses, while the uniform keyspace is around 10^20. An eleven-order-of-magnitude error means the sentences 'the weakest passwords at length 12 are still vulnerable to offline attacks' and 'length 18 is safe' are not supported by the evidence. The thresholds may be conservative or unsafe; we simply do not know.\n\nNone of this kills the paper. The character-frequency analysis, the χ² testing, and the finding of non-uniform character distributions in several generators are solid and new. The storage and autofill replications give the field an updated picture and confirm that browser-based managers still lag. The recommendation to filter weak generated passwords is sensible no matter which estimator we trust.\n\nWho should read this: anyone who builds or evaluates password managers, and researchers working on password strength estimation. It deserves peer review, but the guessability section needs reworking—either calibrate the estimators on random baselines or present those results as 'patterns detected by a generator-aware model' rather than as real-world attack costs.\n\nRecommendation: send to peer review. Conditional accept at best; the generation study is worth publishing, but not with the current safety thresholds presented as clean results.","headline":"The generation analysis is a real first, but the guessability claims are built on estimators not calibrated for random strings, so the headline numbers are shaky.","tokens_in":21978,"tokens_out":5298,"would_cite":true,"duration_ms":52822,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Thirteen password managers still have significant security flaws in generation, storage, and autofill, with browser-based tools falling furthest behind.","keywords":["password managers","password generation","password storage","autofill security","clickjacking","guessability estimation","browser security","security evaluation"],"falsifier":"Generate the same 147-million-password corpora and run an exact brute-force search over each generator's character-selection algorithm for the short lengths, recording the true number of attempts needed to find each password; then compare those counts with the zxcvbn and neural-network estimates. If the estimates diverge, the 10- and 18-character safety thresholds need revision.","tokens_in":111,"feed_emoji":"🔐","tokens_out":9064,"duration_ms":144241,"temperature":0.7,"pith_summary":"Five years after earlier studies found serious flaws in password managers, this paper asks whether the tools have become safe enough for broad adoption. It argues they have improved but are not there yet: across thirteen popular managers, generated passwords occasionally fall to online and offline guessing, vault metadata is not always encrypted, and autofill remains vulnerable to clickjacking and, in browser-based managers, to credential harvesting. The evaluation covers the full password-manager lifecycle—generation, storage, and autofill—and its 147-million-password corpus is the first large-scale test of password generators inside managers. The practical stakes are direct: users who pick the wrong tool or leave default settings untouched can have credentials stolen with little effort.","feed_headline":"Password managers improved, but 13-tool test finds lingering flaws","feed_subtitle":"Generation, storage, and autofill still have holes—and browser-based tools lag farthest behind.","key_machinery":"The evaluating machinery is the password-manager lifecycle—generation, storage, and autofill—applied uniformly to thirteen tools. For generation, 147 million passwords were produced across character-class and length settings, then tested for non-randomness with Shannon entropy, a $\\chi^2$ test with Bonferroni correction, the zxcvbn estimator, and an LSTM recurrent neural network guesser; the guessability outputs are compared against thresholds of $10^6$ guesses for online attacks and $10^{14}$ for offline attacks. For storage, local vault files were manually inspected for encryption, key derivation, and metadata exposure. For autofill, the authors built websites that reproduce and extend the XSS, network-injection, iframe, and clickjacking attacks from prior work, recording whether each manager requires user interaction and whether it refuses to fill in cross-origin iframes or mismatched forms.","core_discovery":"The paper's central claim is that while app- and extension-based password managers have addressed many vulnerabilities found in prior work, serious security problems remain, and browser-based managers lag well behind. In password generation, nearly all passwords of length 12 or longer resist online and offline guessing, yet shorter generated passwords can be randomly weak; the authors recommend a minimum of 10 characters for online resistance and 18 for offline resistance. In storage, most extension-based managers now encrypt their vaults with AES-256, but metadata such as extension settings, website icons, or email addresses leaks, and Chrome and Opera on Linux fall back to plaintext password storage when no keyring is available. In autofill, only 1Password X and Safari always require user interaction, several managers autofill inside same-origin iframes and can be attacked by clickjacking—overlaying invisible page elements to trick users into approving the fill—and Firefox's built-in manager will autofill cross-origin iframes by default, exposing all stored credentials to a network-injection or cross-site scripting (XSS) attack. The paper concludes that users should avoid Firefox's built-in manager and prefer app- or extension-based managers, and that no tested manager implements the nonce-based autofill defense recommended in prior work.","pith_inferences":["Editorial inference: the 10- and 18-character thresholds inherit the guess estimates of zxcvbn and the neural network, both trained on human-chosen passwords; a brute-force enumeration of a generator's exact algorithm could give a firmer bound for random strings and might shift the thresholds.","Editorial inference: the same 147-million-password corpus could be re-analyzed for collisions or partial overlaps among managers, which would matter if many users generate passwords for the same sites.","Editorial inference: because no tested manager implements nonce-based autofill and extensions cannot rewrite request bodies, the practical path to fixing XSS-exposed autofill is a browser-level API rather than a password-manager change.","Editorial inference: an analogous mobile evaluation is a direct next step; this study only covers desktop environments, and mobile autofill surfaces (keyboards, intents, app links) have different attack vectors."],"forward_implications":["Users who rely on browser-based password managers (Chrome, Edge, Firefox, Internet Explorer, Opera, Safari) face risks that app- and extension-based managers have mostly eliminated; Safari is the notable exception.","Generated passwords shorter than 10 characters can be vulnerable to online guessing and shorter than 18 characters to offline guessing, so managers should filter out easily guessable results or raise their default lengths.","Autofill without user interaction should be the exception, not the default: Dashlane and LastPass default to it, and Bitwarden and RoboForm allow users to disable the interaction requirement.","Clickjacking remains a live threat for same-origin iframe autofill; moving the confirmation outside the web page, as Bitwarden and RoboForm do, is the pattern that resists it.","Firefox's built-in manager, which by default autofills cross-origin iframes, is vulnerable to a password harvesting attack that can drain the whole vault from a compromised network login page."],"supporting_citations":[{"why":"Established the earlier web-vault and bookmarklet vulnerabilities that this paper re-tests across the five extension managers.","marker":"[19]"},{"why":"Defined the auto-fill attack model (XSS, network injection, clickjacking, harvesting) that the paper reproduces and extends.","marker":"[29]"},{"why":"Independently demonstrated XSS-based auto-fill abuse and proposed the nonce-replacement defense; the paper checks that no manager implements it.","marker":"[31]"},{"why":"Provided the earlier vault-format and metadata-privacy analysis that the paper partially replicates for password storage.","marker":"[17]"},{"why":"Supplied the recurrent neural network password guesser used to estimate guess counts for generated passwords.","marker":"[22]"},{"why":"Supplied zxcvbn, used to detect dictionary/pattern weaknesses and to estimate guessing difficulty.","marker":"[34]"},{"why":"Provided the $10^6$ and $10^{14}$ guess thresholds used to classify passwords as online- or offline-vulnerable.","marker":"[16]"},{"why":"Documents that Chromium-based browsers store passwords in plaintext on Linux when no keyring is available, supporting the storage finding for Chrome and Opera.","marker":"[9]"}],"fun_headline_variants":["Browser password managers still expose metadata and clickjacking risks","Generation weak, storage leaks, autofill at risk: 13 password managers","Password generation still weak in managers, study of 13 tools finds","Avoid Firefox's password manager, study of 13 tools recommends"],"cache_read_input_tokens":24064,"weakest_assumption_plain":"The paper treats zxcvbn's and the neural network's guess-count estimates as reliable measures of how hard generated random passwords are to crack; if those estimators overstate difficulty for random strings, the recommended safe lengths of 10 and 18 characters would be too low.","fun_headline_variants_meta":{"raw":{"variants":["Browser password managers still expose metadata and clickjacking risks","Generation weak, storage leaks, autofill at risk: 13 password managers","Password generation still weak in managers, study of 13 tools finds","Avoid Firefox's password manager, study of 13 tools recommends"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001131,"raw_usage":{"total_tokens":4731,"prompt_tokens":1006,"completion_tokens":3725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":3650}},"tokens_in":622,"tokens_out":3725,"duration_ms":27390,"temperature":1.0,"reasoning_tokens":3650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:18:03.528853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate the same 147-million-password corpora and run an exact brute-force search over each generator's character-selection algorithm for the short lengths, recording the true number of attempts needed to find each password; then compare those counts with the zxcvbn and neural-network estimates. If the estimates diverge, the 10- and 18-character safety thresholds need revision.","supporting_citations":[{"cited_title":"On the security of password manager database formats","cited_arxiv_id":null,"evidence_quote":"Established the earlier web-vault and bookmarklet vulnerabilities that this paper re-tests across the five extension managers."},{"cited_title":"Password managers: Attacks and defenses","cited_arxiv_id":null,"evidence_quote":"Defined the auto-fill attack model (XSS, network injection, clickjacking, harvesting) that the paper reproduces and extends."},{"cited_title":"End-to-end passwords","cited_arxiv_id":null,"evidence_quote":"Independently demonstrated XSS-based auto-fill abuse and proposed the nonce-replacement defense; the paper checks that no manager implements it."},{"cited_title":"A large-scale study of web password habits","cited_arxiv_id":null,"evidence_quote":"Provided the earlier vault-format and metadata-privacy analysis that the paper partially replicates for password storage."},{"cited_title":"Fast, lean, and accurate: Modeling password guessability using neural networks","cited_arxiv_id":null,"evidence_quote":"Supplied the recurrent neural network password guesser used to estimate guess counts for generated passwords."},{"cited_title":"Password-manager friendly (pmf): Semantic annotations to improve the effectiveness of password managers","cited_arxiv_id":null,"evidence_quote":"Supplied zxcvbn, used to detect dictionary/pattern weaknesses and to estimate guessing difficulty."},{"cited_title":"An administrator’s guide to internet password research","cited_arxiv_id":null,"evidence_quote":"Provided the $10^6$ and $10^{14}$ guess thresholds used to classify passwords as online- or offline-vulnerable."},{"cited_title":"Linux password storage","cited_arxiv_id":null,"evidence_quote":"Documents that Chromium-based browsers store passwords in plaintext on Linux when no keyring is available, supporting the storage finding for Chrome and Opera."}],"review_version":1}