REVIEW 3 major objections 6 minor 40 references
That Was Then, This Is Now: A Security Evaluation of Password Generation, Storage, and Autofill in Thirteen Password Managers
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Thirteen password managers still have significant security flaws in generation, storage, and autofill, with browser-based tools falling furthest behind.
desk verdict The generation analysis is a real first, but the guessability claims are built on estimators not calibrated for random strings, so the headline numbers are shaky. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The evaluating machinery is the password-manager lifecycle—generation, storage, and autofill—applied uniformly to thirteen tools. For generation, 147 million passwords were produced across character-class and length settings, then tested for non-randomness with Shannon entropy, a $\chi^2$ test with Bonferroni correction, the zxcvbn estimator, and an LSTM recurrent neural network guesser; the guessability outputs are compared against thresholds of $10^6$ guesses for online attacks and $10^{14}$ for offline attacks. For storage, local vault files were manually inspected for encryption, key derivation, and metadata exposure. For autofill, the authors built websites that reproduce and extend the XSS, network-injection, iframe, and clickjacking attacks from prior work, recording whether each manager requires user interaction and whether it refuses to fill in cross-origin iframes or mismatched forms.
What would settle it
Generate the same 147-million-password corpora and run an exact brute-force search over each generator's character-selection algorithm for the short lengths, recording the true number of attempts needed to find each password; then compare those counts with the zxcvbn and neural-network estimates. If the estimates diverge, the 10- and 18-character safety thresholds need revision.
Extended reading notes
Core claim
The paper's central claim is that while app- and extension-based password managers have addressed many vulnerabilities found in prior work, serious security problems remain, and browser-based managers lag well behind. In password generation, nearly all passwords of length 12 or longer resist online and offline guessing, yet shorter generated passwords can be randomly weak; the authors recommend a minimum of 10 characters for online resistance and 18 for offline resistance. In storage, most extension-based managers now encrypt their vaults with AES-256, but metadata such as extension settings, website icons, or email addresses leaks, and Chrome and Opera on Linux fall back to plaintext password storage when no keyring is available. In autofill, only 1Password X and Safari always require user interaction, several managers autofill inside same-origin iframes and can be attacked by clickjacking—overlaying invisible page elements to trick users into approving the fill—and Firefox's built-in manager will autofill cross-origin iframes by default, exposing all stored credentials to a network-injection or cross-site scripting (XSS) attack. The paper concludes that users should avoid Firefox's built-in manager and prefer app- or extension-based managers, and that no tested manager implements the nonce-based autofill defense recommended in prior work.
Load-bearing premise
The paper treats zxcvbn's and the neural network's guess-count estimates as reliable measures of how hard generated random passwords are to crack; if those estimators overstate difficulty for random strings, the recommended safe lengths of 10 and 18 characters would be too low.
Editorial extensions
If this is right
- Users who rely on browser-based password managers (Chrome, Edge, Firefox, Internet Explorer, Opera, Safari) face risks that app- and extension-based managers have mostly eliminated; Safari is the notable exception.
- Generated passwords shorter than 10 characters can be vulnerable to online guessing and shorter than 18 characters to offline guessing, so managers should filter out easily guessable results or raise their default lengths.
- Autofill without user interaction should be the exception, not the default: Dashlane and LastPass default to it, and Bitwarden and RoboForm allow users to disable the interaction requirement.
- Clickjacking remains a live threat for same-origin iframe autofill; moving the confirmation outside the web page, as Bitwarden and RoboForm do, is the pattern that resists it.
- Firefox's built-in manager, which by default autofills cross-origin iframes, is vulnerable to a password harvesting attack that can drain the whole vault from a compromised network login page.
Reading between the lines
- Editorial inference: the 10- and 18-character thresholds inherit the guess estimates of zxcvbn and the neural network, both trained on human-chosen passwords; a brute-force enumeration of a generator's exact algorithm could give a firmer bound for random strings and might shift the thresholds.
- Editorial inference: the same 147-million-password corpus could be re-analyzed for collisions or partial overlaps among managers, which would matter if many users generate passwords for the same sites.
- Editorial inference: because no tested manager implements nonce-based autofill and extensions cannot rewrite request bodies, the practical path to fixing XSS-exposed autofill is a browser-level API rather than a password-manager change.
- Editorial inference: an analogous mobile evaluation is a direct next step; this study only covers desktop environments, and mobile autofill surfaces (keyboards, intents, app links) have different attack vectors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a security evaluation of thirteen password managers across the full lifecycle of password generation, storage, and autofill. The authors generated 147 million passwords through available interfaces, source-code modifications, and Selenium automation, then analyzed this corpus with Shannon entropy, chi-square tests, zxcvbn, and a Melicher-style LSTM guesser. They also manually examined local vault encryption and metadata handling, and replayed and extended prior autofill attacks including clickjacking, cross-origin iframe filling, and insecure form filling. The paper reports that generated passwords are generally strong at length 12 and above, but that some short generated passwords are vulnerable to online and offline guessing attacks, and it recommends length 10 for online resistance and length 18 for offline resistance. For storage and autofill, the authors find improvements over prior work but also remaining problems, especially in browser-based password managers, including unencrypted metadata, insecure defaults, and clickjacking-vulnerable autofill behavior.
Significance. If the results hold, this is the most comprehensive lifecycle-oriented evaluation of password managers to date and a timely update to five-year-old prior studies. The study has clear strengths: a very large generated-password corpus, published data and analysis scripts, replication of prior attack definitions, and explicit handling of external benchmarks rather than self-referential validation. The autofill and storage findings are concrete, and the paper's recommendation structure (which managers to avoid, which settings matter) is actionable. The main source of uncertainty is the password-generation guessability analysis, which currently rests on estimators that are not calibrated for random strings; that issue affects the paper's headline length recommendations and the more general claim that generated passwords are vulnerable to guessing attacks.
major comments (3)
- [Section 4.2, Table 5, Section 4.3] The generation-strength results and the safe-length recommendations of 10 and 18 characters are derived from guess counts produced by zxcvbn, but the authors' own /dev/random baseline contradicts the calibration of this tool for random strings. The baseline password 'MrKNxQNDAViS' is a 12-character mixed-case random password with a uniform keyspace of 52^12 ≈ 10^20.6, yet Table 5 reports only 10^9 guesses. A tool that underestimates a uniform-random baseline by roughly eleven orders of magnitude is not a sound basis for classifying manager-generated passwords as 'vulnerable to online and offline guessing attacks' or for setting length thresholds. Please reanalyze the corpus using a method appropriate for uniformly random strings, report how many passwords fall below the 10^6 and 10^14 thresholds under that method, and recompute the recommended lengths accordingly.
- [Section 4.2, Listing 1, Section 4.3] The neural-network guess estimates are obtained by training on 80% of each manager's own generated corpus and testing on the remaining 20% (Listing 1). This is a defensible approximation of a generator-aware adversary, but the paper does not state that this is the intended adversary model or compare it with a uniform-baseline model, so the reader cannot distinguish genuine generator bias from model artifacts. Moreover, the memory failure means the RNN results cover only lengths 8 and 12; the statements in Section 4.3 about length 20 and the 18-character offline threshold, as well as the interpolated 10-character online threshold, rest entirely on zxcvbn. Please supply RNN estimates for length 20, or explicitly restrict the generation-strength conclusions to the tested lengths.
- [Section 5, Table 6] The storage evaluation examines only local vault files; Section 5 states that 'the cloud databases are not available to us for direct evaluation.' Because most extension-based and browser-based password managers synchronize vaults through cloud backends by default, the contribution claim of evaluating 'password storage' for thirteen managers is broader than the evidence presented. The Section 6.6 web-vault checks (CSP header inspection and a CSRF test that changes a session timeout parameter) do not substitute for a security evaluation of cloud-side storage. Please either narrow the storage claims in the abstract and contribution list, or add direct evaluation of the cloud backends.
minor comments (6)
- [Section 4.2] The tool name 'zxcvbn' is misspelled as 'zxcbvn' in multiple places in Section 4.2; please correct this throughout.
- [Table 1 and Table 2] The symbols 'G' and '#' are used in the tables without a legend, making it impossible for the reader to determine which symbol denotes a supported feature; please add an explicit legend.
- [Figure 1] The captions for Figure 1 do not define the axes, the meaning of 'ld' and 'all', or the units of the log10 scale; please expand the caption so the plots are interpretable without referring to the body text.
- [Section 3.3] The sentence 'These password managers are all similar in high-level functionality' is too vague; please give a concrete list of the shared and distinguishing features of Chrome, Edge, Firefox, Internet Explorer, and Opera.
- [Section 4.2] The authors state that Safari could not be scripted for password generation and that only 100 passwords were manually analyzed; this limitation should be restated in the contributions where the generation corpus is described, since Safari is otherwise included in the thirteen-manager evaluation.
- [Section 5.2] The claim that 'all extension-based password managers leak the email address used to log in to the password manager' would be easier to verify if the table or text identified the specific file or field where the email address appears.
Circularity Check
No significant circularity: the paper is an empirical evaluation against external baselines and estimators; guessability-estimator concerns are validity issues, not definitional circularity.
full rationale
The paper makes no derivation claim that reduces to its own inputs. It is an empirical measurement study: it generates a corpus of passwords, compares observed character frequencies to uniform expectations using chi-square tests, and evaluates guessability with two external tools (zxcvbn and the Melicher et al. neural-network guesser) against externally stated thresholds of 10^6 and 10^14 guesses. The /dev/random generator and the online Secure Password Generator serve as independent baselines. The LSTM is trained on 80% of each generated corpus and tested on the remaining 20%, but this is an evaluation procedure for guessability rather than a parameter fitted to a target conclusion; the paper does not rename that fit as a prediction. The claims about storage and autofill replicate prior published attack definitions and are tested directly against the password managers. The few self-citations (e.g., references [24] and [27]) are background or recommendation citations and are not load-bearing for the central results. A methodological concern remains that zxcvbn and the LSTM are calibrated primarily for human-chosen passwords and that the LSTM is trained on generated outputs, which could affect the validity of the 10- and 18-character recommendations; however, that is a correctness or measurement-validity issue, not circularity, because the paper explicitly presents these tools as approximations and does not define its findings in terms of them. The evaluation is self-contained against external benchmarks, so a non-finding is appropriate.
Assumptions & free parameters
assumptions (5)
- domain assumption The attacker model permits network injection and XSS injection on websites visited by the user.
- domain assumption Guessing counts above 10^6 resist online attacks and above 10^14 resist offline attacks.
- domain assumption zxcvbn and the Melicher et al. RNN produce accurate guess estimates for randomly generated passwords.
- standard math The chi-square test and Shannon entropy are valid for detecting distributional anomalies in large samples.
- domain assumption The specific versions evaluated represent the security posture of each password manager at the time of the study.
Cite this review
Pith. "Pith review of That Was Then, This Is Now: A Security Evaluation of Password Generation, Storage, and Autofill in Thirteen Password Managers." pith.science (2026). https://pith.science/paper/LZ63PO2W
@misc{pith2026190803296,
author = {Pith},
title = {Pith review of: That Was Then, This Is Now: A Security Evaluation of Password Generation, Storage, and Autofill in Thirteen Password Managers},
year = {2026},
howpublished = {\url{https://pith.science/paper/LZ63PO2W}},
note = {Machine review of arXiv:1908.03296}
}
read the original abstract
Password managers have the potential to help users more effectively manage their passwords and address many of the concerns surrounding password-based authentication, however prior research has identified significant vulnerabilities in existing password managers. Since that time, five years has passed, leaving it unclear whether password managers remain vulnerable or whether they are now ready for broad adoption. To answer this question, we evaluate thirteen popular password managers and consider all three stages of the password manager lifecycle--password generation, storage, and autofill. Our evaluation is the first analysis of password generation in password managers, finding several non-random character distributions and identifying instances where generated passwords were vulnerable to online and offline guessing attacks. For password storage and autofill, we replicate past evaluations, demonstrating that while password managers have improved in the half-decade since those prior evaluations, there are still significant issues, particularly with browser-based password managers; these problems include unencrypted metadata, unsafe defaults, and vulnerabilities to clickjacking attacks. Based on our results, we identify password managers to avoid, provide recommendations on how to improve existing password managers, and identify areas of future research.
Figures
Reference graph
Works this paper leans on
-
[1]
S.M. Bellovin and M. Merritt. Encrypted key exchange: Password-based protocols secure against dictionary attacks. In Proceedings of the 1992 IEEE Symposium on Research in Security and Privacy, pages 72–84. IEEE, 1992
work page 1992
-
[2]
Kamouflage: Loss-resistant password management
Hristo Bojinov, Elie Bursztein, Xavier Boyen, and Dan Boneh. Kamouflage: Loss-resistant password management. In European symposium on research in computer security, pages 286–302. Springer, 2010
work page 2010
-
[3]
The science of guessing: analyzing an anonymized corpus of 70 million passwords
Joseph Bonneau. The science of guessing: analyzing an anonymized corpus of 70 million passwords. In 2012 IEEE Symposium on Security and Privacy, pages 538–552. IEEE, 2012
work page 2012
-
[4]
The quest to replace passwords: A framework for comparative evaluation of web authentication schemes
Joseph Bonneau, Cormac Herley, Paul C Van Oorschot, and Frank Stajano. The quest to replace passwords: A framework for comparative evaluation of web authentication schemes. In 2012 IEEE Symposium on Security and Privacy, pages 553–567. IEEE, 2012
work page 2012
-
[5]
Cristian Bravo-Lillo, Lorrie Cranor, Julie Downs, Saranga Komanduri, Stuart Schechter, and Manya Sleeper. Operating system framed in case of mistaken identity: measuring the success of web-based spoofing attacks on os password-entry dialogs. In Proceedings of the 2012 ACM conference on Computer and communications security, pages 365–377. ACM, 2012
work page 2012
-
[6]
Cracking-resistant password vaults using natural language encoders
Rahul Chatterjee, Joseph Bonneau, Ari Juels, and Thomas Ristenpart. Cracking-resistant password vaults using natural language encoders. In Security and Privacy (SP), 2015 IEEE Symposium on , pages 481–
work page 2015
-
[7]
A usability study and critique of two password managers
Sonia Chiasson, Paul C van Oorschot, and Robert Biddle. A usability study and critique of two password managers. In USENIX Security Symposium, volume 15, pages 1–16, 2006
work page 2006
-
[8]
A cognitive-behavioral framework of user password management lifecycle
Yee-Yin Choong. A cognitive-behavioral framework of user password management lifecycle. In International Conference on Human Aspects of Information Security, Privacy, and Trust, pages 127–137. Springer, 2014
work page 2014
Show all 40 references
-
[9]
Linux password storage
Chromium. Linux password storage. https: //chromium.googlesource.com/chromium/src/+/ master/docs/linux_password_storage.md, 2019. Accessed: 2019-05-20
2019
-
[10]
The best password managers of
CNET. The best password managers of
-
[11]
The tangled web of password reuse
Anupam Das, Joseph Bonneau, Matthew Caesar, Nikita Borisov, and XiaoFeng Wang. The tangled web of password reuse. In NDSS, volume 14, pages 23–26, 2014
2014
-
[12]
Password strength: An empirical analysis
Matteo Dell’Amico, Pietro Michiardi, and Yves Roudier. Password strength: An empirical analysis. In 2010 Proceedings IEEE INFOCOM, pages 1–9. IEEE, 2010
2010
-
[13]
Password managers: Under the hood of secrets management
Independent Security Evaluators. Password managers: Under the hood of secrets management. https://www.securityevaluators.com/ casestudies/password-manager-hacking/,
-
[14]
An investigation into users’ considerations towards using password managers
Michael Fagan, Yusuf Albayram, Mohammad Maifi Hasan Khan, and Ross Buck. An investigation into users’ considerations towards using password managers. Human-centric Computing and Information Sciences, 7(1):12, 2017
2017
-
[15]
Accessed: 2019-02-22
2019
-
[16]
An administrator’s guide to internet password research
Dinei Florêncio, Cormac Herley, and Paul C Van Oorschot. An administrator’s guide to internet password research. In 28th Large Installation System Administration Conference (LISA14) , pages 44–61, 2014
2014
-
[17]
A large-scale study of web password habits
Dinei Florencio and Cormac Herley. A large-scale study of web password habits. In Proceedings of the 16th international conference on World Wide Web, pages 657–666. ACM, 2007. 15
2007
-
[18]
So long, and no thanks for the externalities: the rational rejection of security advice by users
Cormac Herley. So long, and no thanks for the externalities: the rational rejection of security advice by users. In Proceedings of the 2009 workshop on New security paradigms workshop, pages 133–144. ACM, 2009
2009
-
[19]
On the security of password manager database formats
Paolo Gasti and Kasper B Rasmussen. On the security of password manager database formats. In European Symposium on Research in Computer Security, pages 770–787. Springer, 2012
2012
-
[20]
Better managed than memorized? studying the impact of managers on password strength and reuse
Sanam Ghorbani Lyastani, Michael Schilling, Sascha Fahl, Michael Backes, and Sven Bugiel. Better managed than memorized? studying the impact of managers on password strength and reuse. In 27th USENIX Security Symposium, pages 203–220, 2018
2018
-
[21]
The emperor’s new password manager: Security analysis of web-based password managers
Zhiwei Li, Warren He, Devdatta Akhawe, and Dawn Song. The emperor’s new password manager: Security analysis of web-based password managers. In USENIX Security Symposium, pages 465–479, 2014
2014
-
[22]
Fast, lean, and accurate: Modeling password guessability using neural networks
William Melicher, Blase Ur, Sean M Segreti, Saranga Komanduri, Lujo Bauer, Nicolas Christin, and Lorrie Faith Cranor. Fast, lean, and accurate: Modeling password guessability using neural networks. In 25th USENIX Security Symposium, pages 175–191, 2016
2016
-
[23]
The best password managers of
PC Magazine. The best password managers of
-
[24]
Accessed: 2019-02- 22
https://www.pcmag.com/roundup/300318/ the-best-password-managers . Accessed: 2019-02- 22
2019
-
[25]
Let’s go in for a closer look: Observing passwords in their natural habitat
Sarah Pearman, Jeremy Thomas, Pardis Emami Naeini, Hana Habib, Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor, Serge Egelman, and Alain Forget. Let’s go in for a closer look: Observing passwords in their natural habitat. In Proceedings of the 2017 ACM SIGSAC Conference on C...
2017
-
[26]
The autocomplete attribute and login fields
Mozilla. The autocomplete attribute and login fields. https://developer.mozilla.org/en- US/docs/Web/Security/Securing_your_site/ Turning_off_form_autocompletion#The_ autocomplete_attribute_and_login_fields,
-
[27]
Accessed: 2019-11-12
2019
-
[28]
Tls proxies: Friend or foe? In Proceedings of the 2016 Internet Measurement Conference , pages 551–557
Mark O’Neill, Scott Ruoti, Kent Seamons, and Daniel Zappala. Tls proxies: Friend or foe? In Proceedings of the 2016 Internet Measurement Conference , pages 551–557. ACM, 2016
2016
-
[29]
Password managers: Attacks and defenses
David Silver, Suman Jana, Dan Boneh, Eric Yawei Chen, and Collin Jackson. Password managers: Attacks and defenses. In USENIX Security Symposium, pages 449– 464, 2014
2014
-
[30]
Password security: What users know and what they actually do
Shannon Riley. Password security: What users know and what they actually do. Usability News, 8(1):2833– 2836, 2006
2006
-
[31]
End-to-end passwords
Scott Ruoti and Kent Seamons. End-to-end passwords. In Proceedings of the 2017 New Security Paradigms Workshop, pages 107–121. ACM, 2017
2017
-
[32]
Statistics: Cybersecurity data breaches on the rise
Security Scorecard. Statistics: Cybersecurity data breaches on the rise. https://securityscorecard. com/blog/cybersecurity-data-breaches- statistics-on-the-rise , 2018. Accessed: 2019-02-22
2018
-
[33]
How to end password reuse on the web
Ke Coby Wang and Michael K Reiter. How to end password reuse on the web. arXiv preprint arXiv:1805.00566, 2018
2018 arXiv
-
[34]
Password-manager friendly (pmf): Semantic annotations to improve the effectiveness of password managers
Frank Stajano, Max Spencer, Graeme Jenkinson, and Quentin Stafford-Fraser. Password-manager friendly (pmf): Semantic annotations to improve the effectiveness of password managers. In International Conference on Passwords, pages 61–73. Springer, 2014
2014
-
[35]
Protecting users against xss-based password manager abuse
Ben Stock and Martin Johns. Protecting users against xss-based password manager abuse. In Proceedings of the 9th ACM symposium on Information, computer and communications security, pages 183–194. ACM, 2014
2014
-
[36]
W3C. Html. https://www.w3.org/TR/ html52/sec-forms.html#element-attrdef- autocompleteelements-autocomplete, 2019. Accessed: 2019-11-09
2019
-
[38]
zxcvbn: Low-budget password strength estimation
Daniel Lowe Wheeler. zxcvbn: Low-budget password strength estimation. In 25th USENIX Security Symposium, pages 157–173, 2016
2016
-
[39]
Wu et al
T. Wu et al. The secure remote password protocol. In Internet Society Symposium on Network and Distributed System Security, 1998
1998
-
[40]
args": { 3
Shikun Aerin Zhang, Sarah Pearman, Lujo Bauer, and Nicolas Christin. Why people (don’t) use password managers effectively. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019), 2019. 16 A Additional Password Generation Data all l ld ls sd System p χ2 p χ2 p χ2 p ...
2019
-
[2019]
Accessed: 2019-02-22
https://www.cnet.com/news/the-best- password-managers-directory/. Accessed: 2019-02-22
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.