Pith. sign in

REVIEW 5 major objections 5 minor 12 references

The Impact of Developer Experience in Using Java Cryptography

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read More crypto experience does not make Java code safer

desk verdict Useful descriptive study of crypto misuse, but the central null result on developer experience is not actually tested as claimed: Table III correlates absolute counts, not performance ratios, and the group tests exclude the only group that shows a difference. read the letter →

arxiv 1908.01489 v1 pith:GEGGZPNF submitted 2019-08-05 cs.CR cs.SE

Mohammadreza Hazhirpasand , Mohammad Ghafari , Stefan Krüger , Eric Bodden , Oscar Nierstrasz This is my paper · ORCID
classification cs.CRcs.SE
keywords JavaCryptographyArchitecturedeveloperexperiencecryptoAPImisusesecurecodingstaticanalysisGitHubminingempiricalsoftwareengineeringcorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether developers who are more experienced with the Java Cryptography Architecture (JCA) produce fewer insecure uses of its APIs. Mining 2,324 open-source Java projects, the authors label every crypto API use as secure or buggy using a rules-based static analyzer, then compare four experience measures—number of JCA commits, diversity of JCA APIs used, number of projects contributed to, and days of crypto activity—against each developer's ratio of secure to total crypto commits. They find that none of these experience factors correlates with performance: developers who commit more crypto code, use more crypto APIs, or work across more projects are no more likely to write secure code. They also report that on average only 1.4 of every 3.9 crypto uses per project is secure, with five APIs correct less than 6.58% of the time. The result matters because it challenges the common assumption that hands-on crypto experience makes developers safer, and it redirects attention toward API design and tooling.

What carries the argument

The central measure is the performance ratio, a developer's secure crypto commits divided by their total crypto commits; it cancels out the sheer volume of activity, which is what makes the absence of correlation meaningful. The argument runs through three statistical tests: Spearman rank correlation (because the data are not normally distributed), the Kruskal-Wallis test to compare developer groups, and pairwise Wilcoxon signed-rank tests with Bonferroni adjustment as post hoc comparisons. On the data side, a static analyzer implementing CrySL rules labels each JCA API use as secure or buggy, and git blame attributes each commit to the last developer who touched the line. The ratio plus the group comparisons are what allow the authors to separate 'does experience change counts' from 'does experience change the rate of mistakes.'

What would settle it

Manually inspect a random sample of hundreds of crypto commits and compare the manual label (secure vs. buggy) with the tool's label, then rerun the correlation on corrected labels; if the tool's errors favor experienced developers, the null finding disappears.

Watch

Extended reading notes

Core claim

The paper's central claim is a null result: in using Java cryptography APIs, developer experience does not correlate with developer performance. Performance is defined as the fraction of a developer's crypto commits that are secure; the four experience proxies are the number of JCA-related committed lines, the number of distinct JCA APIs used, the number of projects a developer touched, and the number of days on which crypto commits were made. While the number of crypto commits and the number of APIs used correlate with the absolute counts of secure and buggy commits, neither correlates with the secure fraction once developers with very few commits (where a single commit swings the ratio) are set aside. Developers with 5–8 and 9+ crypto commits perform the same, and developers who use different numbers of APIs perform the same. The paper therefore concludes that none of these measures of experience influences whether JCA code is secure.

Load-bearing premise

The load-bearing premise is that the automated checker's classification of each crypto use as secure or insecure is correct, and that the last person to edit each line is the one who wrote it; if either assumption fails in a way that tracks experience, the null result collapses.

Editorial extensions

If this is right

  • If the result holds, hiring or task assignment based on crypto commit counts, API breadth, or project count will not reliably identify developers who write secure JCA code.
  • Secure-by-default API design, example-driven documentation, and automated misuse checks become the main levers, because accumulating experience does not by itself raise the security rate.
  • The five APIs with less than 6.58% correct usage (e.g., SecretKeySpec, IvParameterSpec, Cipher, Signature, PBEParameterSpec) are where API-level redesign or tooling would have the largest effect.
  • Researchers can stop treating 'amount of crypto activity' as a proxy for crypto skill and instead investigate which specific practices distinguish the 27.41% of developers who were always secure from the 42% who were always buggy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to compare the same developers' general Java bug rate with their crypto bug rate; if only the crypto rate fails to improve with experience, the problem is specific to cryptographic APIs rather than to developer skill.
  • Because projects average only 3.9 crypto uses, large parts of the sample have almost no statistical power; re-running the analysis on projects with high crypto usage (say, 20+ crypto commits) could expose an experience effect the aggregate hides.
  • The git blame attribution's reliance on the last committer means refactors or codemods could be misattributed; a replication using full commit history or per-line authoring detection would test whether this artifact is hiding a real correlation.
  • The paper's own call for qualitative studies suggests a concrete next step: interview developers whose JCA code is always secure about their training, documentation use, and tools, and test whether those behaviors, rather than raw experience, predict performance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper mines 2,324 open-source Java projects that use the Java Cryptography Architecture (JCA), analyzes them with the CogniCrypt static analyzer, and attributes each crypto API use to a developer via git blame. It reports that on average 2.5 of 3.9 crypto uses per project are insecure, and it investigates whether four developer-experience factors (number of JCA commits, number of distinct JCA APIs used, number of projects involved in, and number of days committing) correlate with developer performance, defined as the fraction of secure commits. The central claim is that none of these experience factors correlate with developer performance, leading the authors to call for qualitative studies and API-level investigations.

Significance. If the central null result were established, it would be an important and counterintuitive finding for usable security and for the design of crypto APIs and developer training. The paper also contributes a useful descriptive snapshot of JCA misuse in a large corpus of open-source Java projects, including per-API success rates. However, the significance is substantially weakened by the mismatch between the measured quantities and the claimed conclusion: the correlation analysis is performed on absolute counts, not on the performance ratio, and two of the four factors are never tested against the ratio at all. The post hoc exclusion of the largest group in the ratio-based tests further undermines the central claim.

major comments (5)
  1. [III.B, Table III] The correlation analysis uses the absolute numbers of secure and buggy commits as dependent variables, not the performance ratio defined in Section III.B ('more secure than buggy commits' and later 'secure commits divided by total commits'). Table III shows, for example, that #JCA commits correlates 0.53 with #Secure and 0.74 with #Buggy, but a factor can correlate strongly with both absolute counts and have zero correlation with the ratio. Therefore Table III cannot support the abstract's claim that 'none of the factors ... correlate with developer performance'; it only shows co-movement with commit volume.
  2. [III.B.1, III.B.2] For the two factors that are tested against the performance ratio, the only statistically significant difference comes from the first and largest group (635 developers with 2-4 commits; 899 developers with 1-2 APIs), and these groups are then excluded post hoc because 'every commit has a much higher impact on performance.' This procedure turns a rejection of the null hypothesis into an acceptance by deleting the observations that contradict the null. Since the two excluded groups overlap heavily, as noted in Section III.B.2, the conclusion that 'the performance of developers ... is identical' is an artifact of post hoc selection rather than evidence.
  3. [III.B, VI] The factors #projects and #days are never tested against the performance ratio. The paper reports only their correlations with absolute counts (Table III: 0.30/0.27 and 0.34/0.31 for #Project and #Days), which, as argued above, cannot establish the absence of a relationship with performance. The conclusion in Section VI that 'none of these factors influence developer performance' is therefore unsupported for two of the four factors by any direct statistical test.
  4. [III.B.1, Fig. 4] The grouping analysis silently drops developers with exactly one JCA commit (or fails to explain their exclusion): the groups are defined as 2-4, 5-8, and 9+ commits, yet the paper does not state what happened to one-commit developers or how many existed. Since for such developers the performance ratio is necessarily 0 or 1, their inclusion or exclusion can materially change the group comparisons and the reported null result. This needs to be clarified and justified.
  5. [III.C] The secure/buggy labels are generated entirely by CogniCrypt, whose rule set was developed by two co-authors, and the authors state that 'we did not check for the existence of false positives in the results.' Because the central null result depends directly on these labels, a systematic correlation between tool errors and developer experience could turn the null into an artifact. At minimum, the paper should report a manual validation on a sample of the analyzed commits or provide independent evidence of precision on this dataset.
minor comments (5)
  1. [II.A, III.A] The statement 'We analyzed a total of 2,324 projects, i.e., 2,141 plus the 183 initial projects' is confusing because the 183 initial projects appear to be part of the 2,780 projects downloaded via the GitHub API; the paper should clarify how the 2,324 total is obtained without double counting.
  2. [III.B.2] The text says the API groups range 'from 1 to 2, 3, or 3 to more APIs,' while Figure 6 labels the third group '4 to more'; the text and figure should be aligned.
  3. [I] There is a typo in the introduction: 'sigificantly' should be 'significantly.'
  4. [Abstract, VI] The abstract and conclusion refer to 'frequency of committed lines of code' as a factor, but Section III.B lists four factors: number of JCA commits, number of APIs, number of projects, and number of days. Please clarify whether 'frequency' is intended to refer to the #Days factor or to a separate rate-based measure.
  5. [III.C] The threat-to-validity discussion of git blame is honest, but it would be strengthened by a quantitative check; the current statement that 'we studied several cases' does not give the reader a sense of how often refactoring or maintenance commits were misattributed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the null result is empirical, and the CogniCrypt dependency is a validity threat rather than a circular reduction.

full rationale

The paper's conclusions are empirical measurements rather than derived identities. No parameter is fitted to a subset of data and then renamed as a prediction, and no equation-level reduction equates an output with an input. The only close call is the reliance on CogniCrypt for secure/buggy labels; Section III.C states 'we did not check for the existence of false positives in the results as the authors have done so and found the tool to be fairly precise.' That appeal leans on a prior evaluation by tool authors, two of whom are co-authors of this paper, but it is a measurement dependency, not a circular derivation: the four experience factors are collected independently of the tool, and the correlation analysis would still be a meaningful empirical study conditional on the tool's labels. A separate concern, not circularity, is that Table III reports Spearman correlations between the factors and absolute secure/buggy counts, while the abstract claims no correlation with the performance ratio; ratio-based tests are conducted only for commits and API diversity, and the largest group is excluded post hoc. These are correctness risks, not evidence that the result is true by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central null result depends on the correctness of the labeling tool, the git attribution, and the experience proxies. None are independently validated in the paper, and CogniCrypt was authored by two of the co-authors.

free parameters (3)
  • Commit group cutoffs = 2-4, 5-8, 9 or more commits
    Chosen from quartiles of a boxplot after data collection and used in Kruskal-Wallis group comparisons. The first group is later excluded as an artifact.
  • API group cutoffs = 1-2, 3, 4 or more APIs
    Also chosen from boxplot quartiles and used in group comparisons. The text and Figure 6 disagree on the third label.
  • Analysis timeout = 15 minutes per project
    Hand-chosen threshold; roughly 15% of analyses were aborted, which could bias the sample toward projects that analyze quickly.
assumptions (4)
  • domain assumption CogniCrypt's rule set correctly labels crypto API uses as secure or buggy, with false positives below 10% as reported by the tool's authors.
    Invoked in Section II.A.3 and not validated on this dataset. Section III.C says "we did not check for the existence of false positives".
  • domain assumption The last committer identified by git blame is the developer responsible for each crypto API use.
    Invoked in Section II.A.4. The threats section acknowledges that the last commit could be refactoring or maintenance.
  • domain assumption Commit count, API diversity, project count, and active days are valid proxies for developer experience.
    Stated in Section III.B. No validation is provided that these measures capture experience, and no qualitative check with developers was done.
  • domain assumption The snowball-sampled Maven projects are representative enough to support general conclusions about JCA developers.
    Section III.C acknowledges selection limitations: only Maven projects, no Gradle or Ant, and snowballing from 183 initial projects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impact of Developer Experience in Using Java Cryptography." pith.science (2026). https://pith.science/paper/GEGGZPNF

@misc{pith2026190801489,
  author       = {Pith},
  title        = {Pith review of: The Impact of Developer Experience in Using Java Cryptography},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GEGGZPNF}},
  note         = {Machine review of arXiv:1908.01489}
}
read the original abstract

Previous research has shown that crypto APIs are hard for developers to understand and difficult for them to use. They consequently rely on unvalidated boilerplate code from online resources where security vulnerabilities are common. We analyzed 2,324 open-source Java projects that rely on Java Cryptography Architecture (JCA) to understand how crypto APIs are used in practice, and what factors account for the performance of developers in using these APIs. We found that, in general, the experience of developers in using JCA does not correlate with their performance. In particular, none of the factors such as the number or frequency of committed lines of code, the number of JCA APIs developers use, or the number of projects they are involved in correlate with developer performance in this domain. We call for qualitative studies to shed light on the reasons underlying the success of developers who are expert in using cryptography. Also, detailed investigation at API level is necessary to further clarify a developer obstacles in this domain.

Figures

Figures reproduced from arXiv: 1908.01489 by the authors.

Figure 1
Figure 1. The number of secure, total number of commits, and number of JCA [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. The secure versus total number of each API use in percentage [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 5
Figure 5. Performance vs. number of JCA commits by each developer. Therefore, we checked whether the perfor￾mance of developers in the different groups is the same. We used a rank-based nonparametric Kruskal-Wallis test since the three groups did not have a normal distribution and this violates one of the assumptions of the one-way ANOVA test. The test resulted a p-value of 0.001315 (chi￾square 13.269, and df 2), which indica… view at source ↗
Figures from the paper (1 more)
Figure 7
Figure 7. Figure 7: Performance vs. number of APIs leading to the difference of the first group with others. If we account for this phenomenon, we cannot reject the null hypothesis, and conclude that the performance of developers who use different APIs is identical [PITH_FULL_IMAGE:figur…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 9 canonical work pages

  1. [1]

    An empirical study of cryptographic misuse in Android applications,

    M. Egele, D. Brumley, Y . Fratantonio, and C. Kruegel, “An empirical study of cryptographic misuse in Android applications,” in Proceedings of the 2013 ACM SIGSAC Conference on Computer Communications Security, ser. CCS ’13. New York, NY , USA: ACM, 2013, pp. 73–84. [Online]. Available: http://doi.acm.org/10.1145/2508859.2516693

    arXiv 2013

  2. [2]

    CrySL: An extensible approach to validating the correct usage of cryptographic APIs,

    S. Krüger, J. Späth, K. Ali, E. Bodden, and M. Mezini, “CrySL: An extensible approach to validating the correct usage of cryptographic APIs,” in 32nd European Conference on Object-Oriented Programming, ECOOP 2018, July 16-21, 2018, Amsterdam, The Netherlands, 2018, pp. 10:1–10:27

    work page 2018

  3. [3]

    CHIRON: Deployment-quality detection of Java cryptographic vulner- abilities,

    S. Rahaman, Y . Xiao, K. Tian, F. Shaon, M. Kantarcioglu, and D. Yao, “CHIRON: Deployment-quality detection of Java cryptographic vulner- abilities,” arXiv preprint arXiv:1806.06881 , 2018

    arXiv 2018

  4. [4]

    Modelling analysis and auto-detection of cryptographic misuse in Android applications,

    S. Shao, G. Dong, T. Guo, T. Yang, and C. Shi, “Modelling analysis and auto-detection of cryptographic misuse in Android applications,” 2014, pp. 75–80

    work page 2014

  5. [5]

    Evaluation of cryptography usage in Android applications,

    A. Chatzikonstantinou, C. Ntantogian, G. Karopoulos, and C. Xenakis, “Evaluation of cryptography usage in Android applications,” in Inter- national Conference on Bio-inspired Information and Communications Technologies, 2016, pp. 83–90

    work page 2016

  6. [6]

    Why does crypto- graphic software fail?: a case study and open problems,

    D. Lazar, H. Chen, X. Wang, and N. Zeldovich, “Why does crypto- graphic software fail?: a case study and open problems,” 2014, pp. 7:1– 7:7

    work page 2014

  7. [7]

    ‘Jumping through hoops’: Why do Java developers struggle with cryptography APIs?

    S. Nadi, S. Krüger, M. Mezini, and E. Bodden, “‘Jumping through hoops’: Why do Java developers struggle with cryptography APIs?” in 2016 IEEE/ACM 38th International Conference on Software Engineer- ing (ICSE), May 2016, pp. 935–946

    work page 2016

  8. [8]

    Comparing the usability of cryptographic APIs,

    Y . Acar, M. Backes, S. Fahl, S. Garfinkel, D. Kim, M. L. Mazurek, and C. Stransky, “Comparing the usability of cryptographic APIs,” in 2017 IEEE Symposium on Security and Privacy (SP) , May 2017, pp. 154–171

    work page 2017

Show all 12 references
  1. [9]

    Spearman rank correlation,

    J. H. Zar, “Spearman rank correlation,” Encyclopedia of Biostatistics , vol. 7, 2005

    2005

  2. [10]

    You get where you’re looking for: The impact of information sources on code security,

    Y . Acar, M. Backes, S. Fahl, D. Kim, M. L. Mazurek, and C. Stransky, “You get where you’re looking for: The impact of information sources on code security,” in 2016 IEEE Symposium on Security and Privacy (SP). IEEE, 2016, pp. 289–305

    2016

  3. [11]

    Developers deserve security warnings, too: On the effect of integrated security advice on cryptographic API misuse,

    P. L. Gorski, L. L. Iacono, D. Wermke, C. Stransky, S. Möller, Y . Acar, and S. Fahl, “Developers deserve security warnings, too: On the effect of integrated security advice on cryptographic API misuse,” in Fourteenth Symposium on Usable Privacy and Security ( {SOUPS} 2018), 2...

    2018

  4. [12]

    CogniCrypt: Supporting developers in using cryptography,

    S. Krüger, S. Nadi, M. Reif, K. Ali, M. Mezini, E. Bodden, F. Göpfert, F. Günther, C. Weinert, D. Demmler, and R. Kamath, “CogniCrypt: Supporting developers in using cryptography,” in Proceedings of the 32Nd IEEE/ACM International Conference on Automated Software Engineering ,...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.