{"id":"c2810669-7255-419b-99ed-59c29382d230","arxiv_id":"1908.01489","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Developer experience proxies such as commit volume, API diversity, and project count show no clear association with correct use of Java Cryptography Architecture APIs across 2,324 open-source projects.","lead":"This study analyzed 2,324 open-source Java projects and used a static analysis tool to label cryptographic API uses as secure or buggy. It reports that common measures of developer experience, such as commit counts and number of projects, do not predict whether developers use Java cryptography correctly.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central null result is not directly tested: correlations use absolute counts rather than performance ratios, two of four factors are never given performance-based tests, and the largest group is excluded post hoc.","rationale":"The reader's verdict is CONDITIONAL, and my analysis supports keeping that verdict. The most load-bearing concern is not the CogniCrypt false-positive rate, although that is real, but a mismatch between the paper's central claim and its statistical evidence. The abstract claims no correlation between any of the four experience factors and developer performance, yet the only correlation matrix uses absolute secure and buggy commit counts. A factor can correlate strongly with both counts while having no correlation with the ratio of secure to total commits, which is the paper's own definition of performance. The group-based tests for #commits and #APIs actually find a significant difference, but the authors exclude the largest group post hoc because few commits make the ratio extreme; that exclusion is not pre-specified and removes most of the low-experience population. The other two factors are never given any ratio-based test. These are internal inconsistencies in the reported analysis, not disagreements with the field's consensus, and they directly undermine the headline claim. The concrete test would settle the issue by recomputing the correlations and group tests on the performance ratio, including one-commit developers. I agree with the reader that the CogniCrypt labeling and git-blame attribution are additional threats, but those would require external validation and do not account for the fact that even the paper's own numbers do not yet test the claimed null result. Therefore the conditional recommendation remains appropriate, with the requested revisions centered on the performance-ratio analysis.","tokens_in":8473,"tokens_out":3331,"duration_ms":33865,"concrete_test":"Obtain the released dataset (or require the authors to release it) and, for all developers including those with a single commit, compute the performance ratio p = secure/(secure+buggy). Then run Spearman correlations between each of the four experience factors (JCA commits, APIs used, projects, days) and p, and run Kruskal-Wallis/Wilcoxon tests on quartile groups for #projects and #days with pre-specified group definitions and no post hoc exclusion. If any factor shows a significant correlation or group difference, the abstract's 'none of the factors ... correlate' is unsupported; if none do, the central claim survives the corrected analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that none of the four experience factors correlate with developer performance. Section III.B defines performance as making 'more secure than buggy commits' and later as 'secure commits divided by total commits,' but the only correlation analysis (Table III) correlates the four factors with absolute numbers of secure and buggy commits, not with the performance ratio. Positive correlations with absolute counts (e.g., 0.74 between #JCA commits and #Buggy) are compatible with zero correlation with the ratio, so Table III cannot support the abstract's 'none ... correlate.' Ratio-based group tests are run for only two factors, #commits and #APIs. In both cases, the largest group (2–4 commits; 1–2 APIs, with heavy overlap) is significantly different from the others; the authors then discard this group post hoc because 'every commit has a much higher impact on performance' (Section III.B.1, Fig. 5). The remaining factors, #projects and #days, are never tested against the performance ratio at all, yet the abstract and conclusion state that none of the factors influence performance. The threat-to-validity section also acknowledges reliance on CogniCrypt without checking false positives on this dataset, but the more immediate and internal problem is that the measured quantity (absolute counts) does not match the claimed quantity (performance ratio).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper mines 2,324 open-source Java projects that use the Java Cryptography Architecture (JCA), analyzes them with the CogniCrypt static analyzer, and attributes each crypto API use to a developer via git blame. It reports that on average 2.5 of 3.9 crypto uses per project are insecure, and it investigates whether four developer-experience factors (number of JCA commits, number of distinct JCA APIs used, number of projects involved in, and number of days committing) correlate with developer performance, defined as the fraction of secure commits. The central claim is that none of these experience factors correlate with developer performance, leading the authors to call for qualitative studies and API-level investigations.","tokens_in":8735,"tokens_out":4519,"duration_ms":45495,"significance":"If the central null result were established, it would be an important and counterintuitive finding for usable security and for the design of crypto APIs and developer training. The paper also contributes a useful descriptive snapshot of JCA misuse in a large corpus of open-source Java projects, including per-API success rates. However, the significance is substantially weakened by the mismatch between the measured quantities and the claimed conclusion: the correlation analysis is performed on absolute counts, not on the performance ratio, and two of the four factors are never tested against the ratio at all. The post hoc exclusion of the largest group in the ratio-based tests further undermines the central claim.","major_comments":[{"comment":"The correlation analysis uses the absolute numbers of secure and buggy commits as dependent variables, not the performance ratio defined in Section III.B ('more secure than buggy commits' and later 'secure commits divided by total commits'). Table III shows, for example, that #JCA commits correlates 0.53 with #Secure and 0.74 with #Buggy, but a factor can correlate strongly with both absolute counts and have zero correlation with the ratio. Therefore Table III cannot support the abstract's claim that 'none of the factors ... correlate with developer performance'; it only shows co-movement with commit volume.","section":"III.B, Table III"},{"comment":"For the two factors that are tested against the performance ratio, the only statistically significant difference comes from the first and largest group (635 developers with 2-4 commits; 899 developers with 1-2 APIs), and these groups are then excluded post hoc because 'every commit has a much higher impact on performance.' This procedure turns a rejection of the null hypothesis into an acceptance by deleting the observations that contradict the null. Since the two excluded groups overlap heavily, as noted in Section III.B.2, the conclusion that 'the performance of developers ... is identical' is an artifact of post hoc selection rather than evidence.","section":"III.B.1, III.B.2"},{"comment":"The factors #projects and #days are never tested against the performance ratio. The paper reports only their correlations with absolute counts (Table III: 0.30/0.27 and 0.34/0.31 for #Project and #Days), which, as argued above, cannot establish the absence of a relationship with performance. The conclusion in Section VI that 'none of these factors influence developer performance' is therefore unsupported for two of the four factors by any direct statistical test.","section":"III.B, VI"},{"comment":"The grouping analysis silently drops developers with exactly one JCA commit (or fails to explain their exclusion): the groups are defined as 2-4, 5-8, and 9+ commits, yet the paper does not state what happened to one-commit developers or how many existed. Since for such developers the performance ratio is necessarily 0 or 1, their inclusion or exclusion can materially change the group comparisons and the reported null result. This needs to be clarified and justified.","section":"III.B.1, Fig. 4"},{"comment":"The secure/buggy labels are generated entirely by CogniCrypt, whose rule set was developed by two co-authors, and the authors state that 'we did not check for the existence of false positives in the results.' Because the central null result depends directly on these labels, a systematic correlation between tool errors and developer experience could turn the null into an artifact. At minimum, the paper should report a manual validation on a sample of the analyzed commits or provide independent evidence of precision on this dataset.","section":"III.C"}],"minor_comments":[{"comment":"The statement 'We analyzed a total of 2,324 projects, i.e., 2,141 plus the 183 initial projects' is confusing because the 183 initial projects appear to be part of the 2,780 projects downloaded via the GitHub API; the paper should clarify how the 2,324 total is obtained without double counting.","section":"II.A, III.A"},{"comment":"The text says the API groups range 'from 1 to 2, 3, or 3 to more APIs,' while Figure 6 labels the third group '4 to more'; the text and figure should be aligned.","section":"III.B.2"},{"comment":"There is a typo in the introduction: 'sigificantly' should be 'significantly.'","section":"I"},{"comment":"The abstract and conclusion refer to 'frequency of committed lines of code' as a factor, but Section III.B lists four factors: number of JCA commits, number of APIs, number of projects, and number of days. Please clarify whether 'frequency' is intended to refer to the #Days factor or to a separate rate-based measure.","section":"Abstract, VI"},{"comment":"The threat-to-validity discussion of git blame is honest, but it would be strengthened by a quantitative check; the current statement that 'we studied several cases' does not give the reader a sense of how often refactoring or maintenance commits were misattributed.","section":"III.C"}],"recommendation":"major_revision","confidential_remarks":"The descriptive results on API misuse rates are likely of interest to the ESEM community. The main risk is that the headline claim ('experience does not correlate with performance') is not supported by the analyses as presented; the authors should re-run the analysis on the performance ratio for all four factors, handle the one-commit developers explicitly, and avoid post hoc exclusion of the groups that contradict the null. If the data and scripts are made available, the revision would be much easier to evaluate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked about Hazhirpasand et al. on developer experience and Java crypto misuse. The paper is a decent empirical mining study with a real dataset, but its headline claim—that developer experience doesn't correlate with performance—is not actually tested the way it's described. The abstract says none of the factors correlate, but Table III correlates the four experience factors with absolute counts of secure and buggy commits, not with the per-developer performance ratio. Positive correlations with absolute counts (e.g., 0.74 between JCA commits and buggy commits) are perfectly compatible with zero correlation with the ratio. So the correlation matrix doesn't support the abstract's claim.\n\nWhat the paper does well: it gives a clean descriptive picture of crypto misuse in 2,324 Maven projects using JCA. The numbers—on average 2.5 insecure uses per project, and particular APIs like PBEKeySpec and Cipher with very low correct usage rates—are consistent with earlier studies and add another data point. The dataset collection through contributor snowballing is explained. The writing is honest about limitations, including not checking CogniCrypt false positives on this dataset.\n\nThe soft spots: only two of the four factors (#commits, #APIs) receive a performance-ratio group test. #projects and #days are never tested against the performance ratio at all, yet the conclusion says none of the factors influence performance. And in the two group tests, the group with the fewest commits/APIs is significantly different from the others; the authors exclude it post hoc because 'every commit has a much higher impact' at low counts. That's a real decision, but it's exactly the group where experience differences would be most visible, so excluding it and then claiming no effect is not convincing. The CogniCrypt labeling issue is a secondary concern; it's a co-authored tool and they didn't validate false positives here, but they acknowledge it, so it's a limitation more than a hidden flaw.\n\nOverall, the paper is worth a serious referee, but the central null result needs to be re-analyzed: report correlations with the performance ratio for all four factors, pre-specify group definitions, and release the project list and scripts if possible. As it stands, I'd treat the descriptive statistics as useful and the null result as unproven. I wouldn't cite the main claim in its current form, but I'd probably bring it to a reading group to discuss the mismatch between measurement and claim.","headline":"Useful descriptive study of crypto misuse, but the central null result on developer experience is not actually tested as claimed: Table III correlates absolute counts, not performance ratios, and the group tests exclude the only group that shows a difference.","tokens_in":9249,"tokens_out":2493,"would_cite":false,"duration_ms":23929,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"More crypto experience does not make Java code safer","keywords":["Java Cryptography Architecture","developer experience","crypto API misuse","secure coding","static analysis","GitHub mining","empirical software engineering","correlation analysis"],"falsifier":"Manually inspect a random sample of hundreds of crypto commits and compare the manual label (secure vs. buggy) with the tool's label, then rerun the correlation on corrected labels; if the tool's errors favor experienced developers, the null finding disappears.","tokens_in":8173,"feed_emoji":"🔐","tokens_out":6431,"duration_ms":64826,"temperature":0.7,"pith_summary":"This paper asks whether developers who are more experienced with the Java Cryptography Architecture (JCA) produce fewer insecure uses of its APIs. Mining 2,324 open-source Java projects, the authors label every crypto API use as secure or buggy using a rules-based static analyzer, then compare four experience measures—number of JCA commits, diversity of JCA APIs used, number of projects contributed to, and days of crypto activity—against each developer's ratio of secure to total crypto commits. They find that none of these experience factors correlates with performance: developers who commit more crypto code, use more crypto APIs, or work across more projects are no more likely to write secure code. They also report that on average only 1.4 of every 3.9 crypto uses per project is secure, with five APIs correct less than 6.58% of the time. The result matters because it challenges the common assumption that hands-on crypto experience makes developers safer, and it redirects attention toward API design and tooling.","feed_headline":"More crypto experience does not make Java code safer","feed_subtitle":"A 2,324-project analysis finds no link between experience and secure use of Java cryptography APIs.","key_machinery":"The central measure is the performance ratio, a developer's secure crypto commits divided by their total crypto commits; it cancels out the sheer volume of activity, which is what makes the absence of correlation meaningful. The argument runs through three statistical tests: Spearman rank correlation (because the data are not normally distributed), the Kruskal-Wallis test to compare developer groups, and pairwise Wilcoxon signed-rank tests with Bonferroni adjustment as post hoc comparisons. On the data side, a static analyzer implementing CrySL rules labels each JCA API use as secure or buggy, and git blame attributes each commit to the last developer who touched the line. The ratio plus the group comparisons are what allow the authors to separate 'does experience change counts' from 'does experience change the rate of mistakes.'","core_discovery":"The paper's central claim is a null result: in using Java cryptography APIs, developer experience does not correlate with developer performance. Performance is defined as the fraction of a developer's crypto commits that are secure; the four experience proxies are the number of JCA-related committed lines, the number of distinct JCA APIs used, the number of projects a developer touched, and the number of days on which crypto commits were made. While the number of crypto commits and the number of APIs used correlate with the absolute counts of secure and buggy commits, neither correlates with the secure fraction once developers with very few commits (where a single commit swings the ratio) are set aside. Developers with 5–8 and 9+ crypto commits perform the same, and developers who use different numbers of APIs perform the same. The paper therefore concludes that none of these measures of experience influences whether JCA code is secure.","pith_inferences":["A testable extension would be to compare the same developers' general Java bug rate with their crypto bug rate; if only the crypto rate fails to improve with experience, the problem is specific to cryptographic APIs rather than to developer skill.","Because projects average only 3.9 crypto uses, large parts of the sample have almost no statistical power; re-running the analysis on projects with high crypto usage (say, 20+ crypto commits) could expose an experience effect the aggregate hides.","The git blame attribution's reliance on the last committer means refactors or codemods could be misattributed; a replication using full commit history or per-line authoring detection would test whether this artifact is hiding a real correlation.","The paper's own call for qualitative studies suggests a concrete next step: interview developers whose JCA code is always secure about their training, documentation use, and tools, and test whether those behaviors, rather than raw experience, predict performance."],"forward_implications":["If the result holds, hiring or task assignment based on crypto commit counts, API breadth, or project count will not reliably identify developers who write secure JCA code.","Secure-by-default API design, example-driven documentation, and automated misuse checks become the main levers, because accumulating experience does not by itself raise the security rate.","The five APIs with less than 6.58% correct usage (e.g., SecretKeySpec, IvParameterSpec, Cipher, Signature, PBEParameterSpec) are where API-level redesign or tooling would have the largest effect.","Researchers can stop treating 'amount of crypto activity' as a proxy for crypto skill and instead investigate which specific practices distinguish the 27.41% of developers who were always secure from the 42% who were always buggy."],"supporting_citations":[{"why":"Supplies the CrySL rule set that defines secure versus misused crypto API calls and provided the initial 183 JCA projects.","marker":"[2]"},{"why":"Provides the Spearman rank correlation method used to test relationships between experience factors and secure/buggy commit counts.","marker":"[9]"},{"why":"Describes the CogniCrypt static analyzer (and its secure templates) whose command-line version produced the misuse labels for each API use.","marker":"[12]"}],"fun_headline_variants":["Java crypto skill not tied to safer code","Experience fails to predict crypto API security","More Java crypto use doesn't mean fewer bugs","No link found between dev experience and crypto security","Experienced Java devs aren't safer with crypto"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated checker's classification of each crypto use as secure or insecure is correct, and that the last person to edit each line is the one who wrote it; if either assumption fails in a way that tracks experience, the null result collapses.","fun_headline_variants_meta":{"raw":{"variants":["Java crypto skill not tied to safer code","Experience fails to predict crypto API security","More Java crypto use doesn't mean fewer bugs","No link found between dev experience and crypto security","Experienced Java devs aren't safer with crypto"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000573,"raw_usage":{"total_tokens":2674,"prompt_tokens":878,"completion_tokens":1796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":1727}},"tokens_in":494,"tokens_out":1796,"duration_ms":13237,"temperature":1.0,"reasoning_tokens":1727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:10:55.919441+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of hundreds of crypto commits and compare the manual label (secure vs. buggy) with the tool's label, then rerun the correlation on corrected labels; if the tool's errors favor experienced developers, the null finding disappears.","supporting_citations":[{"cited_title":"CrySL: An extensible approach to validating the correct usage of cryptographic APIs,","cited_arxiv_id":null,"evidence_quote":"Supplies the CrySL rule set that defines secure versus misused crypto API calls and provided the initial 183 JCA projects."},{"cited_title":"Spearman rank correlation,","cited_arxiv_id":null,"evidence_quote":"Provides the Spearman rank correlation method used to test relationships between experience factors and secure/buggy commit counts."}],"review_version":1}