{"id":"2fd7f583-67f8-4898-96bf-60edfdbe7e70","arxiv_id":"2507.09580","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 315-task benchmark shows top LLMs surpass human experts on cryptography knowledge questions, approach them on proofs, but lag 25 to 30 points behind on capture-the-flag exploitation.","lead":"This paper introduces AICrypto, a 315-task benchmark testing how well large language models handle cryptography, from multiple-choice questions to capture-the-flag attacks and formal proofs. It finds that top models now beat human experts on factual crypto questions and approach them on proofs, but still trail experts on practical exploitation tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proof scores may be systematically inflated: 0.90 correlation with human grades does not establish calibration, and the paper's own failure analysis shows plausible-but-flawed proofs that LLM graders could over-credit.","rationale":"The reader's conditional verdict is well founded. I agree that proof contamination (§2.4) is an untested assumption, but I see a more immediate load-bearing dependency: the proof scores themselves are produced by LLM graders and validated only via correlation. A uniform level shift is invisible to Pearson/Spearman correlation and would directly change the reported proof percentages. The 306 human-scored samples make the calibration check feasible. The MCQ and CTF results rest on exact-match metrics, and the CTF official-solution comparison is a reasonable contamination probe, so those pillars are less threatened. I therefore do not move the verdict: it should remain conditional on the authors providing calibration evidence (and ideally a contamination probe for proofs), but the benchmark's core design and contributions stand.","tokens_in":36435,"tokens_out":5556,"duration_ms":73792,"concrete_test":"Using the 306 human-graded samples already collected (§3.4), compute the mean signed difference (average of six LLM-grader scores minus human score) and 95% limits of agreement, both overall and per grader model. If the mean bias exceeds 0.25 points on the 5-point rubric (i.e., 5 percentage points) or the LoA half-width exceeds 1 point, report bias-corrected proof rates for all 17 models and re-test whether gemini-2.5-pro's 85.4% remains close to the human 94.0%.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 assigns proof scores using two LLM grader models (gpt-5.1 and gemini-3-pro-preview), averaging six scores per answer, and validates the pipeline by reporting Pearson 0.9025 and Spearman 0.8973 against human scores on 306 samples. Correlation is a monotonic agreement measure: it can be 0.9 even if every LLM grade is shifted up by a constant, which is exactly the error mode that would inflate gemini-2.5-pro's 85.4% toward the human 94.0% in Fig. 8. The paper's own §4.3 documents that LLM proofs 'appear correct at first glance but reveal critical flaws upon closer examination'; an LLM grader reading the submitted proof against the reference solution has no mechanism to prefer the human-detected flaw over surface-level correctness. Because the human scores for those 306 samples already exist, the missing check is cheap: report the mean signed difference and limits of agreement, not just correlation. Without that calibration, the 'approach human performance on proof problems' pillar of the central claim is not yet established, independent of the separate contamination question for §2.4.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AICrypto, a cryptography benchmark consisting of 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, all curated or authored by cryptography experts. The authors evaluate 17 LLMs using task-specific protocols—single-turn MCQs, an agentic framework for CTF, and best-of-three automated proof grading —and compare against human expert baselines. The central empirical finding is that top LLMs exceed human experts on MCQs (o3 at 97.8% vs. best human 94.1%), approach human performance on proof problems (gemini-2.5-pro 85.4% vs. human 94.0%), and substantially trail human experts on CTF challenges (gemini-2.5-pro 56.0% vs. human 81.2%). The paper also includes contamination analyses for MCQs and CTF, automated proof grading validated by correlation with human scores, and failure case analyses.","tokens_in":36630,"tokens_out":3809,"duration_ms":48288,"significance":"If the reported results hold, AICrypto would be a useful, multi-faceted resource for evaluating LLMs in cryptography, going beyond prior CTF-only benchmarks by adding conceptual and formal-reasoning components. The authors provide explicit expert effort, public code and dataset, human baselines, and a reproducible agent framework; they also ship a substantial contamination analysis for two of the three task types. The distinction between memorized cryptographic knowledge and end-to-end exploitation, as evidenced by the MCQ/CTF gap, is a valuable observation for the community. However, the proof-grading calibration and the contamination status of the proof problems are load-bearing for the paper's claim that LLMs 'approach human performance' on proofs, and the abstract/conclusion overstates the CTF results relative to the paper's own Figure 7.","major_comments":[{"comment":"The abstract claims that state-of-the-art models 'match or even surpass human experts in memorizing cryptographic concepts, exploiting common vulnerabilities, and routine proofs.' The CTF component does not support this. Figure 7 shows gemini-2.5-pro at 56.0% overall success rate versus human 81.2%, a 25-point gap, and even the best model on RSA (gemini-2.5-pro at 78.8%) trails the human baseline of 97.6%. The conclusion's phrase 'basic vulnerability exploitation' is similarly unsupported as stated. Please revise these claims to distinguish recognized textbook attacks (where LLMs do well) from realistic exploitation (where they clearly lag).","section":"Abstract, §4.1, §7, Figure 7"},{"comment":"The automated proof grader is validated only by Pearson (0.9025) and Spearman (0.8973) correlations on 306 samples. Correlation is invariant to additive shifts, so a systematic leniency bias would be invisible to this validation. The headline comparison is gemini-2.5-pro at 85.4% versus human 94.0%; an average grader bias of +8.6 percentage points would erase the gap. Given the paper's own §4.3 finding that LLM proofs 'appear correct at first glance but reveal critical flaws upon closer examination,' the grader has no demonstrated mechanism to avoid over-crediting such proofs. Please report the mean signed difference, standard deviation, and limits of agreement (e.g., Bland-Altman analysis), and ideally per-score-band calibration, between LLM and human scores.","section":"§3.4, Appendix G"},{"comment":"For MCQs (Table 4) and CTF challenges (Table 5) the paper provides contamination probes, but no analogous analysis is reported for the 30 proof problems. Section 2.4 asserts these problems were 'never publicly released online,' yet course materials routinely leak through student uploads, solution repositories, and course websites, and the problems are close enough to human performance that even a few leaked items could change the conclusion. Please run a retrieval-based or n-gram-based contamination probe for the proof problems, or provide explicit evidence that no public trace exists.","section":"§2.4, Appendix E"},{"comment":"The human proof baseline is described as 'select the top five scores from each source' from real assignments/exams. This is not a like-for-like comparison with the LLM protocol, which uses three fresh attempts with a best-of-three score and an open-book, no-time-limit setting. Selecting the top historical scores may systematically inflate the human baseline, while not controlling for differences in problem exposure or grading conditions. Please describe the human baseline more precisely (number of distinct humans, grading rubric applied, whether the same 30 problems were used, and whether the comparison controls for attempt count) or re-measure the baseline under the LLM protocol.","section":"Appendix F.1 (Proof problems human baseline)"}],"minor_comments":[{"comment":"The notation 'ˆ indicates a parallel rule' is confusing; the same symbol is used as 'or' in the grader prompt. Please use a consistent label (e.g., 'parallel rule' or 'OR') in both places.","section":"Figure 6 and Appendix G"},{"comment":"The human CTF baseline is computed on a subset of 100 of the 150 challenges; please state whether the subset is representative across the five categories and whether scoreboard top-10 ranking is available for all categories, since Figure 7 reports category-level human rates.","section":"Appendix F.1 (CTF human baseline)"},{"comment":"The model names in the heatmap x-axis are concatenated without separators (e.g., 'humangeminio3-high'), making the figure difficult to read. Please add spacing or line breaks.","section":"Figure 7"},{"comment":"The expression '442 mod 187' appears to be a typographical artifact (likely 44^2 mod 187). Please render the exponent in LaTeX for clarity.","section":"Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The paper's proof problems are authored by the expert panel, who are also the authors (Appendix F.2). This is not disqualifying, but it heightens the need for the grader-calibration and proof-contamination checks requested in the major comments. If the authors supply the mean signed difference/limits of agreement and a contamination probe for proofs, and adjust the abstract/conclusion CTF framing, I would view the revised manuscript as acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: AICrypto is the first cryptography-specific benchmark for LLMs that combines MCQs, CTF challenges, and proof problems, and it is a solid resource. The main empirical pattern—LLMs at or above human level on multiple choice, close on proof problems, and far behind on CTF exploitation—is credible. But the abstract overstates the CTF result, and the proof-grading validation is missing a calibration check that the authors can easily add.\n\nWhat the paper does well: the construction is careful. The 135 MCQs are rewritten and verified; the 150 CTF challenges are recent, categorized, and evaluated with an agentic framework, including a useful check showing no advantage on challenges with public official solutions; and the 30 proof problems are original course material with detailed rubrics and reference solutions. The automated proof grading, using two LLM graders and averaging six scores, reports high correlation with human scoring on 306 samples. That is real work and a genuine contribution.\n\nThe soft spots are in the claims rather than the core construction. First, the abstract and conclusion say models match or surpass humans on \"exploiting common vulnerabilities,\" but the overall CTF success rate for the best model is 56.0% versus 81.2% for humans. The paper's own Section 4.2 is more careful, so the abstract should align with the evidence. Second, the proof-grading validation reports Pearson and Spearman correlations but not the mean signed difference between LLM and human scores. Correlation does not rule out systematic leniency, and the paper's failure analysis shows LLM proofs that look correct but have hidden flaws. Reporting bias and limits of agreement would settle this. Third, the human baselines are heterogeneous—top CTF scoreboard participants, top five course scores, three PhD students on MCQs—but they are disclosed and reasonable as upper-bound proxies. Fourth, the proof problems are assumed uncontaminated because they were never publicly released; that is stated but not tested. A retrieval probe would be consistent with what they already did for MCQs.\n\nWho should read this: anyone building LLM benchmarks for cybersecurity or using LLMs in offensive security. It deserves peer review with requests for revision, not desk rejection. I would ask the authors to fix the abstract, add the calibration analysis, and say more about how the human baselines compare across task types.","headline":"A solid, much-needed crypto benchmark for LLMs, but the abstract overclaims CTF performance and the proof-grading validation needs a calibration check.","tokens_in":37218,"tokens_out":2968,"would_cite":true,"duration_ms":35473,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 315-task benchmark finds that top language models match or surpass human experts on cryptography quizzes and proofs, but trail by 25–30 points on capture-the-flag exploits.","keywords":["AICrypto","large language models","cryptography benchmark","capture-the-flag challenges","automated proof grading","LLM evaluation","cryptanalysis","data contamination"],"falsifier":"Search public web and code-hosted corpora for the exact statements and reference solutions of the 30 proof problems, and check whether models reproduce the problem wording verbatim; if a substantial fraction is found online or is echoed by models, the proof results are inflated rather than a measure of reasoning.","tokens_in":36225,"feed_emoji":"🔐","tokens_out":6050,"duration_ms":67986,"temperature":0.7,"pith_summary":"This paper builds AICrypto, a benchmark of 315 cryptography tasks: 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, curated by cryptography experts and scored against human expert baselines. Its central finding is a split capability profile: the best LLMs outperform the best human expert on factual crypto questions (97.8% vs. 94.1%), come close on formal proofs (85.4% vs. 94.0%), but trail humans by more than 25 percentage points on capture-the-flag exploitation (56.0% vs. 81.2%). The paper argues this shows LLMs have strong memorized knowledge and surface-level proof mimicry but weak end-to-end vulnerability exploitation, multi-step reasoning, and reliable arithmetic. If true, this maps where LLMs can be trusted in cryptographic work, such as recall and routine proof drafting, and where human oversight is still required.","feed_headline":"Top AI models ace crypto trivia but flub real-world attacks","feed_subtitle":"A 315-task benchmark shows LLMs nearly match human experts on proofs yet trail by 25 points on CTF exploitation.","key_machinery":"The object that carries the comparison is AICrypto itself: a curated, expert-reviewed evaluation of 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, with human expert baselines for all three parts. Its technical mechanisms are an agentic CTF harness that lets models run commands and scripts inside a sandbox over up to 100 turns and is scored by pass@3, and rubric-based automated proof grading in which two grader models each score an answer three times, with the six-score average correlating with human experts at Pearson 0.9025. The benchmark's design is what makes the central comparison, LLMs versus human experts across distinct cryptographic skills, measurable at all.","core_discovery":"The paper establishes a three-tier capability landscape for LLMs in cryptography. On AICrypto's multiple-choice component, the best model makes only three errors in 135 questions, surpassing the best human expert; on proof problems the top model scores 85.4% against a 94.0% human baseline; but on CTF challenges the best model solves only 56.0% of tasks against an 81.2% human rate. Manual failure analysis attributes the gap to inaccurate large-number arithmetic, excessive pattern matching over analysis, shallow mathematical comprehension, and proofs that look plausible but contain logical gaps. Because all tasks were expert-reviewed and human baselines were collected for every component, the paper treats these figures as the first systematic measure of LLM cryptographic proficiency rather than as isolated anecdotes.","pith_inferences":["If the 30 proof problems ever reach public training corpora, the reported near-human proof performance would likely overstate genuine formal-reasoning ability; the paper's own examples of plausible-looking proofs with logical gaps point in that direction.","The CTF gap may be partly an artifact of the 100-turn, single-agent scaffold, so evaluating the same models with stronger planning or multi-agent collaboration could narrow the 25-point gap without changing the underlying model weights.","Because MCQ accuracy is near saturation, the next useful benchmark step is to shift weight toward novel protocol design, side-channel awareness, and problems requiring exact large-number arithmetic.","The paper's contamination checks cover multiple-choice and CTF tasks but not proofs, so a public release of the proof set with independent contamination probes would be a natural validation of the benchmark's central comparison."],"forward_implications":["For factual cryptography knowledge, top LLMs can already serve as reliable assistants, since they outperform the best human expert in this study on multiple-choice questions.","For formal proof work, the best models produce usable routine proofs that approach, but do not reach, expert level, and the benchmark's automated rubric grading makes such evaluation scalable.","For vulnerability exploitation of real cryptographic implementations, current LLMs are not yet dependable, with a 25–30 point gap on CTF challenges and especially weak performance on lattice-based problems.","Reasoning-oriented models and higher reasoning effort consistently outperform general models, indicating that cryptographic capability scales with inference-time computation.","The failure analysis identifies persistent weaknesses, including modular arithmetic errors, attack patterns applied without checking preconditions, and plausible-looking proofs with logical gaps, that future models must address."],"supporting_citations":[{"why":"Provides the prior CTF benchmark whose evaluation approach the CTF agent framework builds on and extends.","marker":"Shao et al., 2024b"},{"why":"Supplies the cybersecurity benchmark baseline that motivates including capture-the-flag cryptographic tasks.","marker":"Zhang et al., 2025b"},{"why":"Underlies the reasoning-and-acting agent loop used for the CTF challenges.","marker":"Yao et al., 2023"},{"why":"Provides the pass@k metric adopted for CTF success rate and best-of-three proof scoring.","marker":"Kulal et al., 2019"},{"why":"The style of automated proof graders that AICrypto adapts for rubric-based proof evaluation.","marker":"Luong et al., 2025"},{"why":"Prior decryption-task study used to motivate the need for a comprehensive cryptography-specific benchmark.","marker":"Li et al., 2025"},{"why":"Cited as evidence that automated evaluation of proof problems is difficult, motivating the rubric-grader design.","marker":"Hendrycks et al., 2021"}],"fun_headline_variants":["LLMs ace crypto theory, flub capture-the-flag challenges","AI models know crypto theory but trip over real attacks","Crypto benchmark: LLMs nearly human on proofs, weak on CTF","LLMs ace crypto quizzes but fail to break real systems","AI passes crypto theory test, fails hands-on exploits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the 30 proof problems, drawn from unpublished course materials, have not leaked into LLM training data; the paper runs contamination probes for the multiple-choice and CTF parts but not for the proofs, so if those problems reached the web, the near-human proof scores would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace crypto theory, flub capture-the-flag challenges","AI models know crypto theory but trip over real attacks","Crypto benchmark: LLMs nearly human on proofs, weak on CTF","LLMs ace crypto quizzes but fail to break real systems","AI passes crypto theory test, fails hands-on exploits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000814,"raw_usage":{"total_tokens":3546,"prompt_tokens":898,"completion_tokens":2648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2563}},"tokens_in":514,"tokens_out":2648,"duration_ms":20957,"temperature":1.0,"reasoning_tokens":2563,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:51:08.288072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search public web and code-hosted corpora for the exact statements and reference solutions of the 30 proof problems, and check whether models reproduce the problem wording verbatim; if a substantial fraction is found online or is echoed by models, the proof results are inflated rather than a measure of reasoning.","supporting_citations":[],"review_version":1}