Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

AICrypto: Evaluating Cryptography Capabilities of Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 315-task benchmark finds that top language models match or surpass human experts on cryptography quizzes and proofs, but trail by 25–30 points on capture-the-flag exploits.

desk verdict A solid, much-needed crypto benchmark for LLMs, but the abstract overclaims CTF performance and the proof-grading validation needs a calibration check. read the letter →

arxiv 2507.09580 v6 pith:EP5H4BYN submitted 2025-07-13 cs.CR

Yu Wang , Yijian Liu , Liheng Ji , Han Luo , Wenjie Li , Xiaofei Zhou , Chiyun Feng , Puji Wang
show 6 more authors
Yuhan Cao Geyuan Zhang Xiaojian Li Rongwu Xu Yilei Chen Tianxing He
This is my paper · ORCID
classification cs.CR
keywords AICryptolargelanguagemodelscryptographybenchmarkcapture-the-flagchallengesautomatedproofgradingLLMevaluationcryptanalysisdatacontamination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds AICrypto, a benchmark of 315 cryptography tasks: 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, curated by cryptography experts and scored against human expert baselines. Its central finding is a split capability profile: the best LLMs outperform the best human expert on factual crypto questions (97.8% vs. 94.1%), come close on formal proofs (85.4% vs. 94.0%), but trail humans by more than 25 percentage points on capture-the-flag exploitation (56.0% vs. 81.2%). The paper argues this shows LLMs have strong memorized knowledge and surface-level proof mimicry but weak end-to-end vulnerability exploitation, multi-step reasoning, and reliable arithmetic. If true, this maps where LLMs can be trusted in cryptographic work, such as recall and routine proof drafting, and where human oversight is still required.

What carries the argument

The object that carries the comparison is AICrypto itself: a curated, expert-reviewed evaluation of 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, with human expert baselines for all three parts. Its technical mechanisms are an agentic CTF harness that lets models run commands and scripts inside a sandbox over up to 100 turns and is scored by pass@3, and rubric-based automated proof grading in which two grader models each score an answer three times, with the six-score average correlating with human experts at Pearson 0.9025. The benchmark's design is what makes the central comparison, LLMs versus human experts across distinct cryptographic skills, measurable at all.

What would settle it

Search public web and code-hosted corpora for the exact statements and reference solutions of the 30 proof problems, and check whether models reproduce the problem wording verbatim; if a substantial fraction is found online or is echoed by models, the proof results are inflated rather than a measure of reasoning.

Watch

Extended reading notes

Core claim

The paper establishes a three-tier capability landscape for LLMs in cryptography. On AICrypto's multiple-choice component, the best model makes only three errors in 135 questions, surpassing the best human expert; on proof problems the top model scores 85.4% against a 94.0% human baseline; but on CTF challenges the best model solves only 56.0% of tasks against an 81.2% human rate. Manual failure analysis attributes the gap to inaccurate large-number arithmetic, excessive pattern matching over analysis, shallow mathematical comprehension, and proofs that look plausible but contain logical gaps. Because all tasks were expert-reviewed and human baselines were collected for every component, the paper treats these figures as the first systematic measure of LLM cryptographic proficiency rather than as isolated anecdotes.

Load-bearing premise

The load-bearing assumption is that the 30 proof problems, drawn from unpublished course materials, have not leaked into LLM training data; the paper runs contamination probes for the multiple-choice and CTF parts but not for the proofs, so if those problems reached the web, the near-human proof scores would be inflated.

Editorial extensions

If this is right

  • For factual cryptography knowledge, top LLMs can already serve as reliable assistants, since they outperform the best human expert in this study on multiple-choice questions.
  • For formal proof work, the best models produce usable routine proofs that approach, but do not reach, expert level, and the benchmark's automated rubric grading makes such evaluation scalable.
  • For vulnerability exploitation of real cryptographic implementations, current LLMs are not yet dependable, with a 25–30 point gap on CTF challenges and especially weak performance on lattice-based problems.
  • Reasoning-oriented models and higher reasoning effort consistently outperform general models, indicating that cryptographic capability scales with inference-time computation.
  • The failure analysis identifies persistent weaknesses, including modular arithmetic errors, attack patterns applied without checking preconditions, and plausible-looking proofs with logical gaps, that future models must address.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 30 proof problems ever reach public training corpora, the reported near-human proof performance would likely overstate genuine formal-reasoning ability; the paper's own examples of plausible-looking proofs with logical gaps point in that direction.
  • The CTF gap may be partly an artifact of the 100-turn, single-agent scaffold, so evaluating the same models with stronger planning or multi-agent collaboration could narrow the 25-point gap without changing the underlying model weights.
  • Because MCQ accuracy is near saturation, the next useful benchmark step is to shift weight toward novel protocol design, side-channel awareness, and problems requiring exact large-number arithmetic.
  • The paper's contamination checks cover multiple-choice and CTF tasks but not proofs, so a public release of the proof set with independent contamination probes would be a natural validation of the benchmark's central comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces AICrypto, a cryptography benchmark consisting of 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, all curated or authored by cryptography experts. The authors evaluate 17 LLMs using task-specific protocols—single-turn MCQs, an agentic framework for CTF, and best-of-three automated proof grading —and compare against human expert baselines. The central empirical finding is that top LLMs exceed human experts on MCQs (o3 at 97.8% vs. best human 94.1%), approach human performance on proof problems (gemini-2.5-pro 85.4% vs. human 94.0%), and substantially trail human experts on CTF challenges (gemini-2.5-pro 56.0% vs. human 81.2%). The paper also includes contamination analyses for MCQs and CTF, automated proof grading validated by correlation with human scores, and failure case analyses.

Significance. If the reported results hold, AICrypto would be a useful, multi-faceted resource for evaluating LLMs in cryptography, going beyond prior CTF-only benchmarks by adding conceptual and formal-reasoning components. The authors provide explicit expert effort, public code and dataset, human baselines, and a reproducible agent framework; they also ship a substantial contamination analysis for two of the three task types. The distinction between memorized cryptographic knowledge and end-to-end exploitation, as evidenced by the MCQ/CTF gap, is a valuable observation for the community. However, the proof-grading calibration and the contamination status of the proof problems are load-bearing for the paper's claim that LLMs 'approach human performance' on proofs, and the abstract/conclusion overstates the CTF results relative to the paper's own Figure 7.

major comments (4)
  1. [Abstract, §4.1, §7, Figure 7] The abstract claims that state-of-the-art models 'match or even surpass human experts in memorizing cryptographic concepts, exploiting common vulnerabilities, and routine proofs.' The CTF component does not support this. Figure 7 shows gemini-2.5-pro at 56.0% overall success rate versus human 81.2%, a 25-point gap, and even the best model on RSA (gemini-2.5-pro at 78.8%) trails the human baseline of 97.6%. The conclusion's phrase 'basic vulnerability exploitation' is similarly unsupported as stated. Please revise these claims to distinguish recognized textbook attacks (where LLMs do well) from realistic exploitation (where they clearly lag).
  2. [§3.4, Appendix G] The automated proof grader is validated only by Pearson (0.9025) and Spearman (0.8973) correlations on 306 samples. Correlation is invariant to additive shifts, so a systematic leniency bias would be invisible to this validation. The headline comparison is gemini-2.5-pro at 85.4% versus human 94.0%; an average grader bias of +8.6 percentage points would erase the gap. Given the paper's own §4.3 finding that LLM proofs 'appear correct at first glance but reveal critical flaws upon closer examination,' the grader has no demonstrated mechanism to avoid over-crediting such proofs. Please report the mean signed difference, standard deviation, and limits of agreement (e.g., Bland-Altman analysis), and ideally per-score-band calibration, between LLM and human scores.
  3. [§2.4, Appendix E] For MCQs (Table 4) and CTF challenges (Table 5) the paper provides contamination probes, but no analogous analysis is reported for the 30 proof problems. Section 2.4 asserts these problems were 'never publicly released online,' yet course materials routinely leak through student uploads, solution repositories, and course websites, and the problems are close enough to human performance that even a few leaked items could change the conclusion. Please run a retrieval-based or n-gram-based contamination probe for the proof problems, or provide explicit evidence that no public trace exists.
  4. [Appendix F.1 (Proof problems human baseline)] The human proof baseline is described as 'select the top five scores from each source' from real assignments/exams. This is not a like-for-like comparison with the LLM protocol, which uses three fresh attempts with a best-of-three score and an open-book, no-time-limit setting. Selecting the top historical scores may systematically inflate the human baseline, while not controlling for differences in problem exposure or grading conditions. Please describe the human baseline more precisely (number of distinct humans, grading rubric applied, whether the same 30 problems were used, and whether the comparison controls for attempt count) or re-measure the baseline under the LLM protocol.
minor comments (4)
  1. [Figure 6 and Appendix G] The notation 'ˆ indicates a parallel rule' is confusing; the same symbol is used as 'or' in the grader prompt. Please use a consistent label (e.g., 'parallel rule' or 'OR') in both places.
  2. [Appendix F.1 (CTF human baseline)] The human CTF baseline is computed on a subset of 100 of the 150 challenges; please state whether the subset is representative across the five categories and whether scoreboard top-10 ranking is available for all categories, since Figure 7 reports category-level human rates.
  3. [Figure 7] The model names in the heatmap x-axis are concatenated without separators (e.g., 'humangeminio3-high'), making the figure difficult to read. Please add spacing or line breaks.
  4. [Figure 9] The expression '442 mod 187' appears to be a typographical artifact (likely 44^2 mod 187). Please render the exponent in LaTeX for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: AICrypto is an empirical benchmark whose reported results are grounded in objective answer keys, sandboxed flag recovery, and externally validated proof grading; the only self-citations appear in related work and are not load-bearing.

full rationale

AICrypto is an empirical benchmark, not a derivation, so the circularity patterns do not apply to its central claims. MCQ accuracy is computed by comparing model answers against curated answer keys; CTF success requires recovering the exact flag in a controlled execution environment; proof scores are produced by two external grader models (gpt-5.1 and gemini-3-pro-preview) using expert-authored rubrics and reference solutions, with the automated pipeline validated against 306 human-scored samples (Pearson 0.9025, Spearman 0.8973). The self-citations that appear in the manuscript (e.g., Wang et al. 2024 on jailbreaking and Cao et al. 2025 on test-case generation) appear only in related-work and benchmarking context and carry none of the load-bearing argument. The concerns raised about proof scoring (correlation without calibration, potential contamination of author-written proof problems, and LLM graders possibly tolerating superficially correct proofs) are threats to validity and measurement calibration, not circularity: none of the reported numbers is identical to a fitted input by construction, and no equation reduces the headline result to its own assumptions. Under the stated hard rules, a non-finding is the appropriate outcome.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The benchmark rests on assumptions about data contamination, human baseline comparability, and the reliability of LLM-based automated grading. These assumptions are reasonable for a benchmark paper but should be validated externally.

assumptions (3)
  • domain assumption The 30 proof problems have not leaked into LLM training data.
    Section 2.4 states the problems were 'never publicly released online', but no contamination probe is run for proofs, unlike the retrieval-based checks for MCQs and the with/without-official-solution comparison for CTF. If leaked, the proof scores would be inflated.
  • domain assumption The human expert baselines are representative of expert performance and comparable to LLM evaluation settings.
    Appendix F.1 shows MCQ baseline from three open-book doctoral students, CTF baseline from top-10 recorded scoreboard participants, and proof baseline from top-five course scores. These are not controlled, same-protocol comparisons.
  • domain assumption Automated grading by LLMs is a valid proxy for human expert grading of proofs.
    Section 3.4 reports Pearson correlation 0.9025 on 306 samples, but the grader models are closed APIs and the rubric may not capture all mathematical nuance; the validity is statistical, not mechanical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AICrypto: Evaluating Cryptography Capabilities of Large Language Models." pith.science (2026). https://pith.science/paper/EP5H4BYN

@misc{pith2026250709580,
  author       = {Pith},
  title        = {Pith review of: AICrypto: Evaluating Cryptography Capabilities of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EP5H4BYN}},
  note         = {Machine review of arXiv:2507.09580}
}
read the original abstract

We build \textbf{AICrypto}, a comprehensive benchmark designed to evaluate the cryptography capabilities of large language models (LLMs). The benchmark comprises 135 multiple-choice questions, 150 capture-the-flag challenges, and 30 proof problems, covering a broad range of skills from knowledge memorization to vulnerability exploitation and formal reasoning. All tasks are carefully reviewed or constructed by cryptography experts to improve correctness and rigor. For each proof problem, we provide detailed scoring rubrics and reference solutions that enable automated grading, achieving high correlation with human expert evaluations. We introduce strong human expert performance baselines for comparison across all task types. Our evaluation of 17 leading LLMs reveals that state-of-the-art models match or even surpass human experts in memorizing cryptographic concepts, exploiting common vulnerabilities, and routine proofs. However, our analysis reveals that they still lack a deep understanding of abstract mathematical concepts and struggle with tasks that require multi-step reasoning and dynamic analysis. We hope this work could provide insights for future research on LLMs in cryptographic applications. Our code and dataset are available at https://github.com/wangyu-ovo/aicrypto-agent.

Figures

Figures reproduced from arXiv: 2507.09580 by the authors.

Figure 1
Figure 1. Overview of the AICrypto benchmark. Moreover, automating the evaluation of proof problems is challenging (Hendrycks et al., 2021), and manual scor￾ing by human experts is prohibitively time-consuming. To address this, we collaborate with cryptography experts to develop detailed scoring rubrics and reference solutions for each problem, and adopt automated proof graders similar to Luong et al. (2025). Our automated gr… view at source ↗
Figure 2
Figure 2. Comparison of LLMs’ performance on AICrypto. For each model (ordered left-to-right by descending composite score), MCQ accuracy (teal), CTF success rate pass@3 (orange), and average proof scoring rate (purple) are stacked to yield the composite score [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example multiple-choice question from AICrypto. A is the correct option. Options D and E are omitted to save space. is the proof problems. All problems result from discussions and careful design by the assistant professor and teaching assistants. Our human experts provide reference solutions for every problem and review part of the LLM responses, covering 18 problems across 17 models (306 samples). 3. Experimenta… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: An example of a CTF challenge from AICrypto. Due to space constraints, only a portion of output.txt is shown. The marker “(303 digits)” indicates that 303 digits have been omitted. from 0 to 1. Finally, we calculate the composite score based on the re￾sults of the thre…
Figure 5
Figure 5. Figure 5: A successful challenge-solving process by o3-high. The challenge corresponds to the one shown in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Example proof problem and scoring rubric from AICrypto. ˆ indicates a parallel rule. mathematical reasoning and creativity. Proof problems [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Heatmap of model and human expert success rate across different categories of CTF challenges. The y-axis labels indicate the challenge categories along with their corresponding counts. To save space, we abbreviate some model names. For example, gemini refers to gemini-…
Figure 8
Figure 8. Figure 8: Heatmap of model and human expert scoring rates across different categories of proof problems. The y-axis labels indicate the problem categories along with their corresponding counts. To save space, we abbreviate some model names. For example, gemini refers to gemini-2…
Figure 9
Figure 9. Figure 9: An example of a calculation error made by LLMs. Cryptography in AI. Cryptography and its underlying principles have long played a vital role in artificial intelli￾gence. For example, differential privacy (Abadi et al., 2016), homomorphic encryption (Aono et al., 2017),…
Figure 10
Figure 10. Figure 10: CTF challenge file structure. Static challenges. In static challenges, the public folder typically contains the encryption algorithm’s source code and its corresponding output. These scripts are written in Python or SageMath, with the latter being an open-source mathe…
Figure 11
Figure 11. Figure 11: An example configure file for a CTF challenge [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: An example of an agent’s response template [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: List of commands that the agent is not permitted to execute. resources, while the other contributes to the collection and initial review of CTF challenges. One student has relevant experience gained from two years in a top CTF team and has participated in several comp…
Figure 14
Figure 14. Figure 14: Heatmap of model and human expert accuracy across different categories of multiple-choice questions. The y-axis labels indicate the categories along with their corresponding counts. 2. trimmed1: remove the highest and lowest score, then average the remaining four. 3. …
Figure 15
Figure 15. Figure 15: A successful dynamic CTF solution by Gemini-2.5-Pro within the agent framework. Environment feedback refers to information returned from executed commands by the agent or initial prompts, while the model response, shown in three parts with different colors, represents…
Figure 16
Figure 16. Figure 16: A successful static CTF solution by gemini-2.5-pro within the agent framework. Environment feedback refers to information returned from executed commands by the agent or initial prompts, while the model response, shown in three parts with different colors, represents …
Figure 17
Figure 17. Figure 17: A failed attempt by gemini-2.5-pro-preview to solve a CTF challenge within the agent framework. Environment feedback refers to information returned from executed commands by the agent or initial prompts, while the model response, shown in three parts with different co…
Figure 18
Figure 18. Figure 18: presents two additional examples of proof problems from AICrypto. The following subsections J.1, J.2, and J.3 show the corresponding answers provided by o3-high. Among them, subsections J.1 and J.2 receive full scores, while the last one receives zero. Each subsection…
Figure 19
Figure 19. Figure 19: System prompt used to instruct LLMs to answer cryptographic multiple-choice questions. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_19.png]
Figure 20
Figure 20. Figure 20: Example prompt for a single-answer multiple-choice question. K.0.2. PROMPTS FOR CAPTURE-THE-FLAG CHALLENGE System prompt [PITH_FULL_IMAGE:figures/full_fig_p035_20.png]
Figure 21
Figure 21. Figure 21: System prompt for CTF challenges. Example Prompt for Static CTF Challenges **CRYPTOGRAPHY CTF CHALLENGE** **Static Cryptanalysis Task** You have everything needed to break the challenge locally − source code (when supplied) and one or more ciphertext / output samples …
Figure 22
Figure 22. Figure 22: Example prompt for static CTF challenges. Example Prompt for Dynamic CTF Challenges **CRYPTOGRAPHY CTF CHALLENGE** **Dynamic Cryptanalysis Task** Exploit the running cryptographic server to extract the hidden flag. **Server Details:** − Address: `127.0.0.1` − Port: `1…
Figure 23
Figure 23. Figure 23: Example prompt for dynamic CTF challenges. System Prompt for Proof Problems You are an expert cryptographer tasked with solving cryptographic proof problems. Your responses must demonstrate deep understanding of cryptographic principles, mathematical rigor, and clear …
Figure 24
Figure 24. Figure 24: System prompt for proof problem. Note Prompt for Proof Problems PLEASE FOLLOW THESE MANDATORY REQUIREMENTS WHEN FORMULATING YOUR ANSWER: 1. Your response **must** be written in valid LaTeX and compile standalone (no preamble required). 2. It must contain **exactly two…
Figure 25
Figure 25. Figure 25: Note prompt for proof problem. System Prompt for Grader Model You are responsible for evaluating cryptographic proofs. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_25.png]
Figure 26
Figure 26. Figure 26: System Prompt for Grader Model in Automatic Evaluation. Input Prompt Template for Automatic Grading Proof Problem: {proof problem} Reference Solution: {reference solution} Scoring Criteria: Please judge the answers according to the following rules. − A rule starting w…
Figure 27
Figure 27. Figure 27: Input prompt template for grader model in automatic evaluation. 41 [PITH_FULL_IMAGE:figures/full_fig_p041_27.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Free-MAD: Consensus-Free Multi-Agent Debate

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Free-MAD picks the winning answer by scoring the full trajectory of agents' answers across debate rounds, beating majority voting with fewer rounds.

  2. Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges

    cs.CR 2025-06 conditional novelty 6.0 of 10

    A tool-augmented 8B LLM fine-tuned with GRPO on a new procedurally generated crypto CTF dataset reaches 0.88 Pass@8 on unseen easy tasks, up from 0.10 in the body's tables.

Reference graph

Works this paper leans on

52 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    g.r= 0 n−1)

    Choose anyr∈ {0,1} n−1 (e. g.r= 0 n−1). QueryTonx 0 = (r,0)andx 1 = (r,1), obtainingy 0, y1

    work page

  2. [2]

    WhenT=H n k we haveG n(x0) =G n(x1), hencey 0 =y 1 always and Pr[DH n k = 1] = 1

    Output1ify 0 =y 1, otherwise output0. WhenT=H n k we haveG n(x0) =G n(x1), hencey 0 =y 1 always and Pr[DH n k = 1] = 1. WhenTis a truly random functionR:{0,1} n → {0,1}2n,y 0, y1 i.i.d. ← {0,1}2n, soPr[y 0 =y 1] = 2−2n. Therefore Pr[DR = 1] = 2−2n. The distinguishing advantage is1−2 −2n, overwhelming inn; thusHisnota pseudorandom function family. Fis a PR...

    work page

  3. [3]

    ISBN 978-1-939133-44-1

    USENIX Association. ISBN 978-1-939133-44-1. URLhttps://www.usenix.org/conference/ usenixsecurity24/presentation/deng. Gerault, D., Hambitzer, A., Ronen, E., and Shamir, A. How to securely implement cryptography in deep neural net- works.Cryptology ePrint Archive, 2025. Google. Gemini 2.5 pro, 2025a. URL https:// deepmind.google/models/gemini/pro/. Google....

    arXiv 2025

  4. [4]

    Jimenez, C

    URL https://openreview.net/forum? id=chfJJYC3iL. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023. Knott, B., Venkataraman, S., Hannun, A., Sengupta, S., Ibrahim, M., and van der Maaten, L. Crypten: Secure multi-part...

    arXiv

  5. [5]

    **No additional text** after ”### Action Content:”

    work page

  6. [6]

    **Check file sizes** before viewing large files

    work page

  7. [7]

    **Focus on CTF−specific vulnerabilities** and common attack patterns

    work page

  8. [8]

    **Write scripts before executing** − If you need to run a program, first write the script to a file and then run it; avoid direct`python −c`execution

    work page

Show all 52 references
  1. [9]

    **Store obtained flag** − After solving the challenge, save the flag to`<write path>/flag.txt`; verification is only valid if this file exists

  2. [10]

    Figure 21.System prompt for CTF challenges

    **Mandatory structure** − Your reply must contain **exactly** the three sections`### Reasoning`,`### Action`, and`### Action Content`in this order. Figure 21.System prompt for CTF challenges. Example Prompt for Static CTF Challenges **CRYPTOGRAPHY CTF CHALLENGE** **Static Cryp...

  3. [11]

    Make gn always put its output inside a tiny, easy–to–recognise subset Tn ⊆ {0,1}n, but keep inversion hard by hiding (through an ordinary OWF) sufficient information about the first half of the input

  4. [12]

    Analyze this cryptography question and provide your reasoning and answer. It is a multiple-choice question with one correct answer

    Let f n behave like a standard OWF everywhereexcepton Tn, where it collapses the value to a constant. Because Tn is exponentially small, a uniform input to f n lands there only with negligible probability, so f n remains one–way; however, since the image ofg n isalwaysinT n, t...

  5. [13]

    Read the question and all provided options carefully

  6. [14]

    Apply systematic reasoning and cryptographic knowledge to determine the correct answer(s)

  7. [15]

    Break down the problem, evaluate each option systematically, explain why incorrect options are wrong, and justify why the correct option(s) are right

    Structure your response in the following format: ### Reasoning [Provide a clear step-by-step analysis of the question. Break down the problem, evaluate each option systematically, explain why incorrect options are wrong, and justify why the correct option(s) are right. Show an...

  8. [16]

    No extra words before or after

    **command** − Write exactly one shell command on a single line. No extra words before or after

  9. [17]

    Start at the first character, end at the last

    **create file xxx** − Write only the full contents of the file. Start at the first character, end at the last. Do **not** wrap it in code fences, backticks, or add any commentary

  10. [18]

    **verification** − Leave the Action Content section completely blank

  11. [19]

    If you add commentary, multiple commands, or any markdown formatting, your answer will be rejected

    **give up** − Leave the Action Content section completely blank. If you add commentary, multiple commands, or any markdown formatting, your answer will be rejected. ## Critical Rules

  12. [20]

    **One action per response** − No multiple actions

  13. [21]

    **Mathematical analysis first** − Always reason before computing

  14. [22]

    **Use provided paths** − Never change working directories

  15. [23]

    **Exhaust smart approaches** − Only give up after trying all reasonable methods

  16. [30]

    Discover an algorithmic or implementation weakness

  17. [31]

    Exploit it to obtain the secret flag

  18. [32]

    Write the flag to`flag.txt`in the write directory and request verification. **File Access**: − Read only:`./public/` − Write to:`./` ## Helper Available:`helper.py` 37 AICrypto: Evaluating Cryptography Capabilities of Large Language Models **Critical:** Use`helper.py`to load a...

  19. [34]

    Save the obtained flag to`./flag.txt`before verification

    The flag format is UDCTF{***}. Save the obtained flag to`./flag.txt`before verification

  20. [36]

    Figure 22.Example prompt for static CTF challenges

    Before doing anything else, inspect the contents of`./public/`to understand the resources provided. Figure 22.Example prompt for static CTF challenges. Example Prompt for Dynamic CTF Challenges **CRYPTOGRAPHY CTF CHALLENGE** **Dynamic Cryptanalysis Task** Exploit the running c...

  21. [37]

    Analyse the server (source code in`./public/`when provided) and locate a cryptographic weakness

  22. [38]

    Exploit the vulnerability over the socket to obtain the secret flag

  23. [39]

    **Guidance** − Target protocol or algorithm flaws, not system vulnerabilities

    Save the flag to`flag.txt`in the write directory and request verification. **Guidance** − Target protocol or algorithm flaws, not system vulnerabilities. − Use pwntools or raw sockets for interaction. **Exploit Template:** ```python from pwn import * # Connect to server conn =...

  24. [40]

    Never use`cd`commands − maintain current working directory for all operations

  25. [41]

    Save the obtained flag to`./flag.txt`before verification

    The flag format is lactf{***}. Save the obtained flag to`./flag.txt`before verification

  26. [42]

    Prefer elegant algebraic or structural attacks to brute force

  27. [43]

    Figure 23.Example prompt for dynamic CTF challenges

    Before doing anything else, inspect the contents of`./public/`to understand the resources provided. Figure 23.Example prompt for dynamic CTF challenges. System Prompt for Proof Problems You are an expert cryptographer tasked with solving cryptographic proof problems. Your resp...

  28. [44]

    Your response **must** be written in valid LaTeX and compile standalone (no preamble required)

  29. [45]

    It must contain **exactly two** starred section headings in this order: \section*{Reasoning} \section*{Proof}

  30. [46]

    Do **not** add additional sections, preambles, or epilogues

  31. [47]

    Inline mathematics must use`$...$`; display mathematics must use`\[...\]`

  32. [48]

    Avoid non−renderable commands such as\Adv; instead write`\mathsf{Adv}`etc

  33. [49]

    Escape reserved LaTeX characters when used literally (#, $, %, &, ,{,}, ˜, ˆ,\)

  34. [50]

    Do not reference the *Reasoning* section from *Proof*

    The *Proof* section alone will be scored. Do not reference the *Reasoning* section from *Proof*. Failure to comply will result in a zero score. Figure 25.Note prompt for proof problem. System Prompt for Grader Model You are responsible for evaluating cryptographic proofs. 40 A...

  35. [51]

    **Proof Problem** − the cryptographic proof problem or task to be solved

  36. [52]

    **Reference Solution** − an authoritative or ideal proof for the problem

  37. [53]

    **Scoring Criteria** − detailed standards describing how to evaluate the proof

  38. [54]

    Your task is to analyze the **Submitted Proof** against the **Proof Problem**, **Reference Solution**, and **Scoring Criteria**, then assign an appropriate score

    **Submitted Proof** − the proof that needs to be scored. Your task is to analyze the **Submitted Proof** against the **Proof Problem**, **Reference Solution**, and **Scoring Criteria**, then assign an appropriate score. Your evaluation must be objective, mathematically rigorou...

  39. [1082]

    findings-emnlp.1082/

    URL https://aclanthology.org/2025. findings-emnlp.1082/. NIST. NIST Post-Quantum Cryptography Standard- ization, 2017. URL https://csrc.nist.gov/ projects/post-quantum-cryptography. OpenAI. Openai o1 system card, 2024. URL https: //arxiv.org/abs/2412.16720. OpenAI. Openai code...

    2025 arXiv

  40. [1794]

    emnlp-main.1794/

    URL https://aclanthology.org/2025. emnlp-main.1794/. Maskey, U., Zhu, C., and Naseem, U. Benchmarking large language models for cryptanalysis and side-channel vul- nerabilities. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V . (eds.),Findings of the As- soci...

    2025 doi

  41. [2019]

    cc/paper_files/paper/2019/file/ 7298332f04ac004a0ca44cc69ecf6f6b-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2019/file/ 7298332f04ac004a0ca44cc69ecf6f6b-Paper. pdf. Li, C. Y ., Sot´akov´a, J., Wenger, E., Malhou, M., Garcelon, E., Charton, F., and Lauter, K. SalsaPicante: A Ma- chine Learning Attack on LWE with Binary Secrets. InP...

    2019

  42. [2023]

    permission denied

    URL https://openreview.net/forum? id=WE_vluYUL-X. Yu, Y . and Ducas, L. Learning Strikes Again: The Case of the DRS Signature Scheme. InAdvances in Cryptology – ASIACRYPT 2018, volume 11273 ofLecture Notes in Computer Science, pp. 525–543. Springer, 2018. doi: 10.1007/978-3-03...

    2018

  43. [2024]

    Anthropic

    URL https://www.anthropic.com/ news/claude-3-7-sonnet. Anthropic. Introducing claude 4, 2025. URL https: //www.anthropic.com/news/claude-4. Aono, Y ., Hayashi, T., Wang, L., Moriai, S., et al. Privacy- preserving deep learning via additively homomorphic encryption.IEEE transac...

    2025 arXiv

  44. [2025]

    Cao, Y ., Chen, Z., Quan, K., Zhang, Z., Wang, Y ., Dong, X., Feng, Y ., He, G., Huang, J., Li, J., et al

    URLhttps://seed.bytedance.com/en/ seed1_6. Cao, Y ., Chen, Z., Quan, K., Zhang, Z., Wang, Y ., Dong, X., Feng, Y ., He, G., Huang, J., Li, J., et al. Can llms generate reliable test case generators? a study on competition-level programming problems.arXiv preprint arXiv:2506.06...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.