Pith. sign in

REVIEW 2 major objections 1 minor 28 references

MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation

T0 review · 2 major / 1 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM code repair on MPC repositories succeeds in only 17.1 percent of cases after security verification.

desk verdict MPC-Patch-Bench adds a needed repo-level benchmark for secure MPC code repair, but the uncalibrated verifier rules make the 17.1% and 40% figures hard to interpret. read the letter →

arxiv 2606.11416 v1 pith:U7QCL2YA submitted 2026-06-09 cs.CR cs.AI

Yukuan Zhang , Mengxin Zheng , Qian Lou This is my paper · ORCID
classification cs.CRcs.AI
keywords multi-partycomputationLLMcoderepairsecuritybenchmarkpatchMPCcryptographicverificationrepositorylevel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper creates MPC-Patch-Bench to test LLMs on fixing real code in multi-party computation projects at the repository level. It shows that standard benchmarks do not work for MPC because they ignore the need for cryptographic safety and numerical accuracy in addition to functional correctness. Using a specialized curation process and an MPC Verifier with dynamic and static checks, the benchmark contains 205 tasks. Evaluation reveals that even the best LLM fixes only about a fifth of the tasks, and many more are rejected when security rules are applied.

What carries the argument

MPC-Patch-Bench, consisting of a Data Curation Framework using a domain-specific agent with three cryptographic layers and a human-AI completion engine, plus an MPC Verifier with dynamic differential testing and MPC-specific static analysis rules.

What would settle it

A patch that passes all functional tests and the MPC Verifier but is later shown to leak private data or produce incorrect results under actual MPC execution.

Watch

Extended reading notes

Core claim

MPC-Patch-Bench provides 205 verified repository-level tasks for MPC code repair, where the strongest LLM achieves 22.9 percent functional resolution but only 17.1 percent verified resolution after the MPC Verifier rejects up to 40 percent of functionally correct patches for cryptographic or numerical-fidelity issues.

Load-bearing premise

The three cryptographic layers used by the curation agent and the static-analysis rules inside the MPC Verifier correctly identify all security-relevant defects without excessive false positives or missed vulnerabilities that would change the reported resolution rates.

Editorial extensions

If this is right

  • LLM-based code repair for MPC requires dedicated security verification beyond functional tests.
  • Up to 40 percent of patches that pass tests may still violate MPC security properties.
  • Benchmarks for privacy-preserving code must include domain-specific verifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This benchmark could be used to fine-tune LLMs specifically for cryptographic code.
  • Similar approaches might apply to other security-critical domains like zero-knowledge proofs.
  • The low resolution rates suggest LLMs lack understanding of MPC invariants.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces MPC-Patch-Bench, a repository-level benchmark for LLM-based code repair in Secure Multi-Party Computation (MPC) software. It describes a Data Curation Framework using a domain-specific agent with three cryptographic layers plus human-AI test synthesis to produce 205 verified instances across two MPC frameworks, and an MPC Verifier combining dynamic differential testing against plaintext oracles with static rules for unsafe reveals, insecure arithmetic, and illegal casts. Evaluation shows the strongest LLM achieves 22.9% functional resolution, reduced to 17.1% verified resolution, with up to 40% of functionally passing patches rejected on security or fidelity grounds.

Significance. If the curation and verification pipeline is shown to be reliable, the benchmark would provide the first repository-level evaluation resource for LLM agents on MPC repair tasks, quantifying the gap between functional correctness and cryptographic safety in this domain and motivating security-aware repair methods.

major comments (2)
  1. [Abstract, §3] Abstract and §3: the central claim of 205 fully verified instances and the 22.9%→17.1% drop (plus 40% rejection rate) rests on the three cryptographic curation layers and the MPC Verifier's static rules plus differential testing, yet the manuscript supplies no calibration against a ground-truth set of known-good/known-bad patches, no inter-rater agreement metrics with MPC experts, and no ablation of how rule changes affect instance counts or rejection statistics.
  2. [Abstract] Abstract: the description of verification is limited to high-level statements ('dynamic differential testing' and 'MPC-specific static analysis rules'); concrete oracles, test harness details, and how the 40% rejection statistic was measured are absent, making it impossible to assess whether the reported verified resolution rate is reproducible or sensitive to implementation choices.
minor comments (1)
  1. The paper would benefit from an explicit limitations section discussing potential false-positive rates in the static rules or coverage gaps in the differential-testing oracles.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive comments on the validation of our curation and verification pipeline. We address each major comment below and will incorporate revisions to improve transparency and reproducibility.

read point-by-point responses

  1. Referee: [Abstract, §3] Abstract and §3: the central claim of 205 fully verified instances and the 22.9%→17.1% drop (plus 40% rejection rate) rests on the three cryptographic curation layers and the MPC Verifier's static rules plus differential testing, yet the manuscript supplies no calibration against a ground-truth set of known-good/known-bad patches, no inter-rater agreement metrics with MPC experts, and no ablation of how rule changes affect instance counts or rejection statistics.

    Authors: We agree that the current manuscript lacks an explicit calibration study against a ground-truth set of known-good/known-bad patches and does not report inter-rater agreement metrics. The 205 instances were produced via the three-layer cryptographic filter plus human-AI test synthesis, which we treat as the verification mechanism. In the revision we will add (i) a new ablation subsection quantifying how relaxing or tightening each static rule changes the final instance count and the 40% rejection rate, and (ii) a table reporting agreement between the domain-specific agent outputs and the human reviewers who validated the synthesized tests. We do not currently possess an external ground-truth patch corpus, so the ablation will serve as the primary sensitivity analysis. revision: yes

  2. Referee: [Abstract] Abstract: the description of verification is limited to high-level statements ('dynamic differential testing' and 'MPC-specific static analysis rules'); concrete oracles, test harness details, and how the 40% rejection statistic was measured are absent, making it impossible to assess whether the reported verified resolution rate is reproducible or sensitive to implementation choices.

    Authors: Section 3 of the full manuscript already specifies the plaintext oracles used for differential testing and enumerates the three static rules (unsafe reveals, insecure arithmetic, illegal casts). The 40% rejection figure is obtained by running the MPC Verifier on all functionally passing patches produced by the evaluated LLMs. To address the abstract-level concern we will (a) expand the abstract with one additional sentence naming the oracle construction and the exact rejection criteria, and (b) add a short appendix with pseudocode of the test harness and the precise measurement procedure for the rejection rate. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity detected

full rationale

The paper constructs MPC-Patch-Bench via a curation agent (three cryptographic layers) and MPC Verifier (dynamic testing plus static rules), then reports empirical LLM resolution rates (22.9% functional, 17.1% verified) on the resulting 205 instances. These rates are direct evaluation outcomes on the newly defined benchmark rather than predictions or derivations that reduce by construction to fitted parameters, self-citations, or renamed inputs. No equations, uniqueness theorems, ansatzes, or self-referential steps appear in the provided text; the work is an empirical benchmark introduction self-contained against external general-purpose suites.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Only the abstract is available, so the ledger is limited to the high-level claims; the central claim rests on the correctness of the curation agent and verifier without external evidence supplied here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation." pith.science (2026). https://pith.science/paper/U7QCL2YA

@misc{pith2026260611416,
  author       = {Pith},
  title        = {Pith review of: MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7QCL2YA}},
  note         = {Machine review of arXiv:2606.11416}
}
read the original abstract

Repository-level benchmarks for evaluating Large Language Model (LLM) code repair on Secure Multi-Party Computation (MPC) software do not yet exist, and directly transplanting general-purpose benchmarks such as SWE-bench fails on three structural fronts: (i) MPC repositories are dominated by generic Python infrastructure rather than cryptographic logic; (ii) high-value MPC fixes lack the standardized tests rigid extraction pipelines require; and (iii) standard fail-to-pass evaluation is insufficient for code that must also be cryptographically safe. MPC is increasingly deployed for privacy-preserving machine learning, biomedical collaboration, and secure analytics. Existing MPC-specific code-synthesis efforts cover only operator-level or single-framework tasks; evaluating LLM agents on real repository-level MPC repair instead demands MPC-aware data curation and a verifier matched to the security and numerical-fidelity guarantees MPC programs must obey neither of which existing benchmarks provide. We introduce MPC-Patch-Bench, a repository-level benchmark organised around two frameworks. (1)The Data Curation Framework combines a domain-specific curation agent that filters raw pull requests through three cryptographic layers with a human-AI completion engine that synthesizes missing problem statements and Fail-to-Pass/Pass-to-Pass tests, yielding 205 fully verified instances. (2)The MPC Verifier provides dedicated security and numerical-fidelity checks via dynamic differential testing against plaintext oracles and MPC-specific static analysis rules that flag unsafe reveals, insecure arithmetic, and illegal public/private casts. The strongest evaluated LLM functionally resolves only 22.9% of MPC-Patch-Bench tasks; the MPC Verifier further reduces verified resolution to 17.1%, with up to 40% of functionally-passing patches rejected for cryptographic or numerical-fidelity violations.

Figures

Figures reproduced from arXiv: 2606.11416 by the authors.

Figure 1
Figure 1. Overview of MPC-Patch-Bench: the Data Curation Framework produces the verified [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Data Curation Framework: from 7,305 raw GitHub PRs through agent-driven curation [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The dual-stream evaluation framework for MPC-Patch-Bench. The framework parallelizes [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Distribution of MPC-Patch￾Bench’s 205 task instances across five MPC libraries. CrypTen dominates (29.8%, n = 61), while SecretFlow contributes the small￾est share (12.7%, n = 26) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Cumulative distribution functions (CDFs) of MPC-Patch-Bench tasks: (left) gold patch [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: MPC-Patch-Bench model ranking and overall functional resolution rates across 205 task [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Functional vs verified resolution per model on MPC-Patch-Bench, with attrition split into [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Example of domain-specific cryptographic static analysis. (A) A customised Semgrep rule [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Difficulty analysis of MPC-Patch-Bench tasks. (Left) A scatter plot of Fail-to-Pass test [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 3 canonical work pages

  1. [1]

    2024 , url =

    Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan , booktitle =. 2024 , url =

    2024

  2. [2]

    and Wettig, Alexander and Khandpur, Kabir and Zhang, Yanzhe and Hui, Binyuan and Press, Ofir and Schmidt, Ludwig and Yang, Diyi , booktitle =

    Yang, John and Lieret, Kilian and Jimenez, Carlos E. and Wettig, Alexander and Khandpur, Kabir and Zhang, Yanzhe and Hui, Binyuan and Press, Ofir and Schmidt, Ludwig and Yang, Diyi , booktitle =. 2025 , url =

    2025

  3. [3]

    2024 , url =

    Pang, Qi and Yuan, Yuanyuan and Wang, Shuai , booktitle =. 2024 , url =

    2024

  4. [4]

    2024 , url =

    Dong, Xiaoning and Xin, Peilin and Xu, Wei , journal =. 2024 , url =

    2024

  5. [5]

    2021 , eprint =

    Evaluating Large Language Models Trained on Code , author =. 2021 , eprint =

    2021

  6. [6]

    Program Synthesis with Large Language Models

    Program Synthesis with Large Language Models , author =. arXiv preprint arXiv:2108.07732 , year =

    work page Pith review arXiv

  7. [7]

    Is Your Code Generated by

    Liu, Jiawei and Xia, Chunqiu Steven and Wang, Yuyao and Zhang, Lingming , booktitle =. Is Your Code Generated by

  8. [8]

    Yu, Hao and Shen, Bo and Ran, Dezhi and Zhang, Jiaxin and Zhang, Qiang and others , journal =

Show all 28 references
  1. [9]

    Lai, Yuhang and Li, Chengxi and Wang, Yiming and Zhang, Tianyi and Zhong, Ruiqi and Zettlemoyer, Luke and Yih, Wen-tau and Fried, Daniel and Wang, Sida and Yu, Tao , booktitle =

  2. [10]

    2025 , url =

    Liu, Jinyu and Tan, Gang and Maeng, Kiwan , journal =. 2025 , url =

    2025

  3. [11]

    2021 , url =

    Knott, Brian and Venkataraman, Shobha and Hannun, Awni and Sengupta, Shubho and Ibrahim, Mark and van der Maaten, Laurens , booktitle =. 2021 , url =

    2021

  4. [12]

    2021 , url =

    Ziller, Alexander and Trask, Andrew and Lopardo, Antonio and others , journal =. 2021 , url =

    2021

  5. [13]

    2020 , url =

    Keller, Marcel , booktitle =. 2020 , url =

    2020

  6. [14]

    Proceedings of the ACM on Software Engineering , volume =

    Metamorphic Testing of Secure Multi-Party Computation (MPC) Compilers , author =. Proceedings of the ACM on Software Engineering , volume =. 2024 , publisher =. doi:10.1145/3639477.3640030 , url =

    2024 doi

  7. [15]

    2025 , url =

    Ruan, Wenqiang and Lin, Xin and Zhou, Ruisheng and Lin, Guopeng and Yu, Shui , booktitle =. 2025 , url =

    2025

  8. [16]

    2023 , url =

    Ma, Junming and Zheng, Yancheng and Feng, Jun and Zhao, Derun and Wu, Haoqi and Fang, Wenjing and Tan, Jin and Yu, Chaofan and Zhang, Benyu and Wang, Lei , booktitle =. 2023 , url =

    2023

  9. [17]

    Private Machine Learning in

    Dahl, Morten and Mancuso, Jason and Dupis, Yann and Decoste, Ben and Giraud, Morgan and Livingstone, Ian and Patriquin, Justin and Uhma, Gavin , year =. Private Machine Learning in. 1810.08130 , archivePrefix =

    arXiv

  10. [18]

    2019 IEEE Symposium on Security and Privacy (SP) , pages =

    SoK: General purpose compilers for secure multi-party computation , author =. 2019 IEEE Symposium on Security and Privacy (SP) , pages =. 2019 , organization =

    2019

  11. [19]

    ACM Transactions on Privacy and Security (TOPS) , volume=

    MOTION--a framework for mixed-protocol multi-party computation , author=. ACM Transactions on Privacy and Security (TOPS) , volume=. 2022 , publisher=

    2022

  12. [20]

    2017 IEEE Symposium on Security and Privacy (SP) , pages=

    SecureML: A System for Scalable Privacy-Preserving Machine Learning , author=. 2017 IEEE Symposium on Security and Privacy (SP) , pages=. 2017 , organization=

    2017

  13. [21]

    Proceedings on Privacy Enhancing Technologies , volume=

    SecureNN: 3-Party Secure Computation for Neural Network Training , author=. Proceedings on Privacy Enhancing Technologies , volume=. 2019 , publisher=

    2019

  14. [22]

    2019 IEEE Symposium on Security and Privacy (SP) , pages=

    Helen: Maliciously Secure Coopetitive Learning for Linear Models , author=. 2019 IEEE Symposium on Security and Privacy (SP) , pages=. 2019 , organization=

    2019

  15. [23]

    Science , volume=

    Realizing private and practical pharmacological collaboration , author=. Science , volume=. 2018 , publisher=

    2018

  16. [24]

    Proceedings on Privacy Enhancing Technologies , volume=

    Students and Taxes: a Privacy-Preserving Study Using Secure Computation , author=. Proceedings on Privacy Enhancing Technologies , volume=

  17. [25]

    Foreshadow: Extracting the Keys to the Intel

    Van Bulck, Jo and Minkin, Marina and Weisse, Ofir and Genkin, Daniel and Kasikci, Baris and Piessens, Frank and Silberstein, Mark and Wenisch, Thomas F and Yarom, Yuval and Strackx, Raoul , booktitle=. Foreshadow: Extracting the Keys to the Intel

  18. [26]

    2021 IEEE Symposium on Security and Privacy (SP) , pages=

    SoK: Fully Homomorphic Encryption Compilers , author=. 2021 IEEE Symposium on Security and Privacy (SP) , pages=. 2021 , organization=

    2021

  19. [27]

    2014 IEEE Symposium on Security and Privacy (SP) , pages=

    Wysteria: A Programming Language for Generic, Mixed-Mode Multiparty Computations , author=. 2014 IEEE Symposium on Security and Privacy (SP) , pages=. 2014 , organization=

    2014

  20. [28]

    Zahur, Samee and Evans, David , howpublished=. Obliv-. 2015 , url=

    2015

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.