{"id":"c7444fb1-d2d8-4156-8059-27291da1d4c5","arxiv_id":"2501.00363","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SPDZCoder combines hand-crafted expert rules with LLM prompting to translate Python to MP-SPDZ, reporting 85.9% pass@1 on its new SPDZEval benchmark, far above the best direct-LLM baseline.","lead":"The paper presents SPDZCoder, a rule-based system that uses large language models to translate ordinary Python functions into MP-SPDZ code for secure multi-party computation. It also introduces a new benchmark of 313 translation tasks and reports large accuracy gains over direct LLM translation, though the evaluation uses a test-guided repair step.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported pass@1 includes test-case-guided repair (max_feedback=3), so the 85.94% headline is not one-shot; over half the margin over the best baseline may come from oracle access, and the qualitative claim needs the w/orepair comparison plus a repair-enabled baseline.","rationale":"The reader's weakest assumption identifies the exact load-bearing concern. The paper claims that SPDZCoder makes Python-to-MP-SPDZ translation practical, measured by pass@1. For that claim to hold, pass@1 must be comparable across methods and must mean one generated program per problem evaluated against unseen tests. Neither holds: the repair stage executes the evaluation test cases during generation, and the strongest baseline (API-Doc) is given no such feedback. This is not merely a naming issue: Table 6 attributes 12.78 pass@1 points to the repair stage, and the headline margin over the best baseline is 22.36 points. The paper's own w/orepair result (73.16 vs. 63.58) is the cleanest support for the qualitative claim and should be the headline comparison; it would be substantially strengthened by giving the same repair loop to baselines. I do not think the paper should be rejected: the w/orepair ablation, the consistency across backbone models (Table 5), and the refactoring ablation all suggest a real effect. But the reported numbers need correction, the benchmark and prompts must be released, and the evaluation protocol must be explicit. This is exactly the reader's CONDITIONAL conclusion, so no verdict change is needed.","tokens_in":23932,"tokens_out":6347,"duration_ms":65727,"concrete_test":"Run the OpenAI-o1 API-Doc baseline through the identical repair stage used by SPDZCoder (FixCompilationRuntimeError and FixFunctionalityError prompts, max_feedback=3) against the same SPDZEval test cases, and report pass@1/pass@2 after repair alongside first-attempt (no-execution-feedback) pass@1 for all methods. If repaired o1 reaches or exceeds 85.94%, the claim that expert rules, rather than oracle-guided repair, drive the advantage fails; if it stays near 63.58%, the concern is partially mitigated but the headline metric still needs relabeling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1's headline comparison (85.94% vs. 63.58%) is not a fair pass@1 comparison. Stage 3 (§3.2) repairs generated code by executing it against SPDZEval's test cases, with max_feedback=3 (§4.2), so 'pass@1' means 'one pipeline run that may use the evaluation test oracle up to four attempts,' not 'one generated sample evaluated against unseen tests.' The API-Doc baselines, including the best OpenAI-o1, get no such repair loop, while SPDZCoder gains 12.78 absolute points from repair (Table 6, 73.16→85.94). That gain is more than half of its 22.36-point headline margin. UniTrans and InterTrans also use test execution, but the best baseline is API-Doc, so the headline comparison still stacks an oracle-guided pipeline against direct generation. The w/orepair row (73.16 vs. 63.58) shows SPDZCoder still leads without repair, so the qualitative claim may survive; however, the quantitative central claim as stated is inflated by unequal oracle access, and the term pass@1 is misleading. The paper does not report whether the best baseline would also improve if given the same repair prompts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPDZCoder, a rule-based framework for translating Python functions into MP-SPDZ (secure MPC) code. It collects expert knowledge about semantic-expressing differences between Python and MP-SPDZ, distills it into refactoring rules (for high-level differences such as data-obliviousness and missing libraries) and a generation rule with in-context demonstrations (for low-level naming and syntax differences), and applies them in a three-stage pipeline: refactor to a Canonical Form Python (CFP), generate MP-SPDZ, and repair via execution feedback. The authors introduce SPDZEval, a manually constructed benchmark of 313 Python/MP-SPDZ function pairs in six splits, and report pass@1/pass@2 of 85.94%/92.01% versus 63.58%/76.36% for the best baseline (OpenAI-o1 with API documentation). Ablations show that removing the repair stage drops pass@1 to 73.16% and removing refactoring drops it to 33.23%.","tokens_in":24132,"tokens_out":6914,"duration_ms":62509,"significance":"If the reported results are validated, the work is a meaningful step toward automating privacy-computing code generation: the framework is fully prompt-based, requires no fine-tuning data, and the ablation study gives a clear view of the contribution of each stage. The construction of a function-level Python-to-MP-SPDZ benchmark is a useful resource for the community, and the paper is honest in discussing the repair stage as optional and in reporting the without-repair ablation. However, the headline pass@1 metric is not a standard single-shot pass@1 because the repair stage uses the evaluation test cases as an oracle, which limits the strength of the central quantitative claim as currently written.","major_comments":[{"comment":"The headline comparison in Section 5.1 (SPDZCoder 85.94% vs. best baseline 63.58% pass@1) is not a fair single-attempt pass@1 comparison. The repair stage (Section 3.2) executes the generated code against the benchmark's test cases and iterates up to max_feedback=3 times (Section 4.2), so 'pass@1' actually means one pipeline run that may use the evaluation test oracle up to four attempts. The API-Doc baselines, including the best OpenAI-o1, are given no such repair loop, while Table 6 shows that repair contributes 12.78 absolute points (73.16% to 85.94%), more than half of the 22.36-point margin. The paper should either report the w/orepair ablation (73.16% vs. 63.58%) as the primary pass@1 result, run the baselines with the same repair mechanism, or explicitly rename the metric (e.g., pass@1-with-repair) and discuss the oracle access.","section":"Section 3.2, Section 4.2, Table 4, Table 6, Section 5.1"},{"comment":"The definition of pass@k states that k code samples are generated per problem and the problem is solved if any sample passes the unit tests. In the SPDZCoder pipeline, however, the repair stage can produce multiple outputs from a single initial sample, all informed by execution feedback from the same test cases used for scoring. The paper should clarify how pass@1 and pass@2 are computed from the repetition and max_feedback hyperparameters, and whether the test cases used for repair and for final evaluation are identical. If they are identical, the evaluation is not a standard held-out pass@k and the reported numbers should be labeled accordingly.","section":"Section 4.4.2"},{"comment":"UniTrans and InterTrans also use test-execution feedback, but the best baseline in the headline comparison (API-Doc with OpenAI-o1) does not. Since the paper does not report API-Doc baselines augmented with the same repair prompts, the reader cannot tell how much of the 22.36-point margin comes from the repair advantage rather than from the rule-based refactoring and generation stages. The w/orepair row (73.16%) suggests the qualitative conclusion may still hold, but the quantitative central claim as written is not supported by a like-for-like comparison.","section":"Section 4.4.1, Table 4"}],"minor_comments":[{"comment":"'secrete int' and 'secrete fixed float' should be 'secret int' and 'secret fixed float'.","section":"Section 2.2"},{"comment":"'incoporate' (Section 3.2) and 'repitition' (Section 4.2) are typos for 'incorporate' and 'repetition'.","section":"Section 3.2, Section 4.2"},{"comment":"The caption spells the benchmark name as 'SDPZEval'; it should be 'SPDZEval'.","section":"Table 3 caption"},{"comment":"In the API-Doc DeepSeek-V2.5 row, the pass@2 array column shows '1.4', which appears to be a typo for a three-digit value (likely 81.4); please verify.","section":"Table 4"},{"comment":"The manuscript does not indicate whether the benchmark and the implementation will be made publicly available; a repository link or an explicit statement would improve reproducibility.","section":"General"},{"comment":"The generation configuration reports temperature but not top-p or seed values, although the Internal Validity paragraph mentions these hyperparameters; concrete values should be provided for reproducibility.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The evaluation-fairness issue is the central obstacle to publication. If the authors can show that the best baselines do not close the gap when given the same repair loop, or if the without-repair comparison is made primary, the paper's contribution is solid and within the journal's scope. I would also ask the editor to ensure the authors address the benchmark release and the apparent tabular typo during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This is the first Python-to-MP-SPDZ translation benchmark and the first rule-based LLM pipeline aimed at that task. It is also the case that the headline 85.94% pass@1 is not one-shot: the repair stage executes generated code against the benchmark's own test cases and retries up to three times. The numbers are not what pass@1 normally means.\n\nThe good parts are real. SPDZEval, with 313 function pairs across six difficulty classes, fills an actual gap; nothing like it existed. The three-stage pipeline—refactoring to a canonical Python form, in-context generation with pattern-matched demonstrations, then repair—is clearly described and the ablation is genuinely informative. Without refactoring, correctness collapses to 33%, which shows the expert rules are doing substantial work rather than the LLM coasting. And even without the repair stage, SPDZCoder beats the strongest baseline (73.16% vs 63.58% pass@1), so the qualitative claim probably survives, just with a smaller margin than advertised.\n\nNow the soft spots, in proportion. The pass@1 labeling is the main one. The repair stage sees the evaluation test oracle during generation, and the baselines—including the best one, OpenAI-o1 with API docs—get no such loop. Over half of the headline margin over the best baseline comes from repair. That does not kill the contribution, but it must be reported differently: either declare the primary result as \"pipeline with test-guided repair\" or, better, report the without-repair score as the honest pass@1 and run a repair-enabled baseline for fairness. Second, neither SPDZEval nor the rule/prompt artifacts are released, so independent verification and extension are impossible right now. Third, the benchmark was constructed by one student with one engineer reviewing; there is no variance or stability analysis, and results are single-run. That makes the precise percentages fragile even if the ranking is plausible. Minor: Table 4 contains a stray \"1.4\" entry that looks like a typesetting error.\n\nWho is this for? People working on LLM-based code translation for MPC or domain-specific DSLs will find a useful benchmark, a sensible decomposition of the problem, and a repeatable methodology. It deserves serious refereeing, but the evaluation protocol has to be fixed before the central quantitative claim can be taken at face value.\n\nMy recommendation: send it to peer review with major revisions. Require a clear without-repair headline, a repair-enabled baseline comparison, and release of the benchmark and prompts.","headline":"A useful first benchmark and a plausible rule-based translation pipeline, but the headline pass@1 is inflated by test-oracle-guided repair and no artifacts are released.","tokens_in":24737,"tokens_out":1926,"would_cite":true,"duration_ms":19999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rule-based refactoring lifts privacy-code translation to 85.9% pass@1.","keywords":["privacy computing","multi-party computation","MP-SPDZ","code translation","large language models","data-obliviousness","code generation benchmark","in-context learning"],"falsifier":"Re-run SPDZCoder on SPDZEval with the test cases withheld from the repair stage (or with repair disabled) and compare against baselines run under the same condition; if the corrected pass@1 approaches the 63.58% baseline level, or if the w/orepair 73.16% advantage disappears, the central claim of superiority collapses. A cleaner test would use a held-out set of test cases generated after the benchmark was built, so the repair oracle cannot see them.","tokens_in":23668,"feed_emoji":"🔐","tokens_out":7852,"duration_ms":76587,"temperature":0.7,"pith_summary":"The paper tries to establish that ordinary Python functions can be translated into MP-SPDZ, a framework for multi-party privacy-preserving computation, by combining expert-derived transformation rules with an LLM, without retraining. The obstacle it identifies is 'semantic-expressing differences' at two levels: low-level mismatches in names and syntax, and high-level mismatches such as the data-obliviousness requirement and the absence of many Python library functions in MP-SPDZ. SPDZCoder handles high-level mismatches by refactoring Python into an intermediate canonical form, handles low-level mismatches with in-context demonstration rules during a single generation step, and then repairs errors by feeding execution messages back to the LLM. On a new 313-function benchmark, the paper reports 85.94% pass@1 and 92.01% pass@2, versus 63.58% and 76.36% for the best baseline. The point of caring is that privacy-computing code is hard to write by hand, so an automated translation path would lower the barrier to using MPC frameworks.","feed_headline":"Expert rules take privacy-code translation to 85.9% pass@1","feed_subtitle":"A new 313-function MP-SPDZ benchmark; the best baseline reaches 63.58%.","key_machinery":"Canonical Form Python (CFP): an intermediate Python representation in which high-level semantic-expressing differences have been eliminated, so that every construct has a direct counterpart in MP-SPDZ and all secret-dependent control flow is data-oblivious. Ten refactoring rules produce CFP by implementing missing functions from a small set of primitives (exp, ln, sqrt, and inverse sqrt), rewriting branches, loops, break/continue, and data-structure operations into oblivious basic forms, and replacing non-oblivious algorithms such as quick sort with oblivious alternatives such as radix sort. The generation rule then converts CFP to MP-SPDZ in one prompt, with pattern-matched demonstrations handling low-level name and syntax mismatches (for example, float to a fixed-point secret type, and an ordinary exponential call to a fixed-point library expression). The repair stage carries the remaining burden by treating execution error messages as feedback for up to three additional LLM passes.","core_discovery":"The central claim is that the main obstacle to Python-to-MP-SPDZ translation is not LLM capability but the absence of explicit knowledge about how the two languages express the same semantics. SPDZCoder encodes that knowledge as ten refactoring rules for high-level differences and one generation rule with in-context demonstrations for low-level differences. It first rewrites Python into Canonical Form Python, in which every remaining statement has a direct or near-direct MP-SPDZ counterpart and all control flow is data-oblivious; then it translates that form to MP-SPDZ in a single LLM pass, applies a self-reflection check, and optionally repairs the output by feeding compilation, runtime, and functionality error messages back to the LLM up to three times. Evaluated on SPDZEval, 313 manually constructed function pairs in six pattern classes, the pipeline reaches 85.94% pass@1 and 92.01% pass@2 with the paper's default backbone, versus 63.58% and 76.36% for the best baseline; without the repair stage the pass@1 falls to 73.16%, and without refactoring to 33.23%, which the paper reads as evidence that the rule-based decomposition, not just test-case feedback, is doing the work.","pith_inferences":["Because the repair stage executes the benchmark's own test cases during generation, the 85.94% pass@1 is an optimistic estimate for settings where test cases are hidden; the 73.16% w/orepair figure is closer to a fair 'test cases unseen' estimate, and the comparison would be cleaner if baselines received the same oracle.","The CFP intermediate could outlive its Python origin: a natural-language-to-MP-SPDZ synthesizer that emits CFP first would inherit the same obliviousness guarantees, a step the paper lists as future work but does not claim.","If the expert-rules cost is amortized, the framework suggests a general recipe for any low-resource programming-language target: elicit semantic-expressing differences from domain experts, cache them as transformation rules, and leave only the residual translation to an LLM.","A stress test beyond the six pattern classes—for example, dynamic memory access or recursive functions, where oblivious counterparts are known to be hard or nonexistent—would reveal how much of the advantage is tied to the benchmark's chosen pattern coverage."],"forward_implications":["A translation approach for a low-resource target language does not need large parallel corpora: expert rules and in-context learning replace fine-tuning data.","The refactoring stage is the main contributor: removing it drops pass@1 from 85.94% to 33.23%, a far larger loss than removing repair, which leaves 73.16%.","The rule-based gain transfers across LLM backbones: swapping in a weaker backbone still outperforms the best baseline under the default backbone.","The obliviousness-related rules are reusable if the source language changes; only the syntactic and library rules need to be re-derived.","SPDZEval gives the community a 313-pair benchmark with test cases for future comparisons of Python-to-MP-SPDZ translation."],"supporting_citations":[{"why":"Defines MP-SPDZ, the target framework whose data types, library functions, and obliviousness constraints create the translation difficulty the paper addresses.","marker":"[33]"},{"why":"Supplies a leading LLM-based code translation baseline that SPDZCoder is compared against and must outperform.","marker":"[57]"},{"why":"Supplies the other leading LLM translation baseline, using intermediate languages, which the paper contrasts with its single-target rule-based approach.","marker":"[42]"},{"why":"Introduces the HumanEval-style functional correctness evaluation that the pass@1 and pass@2 scores build on.","marker":"[11]"},{"why":"Is the cited origin of the pass@k metric used to report correctness in the experiments.","marker":"[34]"},{"why":"Supplies oblivious radix sort, the replacement algorithm used by a refactoring rule for non-oblivious sorting.","marker":"[25]"},{"why":"Motivates the self-reflection component used to mitigate hallucinated MP-SPDZ function names and APIs.","marker":"[30]"}],"fun_headline_variants":["Expert rules boost Python-to-MP-SPDZ code gen to 85.9% pass@1","No fine-tuning needed: LLM plus expert rules hits 85.9% on privacy code","Rule-based SPDZCoder beats LLM-only translation: 85.9% vs 63.6% pass@1","Expert rules fill the gap: Python to MP-SPDZ hits 85.9% without fine-tuning","SPDZCoder: Expert rules translate Python to MP-SPDZ at 85.9% pass@1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline correctness numbers assume it is acceptable for the repair stage to run the generated code against the benchmark's own test cases during generation and retry up to three times while still reporting the result as pass@1.","fun_headline_variants_meta":{"raw":{"variants":["Expert rules boost Python-to-MP-SPDZ code gen to 85.9% pass@1","No fine-tuning needed: LLM plus expert rules hits 85.9% on privacy code","Rule-based SPDZCoder beats LLM-only translation: 85.9% vs 63.6% pass@1","Expert rules fill the gap: Python to MP-SPDZ hits 85.9% without fine-tuning","SPDZCoder: Expert rules translate Python to MP-SPDZ at 85.9% pass@1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3393,"prompt_tokens":1135,"completion_tokens":2258,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":751,"completion_tokens_details":{"reasoning_tokens":2123}},"tokens_in":751,"tokens_out":2258,"duration_ms":15580,"temperature":1.0,"reasoning_tokens":2123,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:52:21.019882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SPDZCoder on SPDZEval with the test cases withheld from the repair stage (or with repair disabled) and compare against baselines run under the same condition; if the corrected pass@1 approaches the 63.58% baseline level, or if the w/orepair 73.16% advantage disappears, the central claim of superiority collapses. A cleaner test would use a held-out set of test cases generated after the benchmark was built, so the repair oracle cannot see them.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the cited origin of the pass@k metric used to report correctness in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MP-SPDZ, the target framework whose data types, library functions, and obliviousness constraints create the translation difficulty the paper addresses."},{"cited_title":"Cogo, and Bram Adams","cited_arxiv_id":null,"evidence_quote":"Supplies the other leading LLM translation baseline, using intermediate languages, which the paper contrasts with its single-target rule-based approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies oblivious radix sort, the replacement algorithm used by a refactoring rule for non-oblivious sorting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the self-reflection component used to mitigate hallucinated MP-SPDZ function names and APIs."}],"review_version":1}