{"id":"88ebc72d-666b-4bc1-891f-4923e5da6886","arxiv_id":"2507.11649","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A threshold-based ZKP protocol for federated evaluation is proposed, but the implemented circuit only checks the threshold, not the loss computation, leaving the central guarantee unsupported.","lead":"This paper presents a federated learning evaluation protocol that uses zero-knowledge proofs to let clients show their local loss is below a threshold without revealing the loss. The implemented prototype admits it does not verify the loss computation, so the core privacy and verifiability claims are not supported as stated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The implemented circuit only checks the threshold comparison, not the forward pass or loss computation (Section VII-F), so the proof does not bind the loss to the model or data and the Section VIII soundness claim is unsupported.","rationale":"The reader's weakest_assumption pinpoints the same gap I find load-bearing: Section VII-F directly contradicts Section IV-C and Section VIII. Everything in the paper's central claim depends on the circuit enforcing the loss computation. Without that enforcement, a proof attests only to a forgable witness value, not to the client's loss, so the soundness guarantee collapses. The HAR model inconsistency (Section VI describes a 1D CNN while Table I lists an MLP) is secondary and does not change the assessment. Because the paper itself admits the implemented circuit omits the forward pass, and no code or artifact is provided for independent verification, the reader's REJECT with high confidence is appropriate. My stress test does not change that verdict.","tokens_in":9407,"tokens_out":3829,"duration_ms":45308,"concrete_test":"Audit the actual Circom circuit (none is shipped). Check whether any R1CS constraint relates the public model hash H_Wg to the claimed loss L_i and to the data. Concretely, run the prototype with a client whose true loss is above T (e.g., an untrained model on MNIST with T=0.5) but set the witness loss to T/2 before witness generation. If the resulting Groth16 proof still verifies, the circuit does not enforce the loss computation, falsifying the Section VIII soundness claim. If no code is available, compare the circuit's R1CS constraint count with the number of multiply operations in the Table I CNN/MLP forward pass; a simple comparison circuit has O(1) constraints, whereas the claimed forward pass would require constraints proportional to the model's parameter and activation count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the central claim is that the ZKP relation R contains the full loss computation: given public model hash H_Wg, threshold T, nonce Nround, and witness (Wg, D_i, L_i), the circuit enforces L_i = average loss of the model over the client's data and L_i < T. Section IV-C claims exactly this, saying the circuit simulates the forward pass of the chosen model and computes the loss for a given input sample, and Algorithm 1 line 20 says the witness is computed via the CNN/MLP forward pass. However, Section VII-F explicitly states: 'The current circuit only verifies the threshold condition rather than the full forward pass and loss computation.' Section VI confirms the implementation used 'a simple loss threshold comparison circuit'. These statements are contradictory. If the circuit only verifies that a private witness value is below T, without constraining that witness to equal the model's loss on the client's data, then soundness is vacuous: a malicious client can claim an arbitrary loss below T regardless of the true local loss. Thus Section VIII's assertion that 'a client cannot generate a valid proof unless the computed loss meets the threshold condition' is invalid for the delivered artifact. The experimental numbers are therefore benchmarks of a trivial threshold-comparison circuit, not of verifiable federated evaluation. Privacy may still hold for an honest prover, but the verifiability guarantee that is the paper's main contribution is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ZKP-FedEval, a federated-learning evaluation protocol in which clients prove in zero knowledge that their local loss on a private dataset is below a public threshold T, without revealing the loss value. The protocol uses a Circom circuit instantiated with Groth16; the server verifies each proof and aggregates the binary outcomes. Experiments on MNIST and HAR report proof-generation time, verification time, proof/communication size, and validation rates. The central advertised property is soundness: a client cannot produce a valid proof unless its computed loss satisfies the threshold. I found that the paper itself contains a direct contradiction on this point: the implementation uses only a threshold-comparison circuit, not a circuit that recomputes the forward pass and loss, so the main verifiability claim is unsupported.","tokens_in":9515,"tokens_out":6208,"duration_ms":68256,"significance":"If the full circuit were implemented, the protocol would offer a useful and reasonably lightweight way to obtain a privacy-preserving, verifiable binary signal about client-side model quality in FL; the self-contained prototype and the discussion of trusted-setup trade-offs are helpful. However, the delivered artifact verifies only an unconstrained threshold comparison. As a result, the soundness guarantee that distinguishes this work from simple client self-reporting is not established, and the performance measurements cannot be attributed to the CNN/MLP evaluation workload described in the protocol. The contribution in its current form is a benchmark of a toy comparison circuit rather than of verifiable federated evaluation.","major_comments":[{"comment":"The protocol description and the implementation are directly contradictory. §IV-C states that the circuit 'simulates the forward pass of the chosen model' and computes the loss for an input sample, and Algorithm 1, line 20, says the witness is computed using the CNN/MLP forward pass and loss function. In contrast, §VI says the ZKP implementation used 'a simple loss threshold comparison circuit,' and §VII-F explicitly admits that 'the current circuit only verifies the threshold condition rather than the full forward pass and loss computation.' If the circuit relation only enforces L_i < T for an unconstrained private value L_i, then a malicious client can submit any value below T without computing the model's loss on its data. The proof then asserts nothing about the client's actual evaluation result, and the soundness guarantee stated in §VIII does not follow.","section":"§IV-C, §VI, §VII-F"},{"comment":"The experimental measurements are taken on the threshold-comparison circuit, not on a circuit that verifies the CNN/MLP forward pass and loss. The reported 0.4–0.5 s proof-generation time for MNIST and 0.12–0.13 s for HAR therefore do not characterize the cost of the protocol as described in §IV and Algorithm 1. A circuit that enforces the full forward pass and loss computation would have substantially more constraints and, in general, different proof-generation times; the paper provides no data for that circuit, so the scalability and overhead conclusions are unsupported.","section":"§VII (Table II, Figures 2–5)"},{"comment":"Even setting aside the threshold-only implementation, the protocol does not specify how the client's private dataset is bound to the proof. The public inputs listed in Algorithm 1 are H_Wg, T, and N_round; there is no commitment, hash, or authenticated identifier for the client's dataset D_i. Consequently, a proof that the witness data yields loss below T does not establish that the client's true local dataset satisfies the threshold, since the prover is free to choose any witness data satisfying the arithmetic constraints. A formal relation definition or a data-commitment mechanism is needed for the claimed verifiability.","section":"§VIII and §IV-C"},{"comment":"The reported proof sizes are inconsistent with the claim that they are Groth16 proofs. Figure 3 and Table II report average proof sizes of about 0.79 KiB for MNIST and 0.26 KiB for HAR, but a Groth16 proof consists of three group elements and has a size determined only by the curve, not by the circuit or dataset. The same circuit instantiation should yield the same proof size for both datasets (up to public-input encoding differences, which should be stated). The paper should specify what exactly is included in the 'proof size' (e.g., JSON encoding, public inputs, hashes) or correct the reported values.","section":"§VII-B, Figure 3, Table II"}],"minor_comments":[{"comment":"Section VI describes the MNIST model as a '2-layer CNN with channels evolving from 32 to 64,' while Table I lists Conv2D(1,16) and Conv2D(16,32) with an FC(32×7×7,10) layer; the two descriptions should be reconciled.","section":"§VI, Table I"},{"comment":"Section VI describes the HAR model as a '1D CNN with temporal convolutions (9 input channels → 64 → 128),' but Table I lists only fully connected layers; please clarify the actual architecture.","section":"§VI, Table I"},{"comment":"The paper states that both IID and basic non-IID data distributions were tested, but §VII reports only aggregate results; at minimum, a sentence explaining whether non-IID affected validation rates and timings should be added.","section":"§VI, §VII"},{"comment":"References [2] and [5] are self-citations with arXiv identifiers dated after the main text; please verify the dates/versions and ensure they are appropriate.","section":"References"}],"recommendation":"reject","confidential_remarks":"The contradiction between the protocol description and the implementation is central and self-admitted. I do not think this can be repaired by a local revision: either the circuit must be reimplemented to cover the full forward pass and loss, or the paper's claims must be reduced to a threshold-comparison demo, which would remove the main contribution. I recommend rejection, while noting that the limitation is disclosed in §VII-F."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline first: the paper's central claim is not supported by the delivered artifact. The design is a threshold-based zk-SNARK for federated evaluation: clients prove their local loss is below T without revealing it, using Groth16/Circom. That is a reasonable, though incremental, application of existing zkML/zkFL tools. The paper does something right: it is clearly written, the protocol steps are spelled out, and it explicitly admits in Section VII-F that the implemented circuit only verifies the threshold condition, not the full forward pass and loss computation. That admission is the paper's undoing, because the abstract and Section VIII assert soundness: 'a client cannot generate a valid proof unless the computed loss meets the threshold condition.' If the circuit does not recompute the loss from model and data, the witness value is unbound, and a malicious client can simply pick any value below T. Privacy may hold for an honest prover, but verifiability, the main contribution, does not.\n\nThe stress-test note is accurate. Algorithm 1 line 20 says the witness is computed via the CNN/MLP forward pass, but Section VI confirms the implementation used 'a simple loss threshold comparison circuit.' These are contradictory. The benchmarks in Figures 2-5 and Table II therefore measure the toy circuit, not verifiable federated evaluation. This is the load-bearing flaw.\n\nMinor issues: an internal inconsistency about the HAR model (Section VI describes a 1D CNN, but Table I and the text elsewhere call it an MLP), and no code or data for replication. The self-citations are not load-bearing and are not a problem.\n\nWho this is for: readers interested in system designs for verifiable FL evaluation. They will get a clean protocol sketch, a clear statement of the engineering gap, and honest benchmarks of a threshold circuit. As a claim of a working verifiable evaluation system, it falls short.\n\nRecommendation: I would accept this for peer review, because the gap between claim and artifact is exactly what a referee should catch, and the fix is principled: either implement the full forward-pass circuit or rescope the paper's language to match the toy implementation. As submitted, the central soundness claim does not hold.","headline":"The protocol design is sensible, but the implemented circuit does not verify the loss, so the paper's central soundness claim is unsupported.","tokens_in":10218,"tokens_out":2699,"would_cite":false,"duration_ms":31069,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZKP-FedEval claims that federated evaluation can become verifiable and privacy-preserving by having each client generate a zero-knowledge proof that its local loss is below a threshold, without revealing the loss value.","keywords":["Federated Learning","Zero-Knowledge Proofs","zk-SNARKs","Groth16","Privacy-Preserving Evaluation","Verifiable Computation","Loss Threshold","Circom"],"falsifier":"Inspect the published Circom circuit and attempt to build a witness that sets the final loss register below $T$ while skipping the convolution and pooling constraints; if the Groth16 verifier accepts such a proof (or if those constraints are absent), the claimed binding between proof and true loss is refuted.","tokens_in":9055,"feed_emoji":"🔐","tokens_out":6540,"duration_ms":68568,"temperature":0.7,"pith_summary":"Federated learning distributes training but often relies on clients sharing evaluation metrics, which can leak information about their private data. This paper proposes ZKP-FedEval, a protocol in which each client computes its local loss and, if the loss is below a server-set threshold, generates a zero-knowledge proof of that fact. The server verifies the proof and aggregates only the binary outcome, so the exact loss is never disclosed yet the server can check the threshold claim. The authors implement the proof system with the Circom language and the Groth16 zk-SNARK, and measure constant proof size and linear verification scaling on MNIST and HAR datasets. The intended consequence is a federated evaluation phase that is both private and sound, at the cost of a single bit of information per client.","feed_headline":"Zero-knowledge proofs let clients prove low loss without revealing it","feed_subtitle":"Servers check that each client's loss is below a threshold while clients reveal only one bit of information.","key_machinery":"The load-bearing object is the threshold-verification arithmetic circuit written in Circom and instantiated with Groth16. It is meant to encode the relation that the hashed weights match the public input and that the fixed-point loss computed over the client's data is below $T$, with the round nonce mixed in to prevent replay. The circuit turns the private witness (model weights and data) into a public, succinct proof; Groth16's constant-size proofs and fast verification are what make the protocol scalable. The caveat carried by the implementation is that the actual circuit, per the paper's Section VII-F, checks only the threshold condition and not the full forward-pass computation.","core_discovery":"ZKP-FedEval's central claim is that the evaluation phase of federated learning can be secured by replacing raw loss reports with zero-knowledge proofs of a threshold predicate. Concretely, the server publishes the global model $W_g$, a loss threshold $T$, and a round nonce $N_{\\mathrm{round}}$; each client computes its local loss $L_i$ and, when $L_i < T$, produces a Groth16 proof $\\pi_i$ for a Circom circuit whose public inputs are the hash of the model, the threshold, and the nonce. The server verifies $\\pi_i$ and counts the client as valid, learning only that the loss is below threshold. The paper claims this arrangement is zero-knowledge, sound against clients who would falsely claim compliance, and resistant to replay through the nonce, and reports experiments showing per-proof verification time and proof size are essentially constant while total server time grows linearly with the number of clients. The authors state that the circuit simulates the forward pass and loss computation, though they also acknowledge in the limitations that the implemented circuit only enforces the threshold condition rather than the full computation.","pith_inferences":["If the circuit is extended to verify the full forward pass, the honest-protocol measurements will likely grow with model depth; the reported 0.4–0.5 s for a small CNN suggests that deep architectures would need circuit-level optimizations or transparent-setup SNARKs with different trade-offs.","The single-bit protocol could be generalized to a small set of threshold bands, trading a few bits of leakage for a more informative evaluation signal while still hiding the exact loss.","The protocol's privacy guarantee covers the evaluation metric only; data poisoning and gradient leakage remain outside its scope, so it should be combined with, not substituted for, robust aggregation."],"forward_implications":["Federated evaluation can be reduced to a single bit per client: whether local loss is below the threshold, which weakens metric-based inference attacks.","Server-side verification cost per proof is constant and total cost scales linearly with the number of valid proofs, making the protocol predictable for larger FL deployments.","A threshold acts as a quality gate: clients whose loss is too high are filtered out without the server knowing their exact loss.","The trade-off between inclusiveness and signal is controlled by $T$; for a well-trained model a low threshold yields 100% valid proofs and reveals little information, while a poorly matched model separates good and bad clients."],"supporting_citations":[{"why":"Motivates the protocol by showing that shared evaluation metrics can leak information about client data.","marker":"[3]"},{"why":"Establishes the unintended feature leakage threat that the threshold-based approach aims to mitigate.","marker":"[4]"},{"why":"Provides the secure aggregation baseline that the paper contrasts with, since it does not verify computation integrity.","marker":"[6]"},{"why":"Supplies the foundational definitions of zero-knowledge proofs that the protocol's guarantees rely on.","marker":"[7]"},{"why":"Provides the Circom circuit description language used to implement the threshold-verification relation.","marker":"[8]"},{"why":"Supplies the Groth16 zk-SNARK scheme that gives constant-size proofs and fast verification.","marker":"[9]"},{"why":"Contributes the MNIST dataset used for the CNN evaluation experiments.","marker":"[10]"},{"why":"Represents prior verifiable federated learning work that this protocol extends or contrasts with.","marker":"[14]"},{"why":"Defines the FedAvg algorithm whose evaluation phase is being secured.","marker":"[17]"},{"why":"Contributes the UCI HAR dataset used for the MLP evaluation experiments.","marker":"[19]"}],"fun_headline_variants":["Zero-knowledge proofs keep federated evaluation loss private","Prove low loss without revealing it: ZKP for FL evaluation","Federated evaluation with ZKP: verifiable and private","ZKP-FedEval: verify client loss without exposing it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The soundness guarantee holds only if the zero-knowledge circuit re-derives the loss from the model and data; the paper's own experiments admit that the implemented circuit checks only the threshold condition, not the full loss computation, so this binding premise is currently unverified.","fun_headline_variants_meta":{"raw":{"variants":["Zero-knowledge proofs keep federated evaluation loss private","Prove low loss without revealing it: ZKP for FL evaluation","Federated evaluation with ZKP: verifiable and private","ZKP-FedEval: verify client loss without exposing it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001144,"raw_usage":{"total_tokens":4747,"prompt_tokens":949,"completion_tokens":3798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":3727}},"tokens_in":565,"tokens_out":3798,"duration_ms":32903,"temperature":1.0,"reasoning_tokens":3727,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:04:36.070642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the published Circom circuit and attempt to build a witness that sets the final loss register below $T$ while skipping the convolution and pooling constraints; if the Groth16 verifier accepts such a proof (or if those constraints are absent), the claimed binding between proof and true loss is refuted.","supporting_citations":[{"cited_title":"Exploiting Unintended Feature Leakage in Collaborative Learning,","cited_arxiv_id":null,"evidence_quote":"Establishes the unintended feature leakage threat that the threshold-based approach aims to mitigate."},{"cited_title":"The knowledge complexity of interactive proof-systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the foundational definitions of zero-knowledge proofs that the protocol's guarantees rely on."},{"cited_title":"Circom: A circuit description language for building zero-knowledge applications,","cited_arxiv_id":null,"evidence_quote":"Provides the Circom circuit description language used to implement the threshold-verification relation."},{"cited_title":"On the Size of Pairing-Based Non-interactive Arguments,","cited_arxiv_id":null,"evidence_quote":"Supplies the Groth16 zk-SNARK scheme that gives constant-size proofs and fast verification."},{"cited_title":"Gradient-based learning applied to document recognition,","cited_arxiv_id":null,"evidence_quote":"Contributes the MNIST dataset used for the CNN evaluation experiments."},{"cited_title":"VerifyNet: Secure and Verifiable Federated Learning,","cited_arxiv_id":null,"evidence_quote":"Represents prior verifiable federated learning work that this protocol extends or contrasts with."},{"cited_title":"A public domain dataset for human activity recognition using smartphones","cited_arxiv_id":null,"evidence_quote":"Contributes the UCI HAR dataset used for the MLP evaluation experiments."}],"review_version":1}