{"id":"7412ed18-d86f-4d07-ac9f-bc4a95dd297f","arxiv_id":"1909.01838","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Given only prediction access, an adversary can exactly recover the weights of a two-layer ReLU network, and semi-supervised learning makes accuracy extraction far more query-efficient.","lead":"This paper shows how an attacker can steal a machine learning model by sending it questions. It introduces a method that exactly recovers the weights of a two-layer neural network using only the answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 6.5's sign-recovery step defines z by A(0)z=0, but all ReLUs are at zero only when A(0)z+B(0)=0; as printed, the three queries cannot recover the sign bit.","rationale":"The paper's central claim is that the direct extraction attack achieves functionally-equivalent extraction of two-layer ReLU networks given exact logits. The reader's weakest assumption concerned the oracle model (exact 64-bit logits, exactly two-layer ReLU, linearly independent rows of A(0)). Those are external preconditions and are largely acknowledged in the paper. My reading found a more internal problem: the Global Sign Recovery phase, as written in Section 6.5, is not mathematically consistent. The text says to choose z in the nullspace of A(0), but the condition that makes every hidden unit sit at its critical point is A(0)z+B(0)=0. With a nonzero bias vector, the printed construction leaves the network at an arbitrary point of its piecewise-linear function, so the three queries OL(z), OL(z+v_i), and OL(z-v_i) do not determine the sign bit. This is not a question of numerical precision or adversarial capabilities; it is a gap in the algorithm specification itself. The attack may well be correct with the affine condition, and the experiments suggest an implementation exists, but the paper does not provide code and the written description is incomplete. A synthetic test that runs the sign-recovery step exactly as printed would settle whether the concern is a typo or a substantive flaw. I therefore keep the reader's CONDITIONAL verdict: the central claim is plausible and well-supported experimentally, but the manuscript should correct or clarify the sign-recovery construction and ideally release the implementation. The disagreement with the reader is on which assumption is least secure: I see the printed algorithm's internal consistency as more immediately load-bearing than the threat-model assumptions, which are already stated as limitations.","tokens_in":23227,"tokens_out":13643,"duration_ms":143180,"concrete_test":"Build a synthetic two-layer ReLU oracle with d=6, h=2, random nonzero A(0) and B(0), and run the printed Section 6.5 procedure: compute z with A(0)z=0, compute v_i with A(0)v_i=e_i, and use the equality tests on OL(z), OL(z±v_i) to recover signs. If, as the equations indicate, the recovered row signs are wrong for any neuron with B(0)_i≠0, repeat with z chosen as any solution of A(0)z+B(0)=0 and compare; this isolates whether the text is missing a bias term or the attack has a deeper flaw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The functionally-equivalent attack depends on Global Sign Recovery (Section 6.5) to fix the last unknown sign of each first-layer row. The text says to take z in the null space of A(0), i.e., A(0)z=0, and then asserts that at z, z+v_i, z-v_i 'all hidden units are 0 except for ReLU_i.' But a hidden unit's input is A(0)_k x + B(0)_k, so A(0)z=0 makes the inputs B(0)_k, not 0. Unless every bias vanishes, the ReLUs at z are generally active or saturated at nonzero values; the subsequent comparisons OL(z+v_i)=OL(z) versus OL(z-v_i)=OL(z) measure whether the ith unit crosses the origin through a shifted baseline, not whether the recovered row sign points in the correct direction. The correct condition is A(0)z+B(0)=0 (an affine solution, solvable because h<d), and then v_i should satisfy A(0)_k·v_i=δ_{ki}. Since this phase is necessary for the extraction, the algorithm as written is not a complete specification of the claimed attack; at minimum an implementer following Section 6.5 literally will fail sign recovery on networks with nonzero biases. The paper's own experiments do not resolve this because no code is provided.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies model extraction with two objectives: accuracy and fidelity. It proposes a learning-based extraction attack that uses the victim model as a label oracle and combines semi-supervised techniques (rotation loss and MixMatch) to improve query efficiency, validating it on the WSL ImageNet model, SVHN, and CIFAR-10. It then argues from theory and experiments that learning-based approaches cannot achieve perfect fidelity because of training nondeterminism. To reach the high-fidelity limit, the paper presents a functionally-equivalent extraction attack for two-layer ReLU networks that recovers the first-layer weights directly from logit queries, followed by algebraic recovery of the second layer. The empirical section reports near-perfect test-set fidelity on MNIST and CIFAR-10, 100% adversarial-example transferability, and a hybrid strategy that repairs numerical errors with a small amount of learning.","tokens_in":23532,"tokens_out":9364,"duration_ms":94762,"significance":"If the direct extraction attack is correct, it is a substantial contribution: it removes the gradient-oracle requirement of Milli et al. and the side-channel requirement of Batina et al., and it achieves query complexity O(dh), which is optimal up to constants for a model with dh first-layer parameters. The hardness statements in Theorems 1 and 2 are useful formal bounds on worst-case extraction and equivalence testing. The large-scale learning-based experiments on the WSL model also provide credible evidence that accuracy extraction can be improved with semi-supervised methods. However, the presented algorithm is not a complete specification as written: the global sign-recovery step in Section 6.5 contains a load-bearing error, and the empirical evaluation measures test-set and adversarial fidelity rather than functional equivalence on all inputs. These issues should be resolved before the headline claim is accepted as stated.","major_comments":[{"comment":"The global sign-recovery procedure is incorrect as printed. The text says to choose z with A(0)z=0 and then asserts that at z, z+v_i, and z-v_i 'all hidden units are 0 except for ReLU_i.' A hidden unit's pre-activation is A(0)_k x + B(0)_k, so A(0)z=0 only gives pre-activations equal to B(0)_k; when the biases are nonzero, the ReLUs at z are generally not all at zero. The correct condition is to solve the affine system A(0)z+B(0)=0, which is solvable because h<d, and then choose v_i such that A(0)_k v_i = delta_{ki}. As written, an implementer following Section 6.5 will fail sign recovery on any network with nonzero biases, and the networks trained in the paper all have biases. Since this step is necessary for the claimed exact extraction, the algorithm is not a complete specification. The text should also specify how the unknown global sign of each recovered row is handled when constructing v_i, and should base the sign decision on a comparison of OL(z+v_i) and OL(z-v_i) rather than on the unstated assumption that all other units are exactly at zero.","section":"Section 6.5"},{"comment":"The absolute-value recovery equations mix vectors and scalars in a way that is not implementable as written. The quantity d^2 OL / d e_j^2 is a K-dimensional logit vector, because A(1)_i is a row in R^K and the formula contains the vector A(1)_i. The text then divides 'quantities' |A(0)_1i A(1)_i| and |A(0)_2i A(1)_i| to obtain a scalar ratio |A(0)_1i / A(0)_2i|. The paper should specify how the K-dimensional logit differences are reduced to scalars, for example by using a fixed nonzero output component or the norm of the logit difference. Without this specification, the ratio operation is undefined and an implementer cannot reconstruct the row weights from the second-derivative measurements.","section":"Section 6.4.1"},{"comment":"The empirical evaluation does not directly verify functional equivalence, which is agreement on all inputs. Section 6.7 reports test-set fidelity and adversarial-example transferability, and for the 100,000-parameter model the test fidelity is 99.98%; the weight-recovery precision is reported as an average number of bits. To support the claim of a 'practically functionally-equivalent extraction attack,' the paper should report agreement on out-of-distribution or uniformly random inputs (and, ideally, the maximum logit discrepancy) for each model size, or explicitly qualify the result as near-functional equivalence subject to numerical error. Releasing the implementation would make this check reproducible, especially given the subtlety of Algorithm 1 and the sign-recovery step.","section":"Section 6.7"}],"minor_comments":[{"comment":"The notation 'viA(0)=ei' appears to have the multiplication order reversed; it should be A(0)v_i=e_i (or the transpose should be defined consistently).","section":"Section 6.5"},{"comment":"Lines 4 and 5 use variables a and b before they are defined; these should be t1 and t2. The displayed formulas for x and y-hat are also garbled by line breaks and should be rewritten with all variables defined.","section":"Algorithm 1"},{"comment":"The objective written as 'minimizing E_x || f_theta(x) - W1 ReLU(Ahat(0)x + Bhat(0) + W0) + W2 ||' uses W0..2, W1, and W2 without defining their shapes or initialization; this should be clarified.","section":"Section 7.2"},{"comment":"The caption says the table reports the transferability rate of adversarial examples, while the text says the hybrid strategy increases fidelity and reports a fidelity value of 99.75%; the table should either include the fidelity numbers or the caption should be corrected.","section":"Table 8"},{"comment":"The assumption that the input dimension exceeds the hidden dimension by at least 100 is stated informally; the paper should state explicitly that all reported experiments satisfy this assumption and discuss what happens as h approaches d.","section":"Section 6.1"},{"comment":"The statement that gradient estimation fails roughly 1 in 10,000 times is not accompanied by a confidence interval or per-model variance; because the paper bases a design decision on this rate, reporting the spread across the five runs would be helpful.","section":"Section 6.7"}],"recommendation":"major_revision","confidential_remarks":"The Section 6.5 issue is real and central, but it is also local and fixable: the sign-recovery step should use an affine zero of the hidden pre-activations and a properly defined v_i. The paper is a strong fit for the venue, and I would not reject on the assumption that the implementation, which is not released, happened to handle this correctly. The authors should also be encouraged to release code so the exactness claim can be checked independently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is the paper that finally breaks the gradient and side-channel requirement: it recovers the weights of a two-layer ReLU network exactly from double-precision logit queries alone, and the reported 100% test-set fidelity for small networks is a measured outcome, not a fitted one. Second, the printed sign-recovery step in Section 6.5 has a real bug that the authors need to fix before anyone can reproduce the attack.\n\nThe taxonomy (accuracy vs. fidelity) is useful and already widely cited. The learning-based half is also genuinely good: MixMatch and rotation loss applied to extraction, with a 16x query-efficiency improvement over prior learning attacks and a real deployment against the WSL Instagram-pretrained model. The hardness results in Theorems 1 and 2 are rigorous and in the appendix. The query complexity argument (O(dh), matching the information-theoretic lower bound up to constants) is clean. Credit where due: the paper does not oversell; Section 8 explicitly lists the defenses that break the attack, and the limitations section for learning-based extraction (nondeterminism prevents perfect fidelity) is honest and empirically supported.\n\nNow the soft spots. The stress-test note is right, and this is my main concern. Section 6.5 says to pick z in the nullspace of \\hat{A}^{(0)} and then query OL(z), OL(z+v_i), OL(z-v_i), claiming \"all hidden units are 0 except for ReLU_i.\" But a hidden unit's input is A^{(0)}_k x + B^{(0)}_k, so nullspace only gives A z=0; the pre-activations are the biases. Unless the biases are all zero, at z the ReLUs are in arbitrary states, and the difference OL(z+v_i) - OL(z) is contaminated by the baseline. The correct condition is to solve A^{(0)} z + B^{(0)} = 0, which is solvable because h<d. As written, an implementer following the text will fail sign recovery on any network with nonzero biases. This is a fixable flaw, but it is a flaw in the paper's central algorithm, and the absence of released code makes it worse.\n\nThe other assumptions are strong but stated clearly: exact 64-bit logits, exactly two-layer ReLU, linearly independent rows. The paper acknowledges that probability-truncating or noise-adding defenses break the attack. So the result is a proof-of-concept with narrow scope, not a universal break.\n\nBottom line: this deserves a serious referee and will likely be accepted after revision. I would send it out, and ask the authors to correct Section 6.5, ideally with a formal statement of the affine nullspace step, and to ship code so the 100% fidelity numbers can be checked. If I were working in model extraction, I would cite it for the taxonomy and the learning attack; the functionally-equivalent claim I'd cite with the sign-recovery caveat.","headline":"First logit-only functionally equivalent extraction attack for two-layer ReLU nets; strong taxonomy and learning attack, but Section 6.5 has a real sign-recovery bug in the printed algorithm.","tokens_in":24032,"tokens_out":5000,"would_cite":true,"duration_ms":45221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A query-only attack recovers a two-layer ReLU network's exact weights from ordinary logit access, giving the first practical functionally-equivalent extraction.","keywords":["model extraction","functionally equivalent extraction","ReLU networks","piecewise linear analysis","query-based attack","model stealing","fidelity vs accuracy","adversarial example transferability"],"falsifier":"Run the attack against an oracle that returns logits rounded to single precision or as probabilities with any truncation: Algorithm 1's equality check between the predicted value at the candidate kink and the fresh queried value will fail, so no critical point is accepted and weight recovery cannot proceed. A second concrete test is to train a two-layer network whose first-layer weight matrix contains two identical rows, violating linear independence, and observe that the per-neuron isolation step can no longer attribute a kink to a single ReLU.","tokens_in":23067,"feed_emoji":"🧠","tokens_out":11183,"duration_ms":102681,"temperature":0.7,"pith_summary":"This paper asks whether a deployed neural network can be stolen from its own answers, and separates model extraction into two goals: accuracy (matching task performance) and fidelity (matching the victim on every input), which the authors argue are in conflict. For accuracy, they show that training on oracle-supplied labels beats training on ground-truth labels, with a 16× query-efficiency gain from semi-supervised learning, demonstrated against a 193-million-parameter production image classifier. For fidelity, they argue that learning-based attacks are fundamentally capped near 93 percent agreement because training non-determinism prevents perfectly retraining the same function, even with identical seeds and data. Their central claim is that a two-layer ReLU network can instead be copied exactly, weights recovered up to the inherent scale-equivalence class, using only ordinary logit queries, by locating each ReLU's activation boundary, measuring weight ratios across that boundary, and solving the last layer algebraically. If correct, the attack turns black-box logit access into functionally identical white-box access, making a whole class of remote classifiers copyable end to end.","feed_headline":"Queries alone can steal a two-layer neural net's exact weights","feed_subtitle":"With only logit answers, the attacker reproduces the victim's predictions on every input.","key_machinery":"The load-bearing object is the piecewise-linear geometry of a two-layer ReLU network: within each linear region every hidden unit's pre-activation has a fixed sign, and the boundaries between adjacent regions are the critical points where exactly one unit's input $A^{(0)}_i x + B^{(0)}_i$ is zero. The argument is carried by Algorithm 1, a 2-linearity test that, given a line segment containing exactly one kink, fits the two adjacent linear segments and computes their intersection in $O(1)$ queries, then verifies the candidate by comparing the predicted function value there against one fresh query, which is also what rejects segments containing more than one kink. Around each located kink the second derivative, the jump in the gradient across the boundary, equals $\\pm A^{(0)}_{ji} A^{(1)}_i$, so taking ratios of these jumps across input directions recovers each row of $A^{(0)}$ up to sign and scale.","core_discovery":"The paper's central claim is that functionally-equivalent extraction of a neural network is achievable in practice: given an oracle that returns exact 64-bit logits for arbitrary inputs, an adversary can recover all weights of a two-layer ReLU network up to the inherent equivalence class (positive scaling and permutation symmetries), with no training and no side channel. The recovery works because ReLU networks are piecewise linear, so the logit function's kinks are exactly the hyperplanes where individual hidden units sit at zero; by sweeping one-dimensional lines through input space and locating each kink with a constant-query 2-linearity test, the attacker measures the gradient jump across each boundary to obtain ratios of the incoming weights, resolves signs through directional second derivatives and a null-space trick, and then solves the final layer by least squares. In experiments on trained MNIST and CIFAR-10 models the extracted network matches the victim's prediction on 100 percent of test inputs for networks up to 50,000 parameters and on 99.98 percent at 100,000 parameters, with first-layer weights recovered to roughly 23 bits of precision and adversarial examples transferring to the victim 100 percent of the time.","pith_inferences":["Editorial inference: a quantitative bits-per-query model of this attack is testable, degrading logit precision from 64-bit to 32-bit to 16-bit should raise the fraction of weight rows recovered with large error, and measuring that curve would give defenders a precise trade-off instead of the all-or-nothing verdict the paper reports.","Editorial inference: the layer-attribution difficulty the paper flags suggests a staircase extension, once the first layer is recovered, its activations on the already-collected queries are known, so the second layer's kinks become visible as kinks in the output with known inputs to that layer; whether this peels further layers or drowns in accumulated finite-difference error is the paper's main o","Editorial inference: the accuracy results indicate that oracle labels leak information beyond ground-truth labels, so re-running the ImageNet recipe with different distillation temperatures would separate dark-knowledge transfer from mere label replication, testing how much of the stolen accuracy gain survives when the victim's outputs are flattened."],"forward_implications":["Any two-layer ReLU classifier served through a logits-returning API can be copied outright: an attacker obtains a model that agrees with the victim on every input up to floating-point precision, using a number of queries proportional to the parameter count, $O(dh)$.","Downstream white-box attacks that need the victim's internal weights, such as adversarial-example crafting, membership inference on confidence scores, and inspection of overlearned internal representations, become available to a black-box adversary after extraction.","Learning-based extraction is bounded: even with the exact training set, hyperparameters, and random seeds, training non-determinism caps fidelity near 93 percent in the paper's Fashion-MNIST experiments, so exact copies must come from direct weight recovery rather than retraining.","The hybrid strategy shows the direct attack's bias-propagation errors are repairable: fixing the extracted first layer and fine-tuning a single bias parameter with gradient descent raises worst-case fidelity from 80 percent to 99.75 percent and preserves 100 percent adversarial-example transferability.","Because the attack is query-optimal up to a constant factor, defenses must limit the information returned per query; the paper shows that returning only top-k scores, labels, or perturbed outputs breaks the functionally-equivalent attack."],"supporting_citations":[{"why":"The gradient-query extraction attack this work extends; contributed the critical-point search analysis and the O(h log h) bound whose gradient queries the new attack simulates with finite differences.","marker":"[19]"},{"why":"The only prior practical direct extraction, via a power side channel; serves as the precision baseline the paper compares against (average weight error 0.0025 versus 0.0000009).","marker":"[25]"},{"why":"Prior learning-based extraction and decision-tree path-finding; provides the fully-supervised extraction baseline and the prediction-API stealing framing the accuracy attack improves on.","marker":"[11]"},{"why":"The WSL oracle, a 193-million-parameter classifier trained on 1 billion proprietary images; the production-scale victim used for the large-scale accuracy-extraction experiments.","marker":"[28]"},{"why":"The statistical-query lower bound for learning deep random networks; cited as the reason learning-based fidelity extraction is sample-inefficient in the worst case.","marker":"[27]"},{"why":"MixMatch, the semi-supervised method whose guessed-label training gives the 16x query-efficiency gain on SVHN and CIFAR-10.","marker":"[38]"},{"why":"Distillation; the temperature-scaled soft-target training that the fully-supervised accuracy-extraction attack uses to train on oracle probabilities.","marker":"[24]"}],"fun_headline_variants":["Query alone yields exact weights of two-layer ReLU network","Exact clone of a two-layer network from logits only","First practical exact model extraction from black-box queries","Steal a network's weights exactly with pure oracle access","Zero-training functional-equivalent extraction from queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack collapses if the victim does not return exact 64-bit logits for arbitrary queries, if the model is not exactly a two-layer ReLU network, or if the rows of the first weight matrix are linearly dependent; the paper itself notes that information-limiting defenses (labels only, top-k scores, or perturbed outputs) break the functionally-equivalent attack.","fun_headline_variants_meta":{"raw":{"variants":["Query alone yields exact weights of two-layer ReLU network","Exact clone of a two-layer network from logits only","First practical exact model extraction from black-box queries","Steal a network's weights exactly with pure oracle access","Zero-training functional-equivalent extraction from queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1568,"prompt_tokens":979,"completion_tokens":589,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":595,"tokens_out":589,"duration_ms":6392,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:20:35.294851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the attack against an oracle that returns logits rounded to single precision or as probabilities with any truncation: Algorithm 1's equality check between the predicted value at the candidate kink and the fresh queried value will fail, so no critical point is accepted and weight recovery cannot proceed. A second concrete test is to train a two-layer network whose first-layer weight matrix contains two identical rows, violating linear independence, and observe that the per-neuron isolation step can no longer attribute a kink to a single ReLU.","supporting_citations":[{"cited_title":"Stealing machine learning models via pre- diction apis,","cited_arxiv_id":null,"evidence_quote":"Prior learning-based extraction and decision-tree path-finding; provides the fully-supervised extraction baseline and the prediction-API stealing framing the accuracy attack improves on."},{"cited_title":"Exploring the limits of weakly supervised pretraining,","cited_arxiv_id":null,"evidence_quote":"The WSL oracle, a 193-million-parameter classifier trained on 1 billion proprietary images; the production-scale victim used for the large-scale accuracy-extraction experiments."}],"review_version":1}