{"id":"1ab6e265-fe91-46c1-bb7c-4100900eea6f","arxiv_id":"1908.02802","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Closest decision-boundary (or 'flip') points are computed for a CIFAR-10 network, showing Taylor approximations underestimate distances and that robustness measures should use these exact points.","lead":"The authors compute the closest point on a trained neural network's decision boundary by solving an optimization problem, and use these points to test common assumptions about decision boundaries. The paper matters because it shows a concrete way to measure a model's true vulnerability to adversarial changes, while also exposing where popular approximations mislead.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Closest flip points are not certified global minima; Section 2 concedes the problem is nonconvex, so reported distances and vulnerability claims may be overestimates.","rationale":"The reader's weakest assumption—that the computed flip points are global minimizers of the nonconvex optimization problem—is exactly the load-bearing concern. Section 2 explicitly concedes that global optimality cannot be guaranteed, and the reported 5% of images where other solvers find closer points is direct evidence that some reported distances are overestimates. Because the paper's comparisons to Taylor approximations and adversarial-attack methods rely on the distance to the 'closest' flip point, any suboptimality can change the numbers and the conclusions. The paper is valuable as a heuristic tool, and the authors are candid about the limitation, but the abstract and several claims use 'exact' and 'closest' without qualification. A conditional verdict is appropriate: the work should be accepted only with claims tightened to 'locally closest' or 'closest found,' and with a reproducible artifact and an optimality-gap analysis. A secondary logical issue is Section 6, where evidence of path-connectedness is used to conclude star-shapedness; this does not affect the main claim but should be corrected. The concrete test proposed would provide evidence about whether the global-minimization concern actually matters in practice, at least on a simplified version of the problem.","tokens_in":7627,"tokens_out":4074,"duration_ms":43937,"concrete_test":"On a subset of 10–20 test images, reduce the feature space to a small number of wavelet coefficients (e.g., 20–50) and retrain a network of the same architecture, so that a certified global optimization method (e.g., spatial branch-and-bound with interval arithmetic, or a solver such as BARON/SCIP) can solve the flip-point problem to provable global optimality. Compare the certified global minimum distances against those obtained by the paper's multi-start/local optimization heuristic. If the certified distance is smaller for any image, the paper's 'closest flip point' and 'weakest vulnerability' conclusions are unsupported; if the distances match, the heuristic is validated on those reduced cases and the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that flip points are the closest decision-boundary points and reveal the model's weakest adversarial vulnerability—requires global optimality of a nonconvex optimization problem. Section 2 states: 'our optimization problem is nonconvex, so we cannot be sure that optimization algorithms will find the global minimizer.' The authors appeal to agreement among multiple solvers as evidence, but immediately note that in fewer than 5% of images the interior point algorithms find closer flip points. This is an explicit acknowledgment that the reported distances are not always minimal; in those cases, the 'closest flip point' is only a local solution, and the reported distance is an upper bound on the true distance. All downstream quantitative results—the underestimation factors for Taylor approximations (Figures 5–6), the angle comparisons (Figure 8), and the 'weakest vulnerability' adversarial examples (Figures 9–12)—are computed relative to possibly suboptimal flip points. No optimality certificate is provided: no interval bounds, no branch-and-bound, no verified second-order sufficient conditions, and no quantification of the optimality gap. The abstract's language of computing 'exact points' on decision boundaries and the word 'closest' throughout are therefore stronger than what the numerical evidence supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes computing 'flip points'—points on the decision boundaries of trained neural networks—by solving a nonconvex optimization problem that minimizes the distance from a given input to a point where the softmax outputs of two classes are equal. The authors use these flip points to compare against first-order Taylor approximations of boundary distance (Section 5), to study the geometry and connectivity of decision regions along straight lines (Sections 4 and 6), and to produce adversarial examples that they argue reveal the 'weakest vulnerability' of a model (Section 7). All experiments use a 12-layer fully connected network with erf activations trained on 200 Daubechies-1 wavelet coefficients of ship/plane images from CIFAR-10, achieving 84.05% test accuracy.","tokens_in":7872,"tokens_out":2989,"duration_ms":30933,"significance":"If the computed flip points are indeed the closest points on decision boundaries, the paper offers a practical tool for interpreting trained networks, auditing approximation methods, and generating minimal adversarial perturbations. The empirical comparisons with Taylor-series distance estimates (Figures 5–8) and with loss-minimization adversarial attacks (Figures 9–12) are valuable and clearly demonstrate that common approximations can be misleading. The paper is also honest in stating the nonconvexity of its optimization problem. However, the central quantitative claims depend on unverified global optimality, the star-shapedness conclusion is not supported by the presented evidence, and the asserted generalization to pixel-based networks is without experimental support. These issues currently limit the contribution to a useful methodology demonstration rather than a fully established set of geometric and robustness conclusions.","major_comments":[{"comment":"The paper's central claim is that it computes 'exact points' on decision boundaries and 'closest' flip points, but the optimization problem is nonconvex and the authors explicitly state that they 'cannot be sure that optimization algorithms will find the global minimizer.' They also report that in fewer than 5% of images, interior-point algorithms found closer flip points than their preferred solution. This means the reported distances are only upper bounds on the true distance in general, and all downstream quantitative results—the underestimation factors in Figures 5–6, the angle distributions in Figure 8, and the 'weakest vulnerability' comparisons in Figures 9–12—are computed relative to possibly suboptimal flip points. The text should either provide optimality certificates (e.g., interval bounds, branch-and-bound, or verified local optimality with a gap estimate) or systematically qualify 'closest' as 'locally closest among computed solutions' and discuss how the <5% gap affects the conclusions.","section":"Section 2"},{"comment":"The claim that the 'ship' region is star-shaped does not follow from the evidence presented. The authors show that for 89% of pairs of correctly classified ships, the straight line segment between them stays in the ship class, and that there exist paths (using multiple lines) connecting every pair. Star-shapedness requires the existence of a single center point that sees every other point in the region via a straight line; pairwise line connectivity is a weaker property that implies path-connectedness but not star-shapedness. The statement 'This indicates that the ship region is star-shaped' should be revised to a claim supported by the data, such as 'the region is path-connected' or 'most pairs are line-connected,' or new evidence for a common center should be provided.","section":"Section 6"},{"comment":"The sentence 'These observations would still hold for networks trained on the pixels rather than wavelet coefficients' is an unsupported generalization. All experiments in the paper use a 200-dimensional wavelet-coefficient feature space, and the geometry of decision boundaries in that space may differ from the geometry in the original pixel space. Since the adversarial-robustness conclusions in this section are intended to apply broadly, the authors should either provide comparable experiments on pixel-based networks or remove this claim and restrict the conclusions to the feature representation actually tested.","section":"Section 7"}],"minor_comments":[{"comment":"The terms 'exact points' and 'closest flip point' are used throughout the abstract and introduction without qualification, despite the nonconvexity caveat in Section 2. Consider consistently using 'computed flip point' or 'locally closest flip point' so that the presentation matches the technical caveats.","section":"Abstract and Section 1"},{"comment":"The statement that using a smaller number of wavelet coefficients 'makes us more likely to find the global solution of our non-convex optimization problem' is presented without supporting evidence. It would be helpful to frame this as a heuristic rationale or provide diagnostic results on smaller problems.","section":"Section 3"},{"comment":"There is a typo in the caption: 'siginificantly' should be 'significantly.'","section":"Section 5, Figure 6"},{"comment":"The discussion of the distance-constrained loss-minimization problem is imprecise: the problem with a distance constraint of 0.5 is always feasible (the original image is feasible), but it may have no solution that changes the label. The text should clarify that the issue is the absence of an adversarial example within the ball, not infeasibility of the optimization problem itself.","section":"Section 7"},{"comment":"The sentence 'This means that we choose the distance between the discretization points small enough such that the output of network can be considered to change linearly between any consecutive points' is a heuristic interpretation of Lipschitz continuity; a Lipschitz bound controls the magnitude of change, not the linearity of the function between points. Consider rewording to avoid overstating what the bound provides.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the authors' prior work (Yousefzadeh and O'Leary 2019) for the flip-point optimization formulation, and it cites that work. This is appropriate, but the present manuscript should more clearly delineate what is new beyond that earlier method. The main technical risk is the unverified global optimality, which is acknowledged in Section 2; if the authors cannot provide certificates, they should substantially soften the 'exact' and 'closest' claims throughout. The star-shapedness claim in Section 6 is a logical gap that should be fixed by either changing the claim or adding a center-based test. The paper is otherwise clearly written and the empirical comparisons are informative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time if you care about interpretability or adversarial robustness. The new content here is the systematic comparison: first-order Taylor distance estimates are off by factors of 2 to 20, the Taylor direction can point outside the feasible image domain, and loss-minimization adversarial methods can miss much closer boundary points or become infeasible. Those findings are concrete and well illustrated. The flip-point method itself comes from their earlier paper, but the comparisons and the geometric observations are new.\n\nThe paper does several things right. It states plainly in Section 2 that the optimization is nonconvex and that they cannot guarantee global optimality, and they hedge by testing multiple solvers. The Lipschitz-based argument for choosing discretization along paths is sensible. The empirical plots are readable and the critique of distance-constrained adversarial attacks is fair, with a nice example where the constraint makes the problem infeasible yet a much closer flip point exists.\n\nNow the soft spots, in proportion. The biggest one is the gap between the abstract's 'exact points' / 'closest flip point' language and what is actually computed. The authors admit that in fewer than 5% of images the interior-point solvers find closer flip points. That means their reported distances are sometimes upper bounds, and every downstream quantity — the underestimation factors, angle distributions, and the 'weakest vulnerability' adversarial examples — is computed relative to possibly suboptimal points. This does not destroy the paper's comparative claims, because the Taylor approximation underestimates by such large factors that a small optimality gap would not change the qualitative conclusion, but it does undermine the word 'exact' and the strength of the vulnerability claim. An optimality certificate, even for a subset of cases, would have helped.\n\nSecond, the star-shaped conclusion does not follow. They show that lines between many pairs of ships stay in the ship region, which indicates connectedness or line-connectedness, but star-shapedness requires a common point from which every other point is visible. They did not test that. The language in Section 6 overreaches.\n\nThird, the claim that the observations would still hold for pixel-based networks is unsupported. The experiments use 200 wavelet coefficients, not pixels, and the authors themselves note that the reduced feature space makes global optimization more likely. Extrapolating to pixel input is an assertion, not a result.\n\nNo code or data is shipped, which is a real drawback for a methods paper. Still, the core empirical comparisons are valuable and the authors are honest about the main theoretical limitation. I'd bring this to a reading group and would send it to review, with the expectation that the authors tighten the 'exact' language, fix the star-shaped claim, and ideally release code. This is a solid, useful paper that needs moderate revision, not a desk reject.","headline":"Useful empirical check on boundary-distance approximations, but the 'exact/closest' claims outrun what the nonconvex numerics can certify.","tokens_in":8329,"tokens_out":1567,"would_cite":true,"duration_ms":19558,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26"],"pacs":[],"model":"deepseek-v4-flash","headline":"For a trained neural network with nonlinear activations, exact points on decision boundaries can be computed by solving a nonconvex optimization problem, and these flip points show that Taylor-based distances and loss-based adversarial…","keywords":["decision boundaries","flip points","adversarial examples","nonconvex optimization","neural network interpretability","Lipschitz continuity","Taylor approximation","robustness"],"falsifier":"Take a small trained network whose decision boundary can be computed analytically, or enumerate all boundary points on a fine grid in a low-dimensional input space, then run the paper's optimization routine and check whether it returns the true closest boundary point. If it returns a flip point that is farther than a grid-found boundary point, then the reported distances, directions, and vulnerability claims are not guaranteed for that case.","tokens_in":7475,"feed_emoji":"🎯","tokens_out":7512,"duration_ms":80508,"temperature":0.7,"pith_summary":"This paper argues that exact points on the decision boundaries of trained neural networks, called flip points, can be computed by solving a nonconvex optimization problem, even though earlier work treated this as intractable. It demonstrates on a 12-layer network trained on ship and plane images that these flip points give distances and directions to the decision boundary that are far more reliable than first-order Taylor approximations, which can underestimate distance by large factors. It also shows that common adversarial-example searches that minimize loss under a distance constraint can miss the closest adversarial change or become infeasible, whereas the closest flip point directly reveals the model's weakest vulnerability. The broader point is that interpretation, generalization, and robustness studies should verify their simplifying assumptions about decision boundaries rather than rely on linear or derivative-based approximations.","feed_headline":"Exact decision-boundary distances expose neural net blind spots","feed_subtitle":"A nonconvex optimization finds the closest input that flips a model's decision, beating Taylor and adversarial-loss approximations.","key_machinery":"The central object is the flip point: a point on the decision boundary where the model's top-two softmax outputs are equal. The carrying machinery is the optimization problem $\\min_z \\|z-x\\|_2$ subject to $f_i(z)=f_j(z)$ and $f_i(z)\\ge f_k(z)$ for all other classes, where $f$ is the softmax output vector and $x$ is the original input. A good starting point for the optimization is the data point itself, and the authors cross-check solutions with several nonlinear optimization algorithms. A second piece of machinery is Lipschitz continuity of the network output, which tells them how finely to discretize straight paths between images so that boundary crossings and the shape of decision regions can be observed reliably.","core_discovery":"The paper's central claim is that for a trained network with nonlinear activations, one can compute exact points on its decision boundaries by minimizing the distance from a given input to a boundary point subject to the conditions that the softmax outputs of the two competing classes are equal and no other class has a higher output. Applied to a 12-layer feedforward network using erf activations and 200 wavelet coefficients, the resulting flip points yield actual boundary distances and directions. These exact quantities reveal that first-order Taylor distance estimates systematically underestimate the true distance, sometimes by a factor of 20 or more, and that the Taylor direction can point far from the closest boundary point. The closest flip point also provides a more direct measure of adversarial vulnerability than minimizing an adversarial loss under a fixed distance constraint, which may find a point much farther away or have no feasible solution. The authors explicitly note in Section 2 that their optimization problem is nonconvex and that they cannot guarantee finding the global minimizer, although consistency across several solvers and the fact that their points are closer than Taylor estimates support the computed results.","pith_inferences":["If the global-minimizer assumption holds, true flip-point distances could replace Taylor-based margin estimates in generalization-gap prediction, potentially yielding tighter and more reliable margin distributions than current approximations.","The same optimization could be turned into a training procedure: the authors note that adding flip points as training data improves accuracy, but they do not design or test a systematic boundary-moving training algorithm.","The observed near-linear growth of the angle between the Taylor direction and the closest flip point as distance increases suggests a predictable degradation of local linearity; testing this relationship on other architectures would show whether it is a general geometric law or an artifact of this model.","Because the paper says its method has been implemented for multiple classes and discrete features, a natural extension is to check whether the distance, direction, and vulnerability conclusions persist beyond the binary continuous-input setting studied here."],"forward_implications":["Distance-to-boundary estimates based on first-order Taylor expansions can be checked against flip points; in the authors' tests they understate the true distance by factors from about 2 to more than 20.","Adversarial-example searches that minimize loss under a distance constraint can be replaced or supplemented by searching for the closest flip point, which yields a closer adversarial change and reveals the weakest vulnerability.","The decision region for a class can be studied directly: in their model the ship region is connected and star-shaped, so hyperplane or locally linear approximations to the boundary are inadequate.","Computing flip points costs about the same as common adversarial-attack methods, making direct boundary investigation a practical tool rather than a theoretical one.","A single distance constraint tuned for one image does not transfer across a dataset, because distances to the closest flip point vary widely between images."],"supporting_citations":[{"why":"Introduces the flip-point formulation that this paper solves and extends to adversarial and geometry analysis.","marker":"Yousefzadeh and O’Leary, 2019"},{"why":"Supplies the interior-point optimization solver used to compute closest flip points.","marker":"Wächter and Biegler, 2006"},{"why":"Supplies the NLopt optimization package used to cross-check the computed flip points.","marker":"Johnson, 2014"},{"why":"Provides the first-order Taylor distance approximation that the paper's flip-point distances are compared against and shown to improve on.","marker":"Elsayed et al., 2018"},{"why":"Uses that Taylor approximation to predict generalization gaps, making it a target of the paper's margin critique.","marker":"Jiang et al., 2019"},{"why":"Represents the loss-minimization-plus-distance-constraint adversarial search that closest flip points are shown to improve on.","marker":"Ilyas et al., 2019"},{"why":"Provides another instance of adversarial loss minimization under distance constraints, used as a comparison baseline.","marker":"Tsipras et al., 2019"},{"why":"Supplies the earlier geometric claim of connected, highly curved decision regions that the flip-point plots confirm.","marker":"Fawzi et al., 2018"},{"why":"Approximates margin using softmax output differences, which the paper demonstrates can be misleading.","marker":"Neyshabur et al., 2017"}],"fun_headline_variants":["Exact boundary points expose neural net blind spots","Neural nets' true decision boundaries computed exactly","Taylor estimates can miss true neural net boundary distance","Closest flip point reveals weakest adversarial attack","Nonconvex optimization finds exact neural decision boundaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole study rests on the assumption that the optimization solver truly finds the closest flip point, and the authors state in Section 2 that they cannot prove this because the problem is nonconvex.","fun_headline_variants_meta":{"raw":{"variants":["Exact boundary points expose neural net blind spots","Neural nets' true decision boundaries computed exactly","Taylor estimates can miss true neural net boundary distance","Closest flip point reveals weakest adversarial attack","Nonconvex optimization finds exact neural decision boundaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1450,"prompt_tokens":940,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":440}},"tokens_in":556,"tokens_out":510,"duration_ms":6232,"temperature":1.0,"reasoning_tokens":440,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:32:48.827728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small trained network whose decision boundary can be computed analytically, or enumerate all boundary points on a fine grid in a low-dimensional input space, then run the paper's optimization routine and check whether it returns the true closest boundary point. If it returns a flip point that is farther than a grid-found boundary point, then the reported distances, directions, and vulnerability claims are not guaranteed for that case.","supporting_citations":[{"cited_title":"On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming","cited_arxiv_id":null,"evidence_quote":"Supplies the interior-point optimization solver used to compute closest flip points."},{"cited_title":"Large margin deep networks for classification","cited_arxiv_id":null,"evidence_quote":"Provides the first-order Taylor distance approximation that the paper's flip-point distances are compared against and shown to improve on."},{"cited_title":"Predicting the generalization gap in deep networks with margin distributions","cited_arxiv_id":null,"evidence_quote":"Uses that Taylor approximation to predict generalization gaps, making it a target of the paper's margin critique."},{"cited_title":"Robustness may be at odds with accuracy","cited_arxiv_id":null,"evidence_quote":"Provides another instance of adversarial loss minimization under distance constraints, used as a comparison baseline."},{"cited_title":"Empirical study of the topology and geometry of deep networks","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier geometric claim of connected, highly curved decision regions that the flip-point plots confirm."},{"cited_title":"Exploring generalization in deep learning","cited_arxiv_id":null,"evidence_quote":"Approximates margin using softmax output differences, which the paper demonstrates can be misleading."}],"review_version":1}