{"id":"4859e25c-2093-415f-af76-68f4ec5e2179","arxiv_id":"2509.18025","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"An expository survey showing that nearly all common neural network activation and loss functions are tame (definable in o-minimal structures), which unlocks convergence guarantees for stochastic subgradient descent on nonsmooth nonconvex objectives.","lead":"This survey explains 'tame geometry', a branch of mathematics that studies well-behaved sets and functions, and shows that the pieces used to build neural networks mostly belong to it, which lets mathematicians prove that stochastic gradient descent converges on nonsmooth, nonconvex training problems. A generalist might read it because it offers a rare rigorous framework that covers realistic deep learning objects instead of simplified stand-ins.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'realistic' half rests on a non-itemized 89% count and ignores non-activation components (e.g. sinusoidal positional encodings); whether a common o-minimal structure covers real architectures is unverified.","rationale":"I read the paper in good faith as an expository survey whose mathematical content is imported from externally validated sources. The derivation from o-minimality to Projection formula 4.5, Chain rule 4.7, and Proposition 4.8 is internally coherent and faithfully reflects Davis et al. and Bolte et al. I find no load-bearing error in the proof exposition. The genuine soft spot is the premise that deep-learning practice is mostly definable in a common o-minimal structure. This premise carries the 'realistic' half of the thesis, and it is supported only by a rough, self-admittedly approximate count over activation functions, with 6% explicitly non-tame and 5% unclear, while non-activation components using unrestricted sine/cosine are not surveyed at all. The reader's conditional verdict already identifies this same weakest assumption and appropriately conditions acceptance on a checkable itemization. My stress-test does not move the verdict: it strengthens the reader's concern by adding the non-activation-component gap, but the required remedy is the same, so the verdict remains CONDITIONAL rather than being overturned.","tokens_in":34689,"tokens_out":8202,"duration_ms":88234,"concrete_test":"Compile from Kunc and Kléma (2024) all 400 activation functions and, for each, provide an explicit definability certificate in a single common structure: either an expression built from semialgebraic operations, exp, restricted analytic functions, erf, or arctan in R_Pfaff, or a cited non-definability result. Separately audit standard architecture blocks—Transformer sinusoidal/RoPE positional encodings, SIREN, layer norm, softmax attention, embedding lookup—for definability in that same structure. If the audited fraction is materially below the claimed ~89%, or if a commonly used block (e.g. sinusoid) is non-tame, narrow the realism claims in Sections 1.2 and 5 accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conditional guarantee, Proposition 4.8, applies only when the entire training objective is definable in a single o-minimal structure. The paper's claim that this covers 'nearly all' deep-learning practice rests on Remark 3.19: about 89% of 400 surveyed activation functions are declared definable 'by direct inspection', about 5% are admitted unclear, about 6% use unrestricted sine/cosine and are non-tame, and 'a few liberties' are taken with the count. This is not a checkable inventory. More importantly, the audit covers only activation/loss functions, not other standard deep-learning components: sinusoidal positional encodings in Transformers, rotary position embeddings, and periodic activation networks such as SIREN use unrestricted sin/cos, which are non-tame by Non-example 3.4, and their definability is not addressed. Remark 3.15 also requires all components to be definable in one common structure; composition across incompatible o-minimal structures can fail to be tame. Remark 3.19 only establishes that the Table 1 functions share R_Pfaff, not that the surveyed 89% share a single structure. Thus the 'realistic' half of the thesis, and hence the transfer of Proposition 4.8 to current architectures, is not established by the evidence supplied. The mathematical core of Section 4 is not in question; the scope claim is.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that o-minimal/tame geometry provides a unified and 'realistic' framework for deep learning, and that it is 'prolific' in yielding convergence guarantees. After introducing semialgebraic geometry and o-minimal structures (R_alg, R_exp, R_an, R_an,exp, R_Pfaff), the paper presents tame properties such as dimension, stratification, and definable choice. The central mathematical section is Section 4, which exposits the convergence of the stochastic subgradient method (SSM) on definable locally Lipschitz functions. The route is: Proposition 4.4 gives a continuous-time descent identity via the Projection formula 4.5 and Chain rule 4.7; Proposition 4.8 then asserts that every limit point of the bounded, square-summable-but-non-summable SSM iteration is Clarke critical, using a Weak Sard + Descent Lyapunov verification attributed to Davis et al. The manuscript closes with extensions to small-batch SSM, automatic differentiation, and conservative fields. The mathematical exposition is largely a careful condensation of known results, and the paper is transparent about relying on external benchmarks.","tokens_in":34845,"tokens_out":4813,"duration_ms":45422,"significance":"If accepted as a survey/expository note, the paper is a useful and largely reliable bridge between tame geometry, nonsmooth optimization, and deep learning. The conditional mathematical result—Proposition 4.8, that definable locally Lipschitz objectives are well-behaved for SSM—is correctly presented as a theorem of Davis et al., and the derivation of the descent identity through Projection formula 4.5, Chain rule 4.7, and Proposition 4.4 is coherent and faithful to the cited sources. The paper deserves credit for its explicit caveats about composition across structures (Remark 3.15), its honest admission of the heuristic nature of the 400-activation-function count (Remark 3.19), and its clear separation of the proven conditional statement from the broader 'realistic framework' claim. However, the paper's central scope claim—that tame geometry covers nearly all deep-learning objects—is not established by the evidence provided. The mathematical core is sound, but the transfer of Proposition 4.8 to 'current deep-learning architectures' requires a stronger, checkable inventory of definability in a common o-minimal structure, including non-activation components. As it stands, the reali","major_comments":[{"comment":"The central 'realistic framework' claim is not supported by a checkable inventory. Remark 3.19 estimates that 'around 89%' of the 400 surveyed activation functions are definable 'by direct inspection', while admitting 'a few liberties' with the count; about 5% are unclear and about 6% use unrestricted sine/cosine. This is not an itemized or reproducible audit. More importantly, the audit covers only activation and loss functions. Common deep-learning components such as sinusoidal positional encodings, rotary position embeddings, and periodic activation networks (e.g., SIREN) use unrestricted sin/cos and are therefore non-tame by Non-example 3.4; the manuscript does not address them. Since Proposition 4.8 requires the entire training objective to be definable in a single o-minimal structure—and Remark 3.15 correctly notes that composition is tame only within one common structure—Remark 3.","section":"§1.2, §5, Remark 3.19, Non-example 3.4, Remark 3.15"},{"comment":"The counting conventions further undermine the 'nearly all' claim. Functions with stochastic parameters (e.g., Noisy ReLU) are counted as definable 'since it is semialgebraic in the two variables z and a'; however, as the footnote itself acknowledges, random variables on an abstract sample space fall outside the definable framework. If such objects are counted as definable, the 89% figure overstates the fraction of practical activation mechanisms that are actually covered by Proposition 4.8. Similarly, the 5% fractional-derivative class is left as 'a bit unclear', and the 6% sine/cosine class is explicitly non-tame. The paper should either present a precise breakdown with examples, or replace 'nearly all' with a more limited statement such as 'the common activations and losses listed in Table 1'. This is not a purely stylistic point: it determines whether the convergence guarantee in Pro","section":"Remark 3.19 (items (1)–(3))"}],"minor_comments":[{"comment":"Typo: '*Eqal contribution' should read '*Equal contribution'.","section":"Title page"},{"comment":"Typo: 'relevence' should be 'relevance'.","section":"§4.1"},{"comment":"Typo: 'Lispchitz' should be 'Lipschitz'.","section":"§4.4.1"},{"comment":"Typo: 'Painlevéve-Kuratowski' should be 'Painlevé-Kuratowski'.","section":"§3.3, Lemma 3.23"},{"comment":"Typo: 'exercice' should be 'exercise'.","section":"Proposition 4.3"},{"comment":"The notation (✓) is explained in the caption, but the parenthetical marks may confuse readers because the same symbol appears in multiple columns; a short example of how to read the table would help.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an expository note rather than a new research contribution, which may be appropriate for the journal if surveys are welcomed. The mathematical core of Section 4 is sound and well-sourced. The main reservation is the mismatch between the grand 'realistic framework' claim and the heuristic, partial audit in Remark 3.19. The authors should either substantially temper the scope claims or supply a rigorous definability audit for actual architectures, including non-activation components and the common-structure requirement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear —,\n\nThis is a serious expository survey, not a new-results paper, and it knows it. The math it presents is faithful: the route from o-minimal structures to Verdier stratifications to the Davis–Drusvyatskiy–Kakade–Lee SSM convergence theorem is accurate, and the proof of Proposition 4.4 (projection formula + chain rule → descent identity) is internally consistent. The chain-rule proof in Section 4.2 is a genuinely useful piece of exposition; so is the Table 1 mapping of activations and losses to R_alg/R_exp/R_Pfaff. The discussion of conservative fields and AD is a good bridge to practice. If your goal is to understand why tame geometry underpins the SSM convergence guarantee, this is a solid entry point.\n\nThe soft spot is not the mathematics but the 'realistic' half of the thesis. The claim that o-minimality covers 'nearly all' deep learning rests on Remark 3.19's rough estimate: about 89% of 400 activation functions 'by direct inspection', with admitted liberties, 5% unclear, 6% non-tame. That is not a checkable inventory, and the survey only counts activation/loss functions. It does not address sinusoidal positional encodings, rotary embeddings, or SIREN-type periodic activations; unrestricted sin/cos are not tame. Remark 3.15's caution about composition across structures is also relevant: tameness is preserved only within a single structure, and the survey does not establish that all listed components share one. None of this undermines the conditional mathematical core — the SSM convergence guarantee is correct for functions that are definable in a common o-minimal structure — but it does narrow the practical reach claimed in Section 1.2 and the conclusion.\n\nMinor gaps: Small Sets Theorem is left as an exercise with no precise pointer, and the Figure 8 Monte Carlo reproduction has no code. Both are easy fixes. No load-bearing errors and no circularity; imported theorems are attributed correctly, and the self-citations are incidental.\n\nI would send this to a serious referee as an expository paper, with a request to either support or scale back the realism claim — an itemized supplement covering non-activation components, or an honest 'these guarantees hold for definable models' framing. It is not ground-breaking, but it is useful and mostly trustworthy.\n\nBest,","headline":"A faithful exposition of tame geometry for DL, with a realism claim that needs either more evidence or a softer statement.","tokens_in":35528,"tokens_out":2612,"would_cite":true,"duration_ms":29518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["03C64","90C26","68T07","49J52"],"pacs":[],"model":"deepseek-v4-flash","headline":"This expository note argues that tame geometry—the study of sets and functions definable in o-minimal structures—is the natural home for deep learning, where nearly every practical activation and loss is definable and that definability is w","keywords":["tame geometry","o-minimal structures","deep learning","stochastic subgradient method","Clarke subdifferential","nonsmooth nonconvex optimization","definable functions","stratification"],"falsifier":"Exhibit a widely used neural-network training objective that is provably not definable in any o-minimal structure—say, a network using an unrestricted sine or cosine activation on an unbounded domain, or a composition that joins functions from two structures whose amalgamation is not o-minimal—and show that the stochastic subgradient method either fails to converge or converges to a point that is not Clarke critical. Alternatively, test the coverage claim directly by carefully classifying all 400 surveyed activation functions and checking whether the 89% figure survives; a substantially lower","tokens_in":34385,"feed_emoji":"🧮","tokens_out":6120,"duration_ms":41312,"temperature":0.7,"pith_summary":"This expository note makes a two-part case. First, it argues that tame geometry—the study of sets and functions definable in o-minimal structures—is realistic for deep learning: the standard activations, losses, compositions, and partial minimizations used in neural networks are definable in one common such structure, so the networks themselves are tame. Second, it argues that this tameness is prolific, because it supplies exactly the finiteness and stratification properties needed to prove convergence of the stochastic subgradient method on the nonsmooth, nonconvex objectives that actually arise in training. The load-bearing result is Proposition 4.8: a definable locally Lipschitz objective with bounded iterates and square-summable, non-summable step sizes has every limit point Clarke critical and the objective values converge. A sympathetic reader should take the paper as a unified exposition of an existing proof chain, not as a claim of brand-new theorems.","feed_headline":"SGD provably reaches critical points on tame deep-learning losses","feed_subtitle":"Virtually all common activations and losses are 'tame', so nonsmooth, nonconvex convergence proofs apply.","key_machinery":"The central objects are o-minimal structures: collections of subsets of R^n closed under Boolean operations, products, and projections, in which one-dimensional definable sets are finite unions of intervals and points. A function is tame (definable) when its graph belongs to such a structure. The argument is carried by four pieces: the Projection formula, which stratifies any definable locally Lipschitz function into finitely many C^1 manifolds and confines each Clarke subgradient to the sum of the Riemannian gradient and the normal space of the stratum; the resulting Chain rule, which differentiates f along absolutely continuous curves stratum by stratum; the Descent identity expressing f(x","core_discovery":"The paper's central claim is that o-minimality is both a realistic and a prolific mathematical framework for deep learning. Realism: nearly all activation and loss functions that appear in practice—ReLU, logistic, tanh, softplus, swish, mish, ELU, GELU, arctan, and standard losses—are definable in a common o-minimal structure, with the Pfaffian closure, a structure built from semialgebraic sets plus antiderivatives, sufficing for all of them; composition with definable linear maps preserves definability. Prolific: definability implies finite stratifications into smooth pieces, a projection formula for Clarke subdifferentials, a chain rule along absolutely continuous curves, and a weak Sard p","pith_inferences":["If the paper's rough estimate holds, the framework reaches far beyond the Table 1 list: most of the 400 surveyed activations are tame, so the convergence theorem should apply to most current architectures out of the box; the 5% fractional-derivative and 6% trigonometric examples are precisely where failures would surface.","An implicit tension: quantitative rate-of-convergence arguments often rely on polynomial boundedness and a field-of-exponents parameter, whereas many practical tame networks require the exponential function and thus live in structures that are not polynomially bounded; closing that gap would require a separate argument the paper does not provide.","A testable extension is to take a non-tame activation such as cos(z) on an unbounded domain, compose it with a simple linear layer, and check numerically whether stochastic subgradient descent still finds Clarke critical points; the theory predicts convergence can fail, with oscillation rather than divergence of the objective as the likely failure mode.","The same tools suggest a design rule for practitioners: keep every component of a network definable in a single o-minimal structure and avoid unrestricted periodic functions if the convergence guarantee is wanted—a far more permissive rule than convexity or smoothness."],"forward_implications":["Any definable locally Lipschitz training objective—including those built from ReLU, logistic, tanh, softplus, swish, mish, ELU, GELU, arctan, and standard losses—has the property that every bounded stochastic subgradient run with square-summable, non-summable step sizes converges to Clarke critical points.","The convergence guarantee extends to automatic-differentiation-based implementations, because definable conservative fields agree with Clarke subdifferentials almost everywhere; this covers the practical tools used for deep learning.","The same finiteness principles imply that central paths in definable convex semidefinite programs converge, and that definable curves have one-sided limits everywhere—a concrete manifestation of the 'no infinite oscillation' principle.","Because definable hypothesis classes have finite VC dimension, tame hypothesis spaces are PAC learnable under the fundamental theorem of statistical learning.","Optimization over tame functions is at least first-order tractable in a strong sense, in contrast to optimization over unrestricted trigonometric functions, which is undecidable even in the box-constrained case."],"fun_headline_variants":["Tame geometry proves SGD converges on deep nets","For tame losses, SGD finds critical points","Why SGD works: most deep learning losses are tame","Tame math ensures SGD convergence in deep learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole argument leans on the premise that real neural-network building blocks are mostly definable in one common o-minimal structure; the paper supports this with a rough count (about 89% of 400 activations 'by direct inspection', about 5% unclear, about 6% excluded) and with the caveat that composition preserves tameness only inside a single structure.","fun_headline_variants_meta":{"raw":{"variants":["Tame geometry proves SGD converges on deep nets","For tame losses, SGD finds critical points","Why SGD works: most deep learning losses are tame","Tame math ensures SGD convergence in deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":934,"prompt_tokens":631,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":375,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":375,"tokens_out":303,"duration_ms":2375,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T15:48:17.862090+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Exhibit a widely used neural-network training objective that is provably not definable in any o-minimal structure—say, a network using an unrestricted sine or cosine activation on an unbounded domain, or a composition that joins functions from two structures whose amalgamation is not o-minimal—and show that the stochastic subgradient method either fails to converge or converges to a point that is not Clarke critical. Alternatively, test the coverage claim directly by carefully classifying all 400 surveyed activation functions and checking whether the 89% figure survives; a substantially lower","supporting_citations":[],"review_version":1}