{"id":"7d849226-c802-481d-8398-26c45da7d6d8","arxiv_id":"2505.09873","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A review of explainable AI methods in genomics whose advertised mathematical proofs of interpretability limitations are not rigorous and partly assume what they set out to show.","lead":"This paper reviews how deep learning models in genomics are made interpretable and divides the methods into input-based and model-based approaches. It then tries to prove mathematically why these methods fail, but several of the proofs are circular or incomplete.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transparent-model proof in §3.2 assumes its conclusion: p_hard_i ≤ p_free_i is asserted, not derived, so the advertised rigorous derivation of performance loss in hard-coded models collapses.","rationale":"I agree with the reader's weakest assumption. The transparent-model proof is the clearest place where the 'rigorous derivation' promise breaks: the key inequality is assumed rather than proven. This is load-bearing because the contribution list explicitly advertises formal proofs for selected limitations, and this is the proof for the transparent-model limitation. Rejecting the paper does not require disputing the survey's factual content or the empirical folklore that constrained models can underperform; it only requires noting that the advertised mathematical support is absent. Independent support for the surrounding review (e.g., a table of tools and references) does not rescue the formal claim. I would keep the reader's REJECT verdict. Other derivations have similar issues (the attention theorem proves ill-conditioning of K^T K rather than instability of softmax(QK^T); the dropout argument assumes the redundancy it claims to explain), but the transparent-model circularity is sufficient and is the best single target.","tokens_in":13214,"tokens_out":6228,"duration_ms":65547,"concrete_test":"Implement Eqs. (23)-(24) on a synthetic binary classification dataset: one 'hard-coded' model with fixed W_prior and trainable W_adapt, and one unconstrained model with trainable W_free, both optimized with the stated losses on identical data. Record per-sample predicted probabilities p_hard_i and p_free_i. If any i has p_hard_i > p_free_i, or if the hard-coded model achieves lower cross-entropy, the asserted pointwise inequality is false as a general statement and the proof's conclusion does not follow. Sweep λ, γ, β and random initializations; report the sign of L_hard - L_free.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is §3.2's proof that hard-coded transparent models underperform unconstrained models. After defining L_hard (Eq. 23) and L_free (Eq. 24), the paper states 'Due to the limitations imposed by hard coding, it holds that p_hard_i <= p_free_i' for each sample i, then uses that inequality to conclude L_hard - L_free >= 0. This pointwise inequality is not a consequence of the model definitions or of any optimization argument; it is a stronger version of the performance-degradation claim being proved. The proof also hand-waves the regularizer difference λ||W_prior-W_ref_prior||^2 + γ||W_adapt||^2 - β||W_free||^2 ≈ 0 without showing the coefficients or norms are small. Because the paper's advertised contribution is rigorous mathematical derivation of interpretability limitations, a proof whose key premise is the conclusion cannot support that contribution. The survey material may be useful, but the central formal claim fails at this step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reviews interpretable deep learning methods in genomics, categorizing them into input-based approaches (convolutional kernel visualization, gradient-based methods, perturbation-based methods) and model-based approaches (attention mechanisms and transparent models based on biological prior knowledge). It surveys applications of these methods to 3D genomics and regulatory genomics, and then attempts to provide formal mathematical derivations of five limitations: convolutional weight scaling instability, dropout-induced neuronal redundancy, vanishing gradients, attention weight instability under multicollinearity, and performance loss in hard-coded transparent models. The paper's advertised contribution is to rigorously reveal the inherent limitations of selected interpretable approaches through mathematical derivation.","tokens_in":13457,"tokens_out":5505,"duration_ms":51152,"significance":"The survey component is useful: the categorization is clear, Table 1 collects relevant tools with URLs and biological applications, and the discussion of concrete use cases (Basset, Enformer, DCell, GenNet) is informative. Some mathematical material, such as Lemma 1 and the standard chain-rule derivation of gradient decay, is correct. However, the paper's central claim of rigorous derivation is not supported: the transparent-model proof assumes its conclusion, the dropout-redundancy 'manifold' argument is not a valid theorem, and the attention condition-number result is applied to a singular matrix whose inverse does not exist. If the theoretical claims were removed or explicitly reframed as heuristic observations, the manuscript could serve as a useful review, but as it stands it does not deliver the formal results it advertises.","major_comments":[{"comment":"The proof that hard-coded transparent models underperform unconstrained models assumes the conclusion. The pointwise inequality p_hard_i ≤ p_free_i is asserted without derivation after Eq. (24), yet this inequality is exactly the performance-degradation claim being proved. In addition, p_hard_i is defined as f(Wprior, W_adapt, m_i), making the predicted probability a function of the true label, which is not a valid model specification. The regularizer difference term λ||Wprior − Wref_prior||² + γ||Wadapt||² − β||Wfree||² ≈ 0 is also asserted without justification of the relative magnitudes of the coefficients or norms. The advertised rigorous derivation of transparency-induced performance loss therefore collapses at this step.","section":"§3.2, Eqs. (23)-(24)"},{"comment":"The dropout-redundancy argument is not a theorem. Eq. (7) already assumes that perturbations in directions vi and vj yield approximately equal outputs for all inputs, which is the definition of redundancy being used to conclude that removing one neuron changes the output little. The introduction of covariant derivatives and the Riemann curvature tensor is not load-bearing: fθ is a scalar-valued function, so the inner product ⟨∇vi∇vj fθ − ∇vj∇vi fθ, fθ⟩ and 'components of the Riemann curvature tensor' are not defined for scalar fields. Moreover, removing a neuron is a discrete structural change, not a small parameter perturbation αvi, so the Taylor expansion in α does not apply. The conclusion Δf ≈ 0 therefore does not follow from the stated assumptions.","section":"§3.1, 'The redundancy of neurons', Eqs. (7)-(9)"},{"comment":"Theorem 2 applies the formula ||A^{-1}||₂ = 1/√λmin to a matrix that is singular by construction. Since multicollinearity is defined by r(K) < n, the matrices K and C = K^T K have zero eigenvalues, so A^{-1} does not exist and the standard condition number κ(A) = ||A||₂ ||A^{-1}||₂ is undefined for the matrix at hand. The conclusion that the condition number 'tends to infinity' confuses the unbounded behavior of nearly singular matrices with the undefined quantity for exactly singular matrices. The argument could be repaired by considering κ(A + δI) with δ → 0 or by using the pseudo-inverse condition number, but this is not what is written.","section":"§3.2, 'Instability of in the estimation of the weights', Theorem 2 and Eq. (22)"},{"comment":"The claim that unconstrained weight optimization 'provably causes' scaling instability is not established. The derivation shows only that if σ(z) − y < 0 and the subsequent factor α1s11 + ... + αLsL1 is positive, then w11 increases; it does not show that this condition holds, that the growth is unbounded, or that a constrained variant would avoid the issue. Additionally, Eq. (1) appears to contain a sign error: standard gradient descent is w_ij ← w_ij − η ∂L/∂w_ij, not w_ij ← −w_ij − η ∂L/∂w_ij. As written, the section is an informal illustration of a known risk, not a rigorous proof of scaling instability.","section":"§3.1, 'The unconstrained learning of weights', Eqs. (1)-(5)"}],"minor_comments":[{"comment":"Typographical errors appear throughout: 'DeePLIFT' should be 'DeepLIFT', 'Bernouli' should be 'Bernoulli', 'Multicolinearity' should be 'multicollinearity', and 'whichϵ is always determined' is missing a space before the epsilon.","section":"Throughout"},{"comment":"Reference [33] is mis-titled: the cited arXiv:1611.02639 paper is 'Axiomatic Attribution for Deep Networks', not 'Gradients of counterfactuals'.","section":"References, [33]"},{"comment":"In the final product, the ordering of weight matrices is presented inconsistently: the sentence 'When W(1), W(L−1),...,W(L) are very small' does not match the previous product W^{(L)} W^{(L−1)} ... W^{(1)}; the subscripts should be ordered consistently.","section":"§3.1, 'Vanishing gradient in DNA sequences'"},{"comment":"The PWM/PFM terminology is used inconsistently: Section 2.1 says a PFM is log-scaled to a PWM, while Section 3.1 says 'transform the PWM into a PFM [28] through the softmax operation.' The paper should clarify which matrix is converted into which.","section":"§2.1 and §3.1"}],"recommendation":"reject","confidential_remarks":"The manuscript could be reconsidered as a survey if the formal proofs are either removed or explicitly replaced with heuristic discussions; in its current form, the central contribution of rigorous derivation of interpretability limitations is not achieved. The paper's fit with q-bio.GN is reasonable for a review, but the mathematical claims would need substantial reworking before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou should know two things about this paper. First, the survey of interpretability methods in genomics is worth reading: the taxonomy of input-based vs. model-based methods is clean, the table of tools with URLs is practical, and the examples are well chosen. Second, the advertised theoretical contribution—rigorous mathematical proofs of the limitations of these methods—does not hold up. The central proof in §3.2 that hard-coded transparent models underperform unconstrained models simply asserts the pointwise inequality p_hard_i ≤ p_free_i, which is exactly the conclusion it is supposed to establish. The regularization difference is then hand-waved to zero. So the paper's main novelty claim fails at its key step.\n\nWhat the paper does well is organize a scattered literature. The vanishing-gradient section correctly walks through the chain rule, and the condition-number section correctly notes that multicollinearity makes covariance matrices singular. These are textbook facts, but they are presented clearly. The citation pattern looks fine; the authors are drawing on the right body of recent work.\n\nThe soft spots are real and load-bearing. The CNN scaling analysis has a sign error in the update rule (Eq. 1) and never derives a concrete bound; it just observes that weights can grow. The dropout-redundancy argument uses Riemann curvature tensors on an undefined manifold and then concludes that removing a redundant neuron leaves the output unchanged—that's assuming what it should show. The attention analysis applies the condition number of K^T K to a setting where that matrix is never inverted, so it doesn't actually speak to attention-weight stability. And the transparent-model proof is circular, as above. None of these would matter much if the paper were positioned as a practical review; they undercut the claim of proving limitations.\n\nWho is this for? Geneticists who want a structured overview of interpretable deep learning will get value from the survey. They should skip the math.\n\nMy recommendation: send it to peer review, but the editor should make clear that the theoretical sections need either rigorous rewriting or removal. If the authors reframe the paper as a survey, it could be a decent contribution.\n\nBest, [Your name]","headline":"The survey of interpretable deep learning in genomics is useful, but the advertised proofs of limitations are not sound—the key proof in §3.2 assumes the inequality it needs to establish.","tokens_in":13918,"tokens_out":7123,"would_cite":false,"duration_ms":65771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the limitations of input-based and model-based interpretability in genomics can be derived mathematically, not merely observed.","keywords":["explainable AI","genomics","deep learning interpretability","attention mechanism","transparent models","vanishing gradient","multicollinearity","mathematical derivation"],"falsifier":"Train the same architecture twice on the same genomic dataset, once with a hard-coded biological prior and once unconstrained, and record per-sample predicted probabilities; if any sample has $p^{\\text{hard}}_i > p^{\\text{free}}_i$ while both models are near convergence, the assumed inequality fails and the claimed loss ordering can reverse.","tokens_in":13051,"feed_emoji":"🧬","tokens_out":8095,"duration_ms":75759,"temperature":0.7,"pith_summary":"This paper is a review of interpretable deep learning in genomics that goes further than cataloguing methods: it tries to prove, from the ground up, why several widely used interpretation tools can be misleading. It argues that unconstrained convolutional kernels can grow into distorted position weight matrices, dropout-trained networks can hide feature importance in redundant neurons, deep ReLU networks can suffer vanishing gradients, multicollinear DNA features can make attention weights numerically unstable, and hard-coded biologically transparent models can lose predictive performance. If the derivations hold, the limitations of input-based and model-based interpretability in genomics stop being empirical folklore and become checkable mathematical facts that a researcher can test on their own model. The review also gives a practical map of current methods and future directions, but the theoretical claims are the pith.","feed_headline":"Proofs pin down when genomics AI explanations fail","feed_subtitle":"A review derives formal limits for convolutional, attention, and transparent models used in genetics.","key_machinery":"The machinery is a set of self-contained derivations attached to each interpretability class. The convolutional-kernel argument expands the update $\\partial L/\\partial w_{11}=(\\sigma(z)-y)(\\alpha_1 s_{11}+\\cdots+\\alpha_L s_{L1})$ to show unconstrained weights can grow, then applies the softmax transform $w'_{11}=e^{w_{11}}/(e^{w_{11}}+e^{w_{12}}+e^{w_{13}}+e^{w_{14}})$ followed by $\\log(w'_{ij}+\\epsilon)$ to show how extreme entries distort the position weight matrix. The dropout argument treats neurons as parameter directions in a high-dimensional manifold and identifies redundant directions through commuting covariant derivatives, giving a locally flat Riemann tensor $R(v_i,v_j,v_i,v_j)\\approx 0$; removal of such a neuron changes the output by $O(\\epsilon^2)$. The vanishing-gradient argument multiplies per-layer factors $\\mathrm{ReLU}'(W^{(l)}Z^{(l-1)}+b^{(l)})W^{(l)}$ and notes the product can go to zero. The attention argument proves Lemma 1 ($\\|A\\|_2=\\sqrt{\\lambda_{\\max}(A^TA)}$) and then Theorem 2: multicollinearity forces $\\lambda_{\\min}=0$, so the condition number $\\kappa(A)=\\|A\\|_2\\|A^{-1}\\|_2\\to\\infty$. The transparent-model argument compares $L_{\\text{hard}}$ and $L_{\\text{free}}$ term by term, reducing the comparison to the pointwise inequality $p^{\\text{hard}}_i\\le p^{\\text{free}}_i$.","core_discovery":"The paper's central claim is that the known weaknesses of interpretable methods in genomic deep learning are not just anecdotal: they follow from explicit mathematical statements about the trained model. Concretely, it asserts that (1) unconstrained gradient-descent updates on convolutional kernels can drive individual weights so large that the softmax-to-log conversion used in motif visualization distorts the importance of sequence positions; (2) dropout's random deactivation forces redundant neurons, so removing one neuron changes output only to second order and perturbation-based attribution understates its importance; (3) the product of layer-wise gradients in ReLU networks can vanish, making gradient-based saliency unreliable; (4) when key vectors in attention are multicollinear, the covariance matrix $K^TK$ is singular and the condition number of the attention matrix tends to infinity, implying numerical instability; and (5) a hard-coded transparent model has loss no less than an unconstrained model when per-sample predicted probabilities satisfy $p^{\\text{hard}}_i\\le p^{\\text{free}}_i$, so biological hard-coding can degrade performance.","pith_inferences":["The condition-number criterion suggests a practical pre-check: compute the smallest eigenvalue of the key-feature covariance before trusting attention; if $\\lambda_{\\min}$ is near machine precision, attention weights should be treated as arbitrary.","The loss-difference proof implies a testable scaling law: the performance gap of a hard-coded transparent model should grow with $\\sum_i m_i\\log(p^{\\text{free}}_i/p^{\\text{hard}}_i)$, so measuring per-sample probability gaps on a benchmark would quantify how much biology has to be lost to gain transparency.","The same loss-comparison template could be extended to other transparency constraints, such as sparse connectivity, modular layers, or fixed motif detectors, by swapping the prior-constraint term in $L_{\\text{hard}}$.","The manifold argument yields a direct experiment: train two networks with and without dropout, delete matched neurons, and measure output change; the dropout-trained network should show smaller deletion effects in directions where the local curvature is flat."],"forward_implications":["Convolutional kernel visualizations should not be read as motifs unless the training imposed constraints on kernel weights; without constraints, near-zero or near-one softmax entries can dominate the log-transformed PWM.","In networks trained with dropout, small output changes after deleting a neuron do not establish that the neuron is unimportant, so perturbation-based attribution maps need redundancy correction to be trustworthy.","For deep ReLU models, zero or tiny input gradients do not mean the input is irrelevant; gradient-based saliency should be paired with path-integral or reference-based methods.","Attention heatmaps from DNA sequences with correlated positions are numerically unstable; the condition number of the key-feature covariance should be reported before interpreting attention weights biologically.","Transparent models that hard-code biological prior interactions can be expected to show higher loss than unconstrained models unless the prior constraints are regularized gently, giving a concrete trade-off between interpretability and predictive accuracy."],"supporting_citations":[{"why":"Supplies the PFM-to-PWM log transformation and the empirically motivated claim that transparency techniques may compromise performance, which the paper turns into formal statements.","marker":"[28]"},{"why":"Defines the dropout mechanism that the neuron-redundancy proof assumes when it models random deactivation of neurons.","marker":"[12]"},{"why":"Defines the query-key dot-product attention whose key covariance the instability theorem analyzes.","marker":"[37]"},{"why":"Provides the ill-conditioning and numerical-instability background used in the attention condition-number argument.","marker":"[40]"},{"why":"Supplies the transparent-model architecture of hierarchical biological units that the hard-coded-versus-free loss comparison targets.","marker":"[22]"}],"fun_headline_variants":["Formal proofs reveal why genomics AI explanations break","Mathematics explains when genomic deep learning insights fail","Rigorous limits for interpretability in genomics AI","Proofs show exact failure points of genomics AI explainers","Convolution, dropout, attention: proofs map their failure modes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, sample by sample, the hard-coded transparent model's predicted probability never exceeds the unconstrained model's; that inequality is asserted rather than proven, and the conclusion that hard-coded models perform worse would not follow if any sample violates it.","fun_headline_variants_meta":{"raw":{"variants":["Formal proofs reveal why genomics AI explanations break","Mathematics explains when genomic deep learning insights fail","Rigorous limits for interpretability in genomics AI","Proofs show exact failure points of genomics AI explainers","Convolution, dropout, attention: proofs map their failure modes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000293,"raw_usage":{"total_tokens":1679,"prompt_tokens":892,"completion_tokens":787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":508,"tokens_out":787,"duration_ms":7877,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:22:33.212808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same architecture twice on the same genomic dataset, once with a hard-coded biological prior and once unconstrained, and record per-sample predicted probabilities; if any sample has $p^{\\text{hard}}_i > p^{\\text{free}}_i$ while both models are near convergence, the assumed inequality fails and the claimed loss ordering can reverse.","supporting_citations":[{"cited_title":"Novakovsky, N","cited_arxiv_id":null,"evidence_quote":"Supplies the PFM-to-PWM log transformation and the empirically motivated claim that transparency techniques may compromise performance, which the paper turns into formal statements."},{"cited_title":"Zhang, H","cited_arxiv_id":null,"evidence_quote":"Provides the ill-conditioning and numerical-instability background used in the attention condition-number argument."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the transparent-model architecture of hierarchical biological units that the hard-coded-versus-free loss comparison targets."}],"review_version":1}