{"id":"55627531-43e8-4d9d-bb08-496b87c88ed6","arxiv_id":"1909.01730","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A temporal convolutional network is shown to be structurally equivalent to Volterra series and parallel Wiener models, and it performs competitively on nonlinear system identification benchmarks.","lead":"This paper links temporal convolutional networks, a modern deep learning architecture for sequences, to classical system identification models such as Volterra series and Wiener models. It also benchmarks TCNs against MLPs and LSTMs on real-world problems including an F-16 aircraft dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Volterra equivalence for ReLU TCNs is unproven: the Taylor-expansion argument of [43] requires an analytic activation, and ReLU is not analytic, so the transfer to TCNs fails as stated.","rationale":"The reader's weakest assumption identifies exactly the gap I find most load-bearing: the Volterra equivalence is transferred from sigmoid TDNNs to ReLU TCNs without justifying the activation-function change. Section III-A explicitly uses the Taylor-expansion argument in [43], and the paper's own parenthetical about expanding 'around the bias values' does not rescue the argument because ReLU's Taylor series around any positive bias is globally linear on the positive half-line and cannot represent the function's zero region. Since the universal-approximation conclusion ('They are thus all universal approximators') is derived from the Volterra equivalence, the central claim rests on an unproven premise. I considered the parallel-Wiener 'squeezing' argument in Section III-B as an alternative concern, but that claim is more definitional: if 'parallel Wiener models' are allowed multivariate nonlinear blocks, the equivalence becomes near-tautological, and if not, the paper's specific claim is less clearly stated; the ReLU gap is sharper and more clearly fatal to the proof as written. I therefore agree with the reader's verdict of CONDITIONAL: the paper's experimental and expository content has value, but the theoretical equivalence requires a corrected proof, likely replacing the Taylor-expansion argument with a universal-approximation argument for ReLU networks or explicitly qualifying the equivalence as approximate.","tokens_in":14448,"tokens_out":10875,"duration_ms":115494,"concrete_test":"Analytically apply the kernel-construction procedure of [43] to a single ReLU neuron y[k] = ReLU(w u[k] + b) with b > 0. Expanding ReLU around the bias b yields the Taylor series b + (x - b) = x on the domain of convergence; all higher-order coefficients are zero. The resulting Volterra series is therefore linear: y[k] = w u[k] + b. Compare this to the true ReLU output on an input where w u[k] + b is negative, for example u[k] = -10, w = 1, b = 0.1: the Volterra series predicts -9.9 while the actual output is 0. This demonstrates the [43] equivalence fails for ReLU, and no amount of added Volterra degree (all higher kernels are zero) can fix the mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A transfers the TDNN–Volterra equivalence of [43] to the 'basic TCN' solely on the grounds that the TCN is 'essentially the same' as a TDNN 'except for zero padding and the use of ReLU activations instead of sigmoids.' The proof in [43] computes Volterra kernels from a Taylor expansion of the activation function. That procedure is valid for sigmoids, which are analytic, but ReLU is not analytic: it is not differentiable at zero, and its Taylor series around any bias b > 0 is just the linear function x (because ReLU(x) = x on a neighborhood of b). Consequently, the infinite-degree Volterra series obtained by the [43] construction for a ReLU neuron y[k] = ReLU(w u[k] + b) is simply w u[k] + b, which disagrees with the true output whenever w u[k] + b < 0. The paper does not provide any alternative proof that ReLU-based TCNs lie in the same function class as Volterra series. Since the headline conclusion in Section III-C — 'The basic TCN architecture is equivalent to Volterra series... They are thus all universal approximators' — relies directly on this equivalence, the central theoretical claim is not supported by the argument given. The conclusion may still be true via other universal-approximation results, but the paper's stated route is invalid for ReLU activations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper aims to build a bridge between deep learning and classical nonlinear system identification by studying temporal convolutional networks (TCNs). It describes the TCN as a stack of nonlinear ARX models, discusses residual blocks, and then makes theoretical connections: Section III-A claims that the basic TCN architecture (without dilation and skip connections) is equivalent to an infinite-degree Volterra series, transferring an earlier result for time-delay neural networks (TDNNs) with sigmoid activations; Section III-B argues that a multi-layer TCN can be reorganized as a parallel Wiener model; Section III-C concludes that basic TCNs, Volterra series, and parallel Wiener models are all universal approximators for time-invariant systems with fading memory. The experimental part evaluates TCN, MLP, and LSTM models on a nonlinear toy problem, the Silverbox benchmark, and an F-16 ground vibration dataset, with separate one-step-ahead and free-run simulation metrics. The code is made publicly available.","tokens_in":14782,"tokens_out":3335,"duration_ms":36574,"significance":"If the theoretical equivalence were rigorously established, the paper would provide a valuable conceptual link between modern deep sequence models and classical system identification structures, offering an interpretation of TCN hyperparameters (depth, kernel size, dilation) in terms of Volterra kernels and block-oriented models. The empirical study is honest and useful: the authors clearly distinguish training, validation, and test data, report both one-step-ahead and free-run errors, compare against established benchmarks, and openly document that standard deep learning techniques such as dropout and batch normalization often do not help in these identification tasks. The reproducible code is a further strength. However, the central theoretical claim is currently not supported for ReLU-based TCNs, because the transfer from the sigmoid-TDNN result in [43] to ReLU activations is made without proof and is in fact invalid as stated. Since the universal-approximation conclusion rests on this equivalence, the theoretical contribution needs substantial revision before the paper can be accepted.","major_comments":[{"comment":"The equivalence between the basic TCN and an infinite-degree Volterra series is not established for ReLU activations. The paper states that the TCN is 'essentially the same' as the TDNN in [42] 'except for the zero padding and the use of ReLU activations instead of sigmoids', and then invokes [43] to conclude equivalence to a Volterra series. However, the proof in [43] explicitly constructs Volterra kernels by Taylor-expanding the activation function. That construction is valid for analytic activations such as sigmoids but not for ReLU, which is not analytic: its Taylor series around any bias b>0 is the linear function x, its Taylor series around b<0 is the zero function, and it is not differentiable at b=0. Consequently, the infinite-degree Volterra series obtained by the [43] construction for a ReLU neuron y[k]=ReLU(w u[k]+b) is either w u[k]+b or 0 locally, and cannot reproduce the actual ReLU output on the region where w u[k]+b changes sign. The paper provides no alternative proof that a ReLU-based TCN belongs to the class of Volterra series, nor a precise statement in terms of uniform approximation on compact sets.","section":"Section III-A"},{"comment":"The headline conclusion that 'The basic TCN architecture is equivalent to Volterra series and parallel Wiener models. They are thus all universal approximators for time-invariant systems with fading memory' depends directly on the Volterra equivalence discussed in Section III-A. Since that equivalence is not proven for ReLU activations, the universal-approximation claim is not supported by the argument given in the paper. The conclusion may be salvageable through a different route, for example by citing general universal-approximation results for neural networks with memory or by proving that ReLU networks can approximate fading-memory operators, but the current manuscript does not provide such a proof. This is a load-bearing gap in the central theoretical contribution.","section":"Section III-C"}],"minor_comments":[{"comment":"In the F-16 subsection, the sentence about MLP training time is duplicated: 'Training the network with this configuration took 4 minutes. Training the network with this configuration took approximately 5 minutes.' One of the two sentences should be removed.","section":"Appendix B"},{"comment":"The experimental tables report single runs without confidence intervals or multiple-seed variation. Given that neural network training is stochastic, reporting the mean and spread over several seeds would strengthen the comparison, especially where the differences between TCN, MLP, and LSTM are small.","section":"Tables II-IV"},{"comment":"The comparison of RMSE values across many published methods is informative but should be read cautiously, because different methods use different sample ranges and experimental setups; the paper already partly acknowledges this by selecting 'no extrapolation' regions, but a brief sentence warning that the table is not a strictly controlled comparison would help the reader.","section":"Table III"},{"comment":"The phrase 'squeezed down to a parallel Wiener model' might be misinterpreted as implying that the number of parameters is preserved. As the authors note, the layers become 'less but larger', so the transformation is a structural reorganization rather than a parameter-preserving reduction; a more precise wording would avoid confusion.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The main issue is that the theoretical section, which is a central contribution of the paper, currently contains a load-bearing gap: the Volterra equivalence is transferred from sigmoid TDNNs to ReLU TCNs without proof, and the Taylor-expansion argument in [43] does not apply to ReLU. The authors should either prove an approximation result for ReLU TCNs (e.g., via piecewise linear approximation or by restricting the statement to uniform approximation on compact input sets), or clearly weaken the claim to what is actually demonstrated. The experimental part is solid and I would be supportive of publication once the theoretical claim is fixed or appropriately scoped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nYou asked about Andersson et al. on TCNs for system identification. The honest quick take: this is a useful bridge paper, but its main theoretical claim—that a basic TCN is equivalent to a Volterra series—is not actually proven for ReLU activations. The stress-test note you forwarded is correct.\n\nWhat's genuinely new here is the experimental work. The authors apply a standard TCN to the Silverbox and F-16 benchmarks, compare against MLP and LSTM, and report a few practical findings: dropout and dilation don't help on these problems, and the TCN does okay on small data. That's useful for anyone thinking about using TCNs in practice. The code is public, which also helps.\n\nThe theoretical section is where it gets shaky. The paper leans on Wray and Green's result that a time-delay neural network with sigmoid activations can be expanded into an infinite Volterra series. From there, it says the TCN is 'essentially the same' except for ReLU and zero-padding, and concludes the equivalence carries over. As the stress-test note spells out, the Taylor-expansion argument in Wray and Green requires the activation to be analytic. ReLU isn't analytic—it's piecewise linear. Around a positive bias, its Taylor series is just the linear term, so you'd miss the kink. The paper doesn't supply an alternative proof. So the statement that a basic TCN is equivalent to a Volterra series, in the same sense as [43]'s sigmoid case, is unsupported. The conclusion may well be true via other routes—ReLU networks are universal approximators for continuous functions—but the paper's stated route is invalid.\n\nThat's a real flaw, but not a reason to dismiss the whole paper. The empirical part stands on its own. My bigger complaint there is that the headline RMSE numbers for Silverbox and F-16 come from a single run with best hyperparameters, so there's no sense of variability. Box plots for the toy problem are nice, but the main comparisons lack error bars. That's a fixable weakness, but it should be addressed.\n\nWho's this for? Someone doing nonlinear system identification who wants a practical look at how TCNs behave on real datasets. The theory needs restructuring—either a proper proof for ReLU, or a more careful statement of what's heuristic. Given the interest in the topic and the honest empirical work, I'd send this to reviewers, not desk-reject it. Expect them to ask for a major revision—mostly to fix the Volterra claim and tighten the experimental reporting.\n\nBest,\n[Your name]","headline":"A useful empirical bridge paper whose headline Volterra-equivalence claim is not actually proven for ReLU; worth refereeing but needs a major revision.","tokens_in":15281,"tokens_out":3519,"would_cite":false,"duration_ms":35989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The basic temporal convolutional network is equivalent to Volterra series and parallel Wiener models, making it a universal approximator for time-invariant systems with fading memory.","keywords":["temporal convolutional network","system identification","Volterra series","block-oriented models","parallel Wiener model","nonlinear ARX","fading memory","deep learning"],"falsifier":"Hand-construct a one-layer ReLU TCN with output $\\hat{y}[k]=\\max(0,u[k]-u[k-1])$ and compare it with a Volterra series: a Volterra series is a polynomial in delayed inputs, so no convergent Volterra series can reproduce the kink along $u[k]=u[k-1]$, and demonstrating this failure would settle that the equivalence as stated needs qualification about domain or approximation topology.","tokens_in":14287,"feed_emoji":"⚙️","tokens_out":8895,"duration_ms":86226,"temperature":0.7,"pith_summary":"The paper sets out to connect two communities by showing that the basic temporal convolutional network (TCN) is not a new model class at all: it is the same as the classical Volterra series and parallel Wiener models. If that equivalence holds, TCNs inherit universal approximation for time-invariant systems with fading memory, and system identification theory about block-oriented models applies to modern deep sequence networks. The paper then tests TCNs against multilayer perceptrons and LSTMs on a nonlinear toy problem, the Silverbox circuit benchmark, and an F-16 ground vibration dataset, reporting that TCNs are competitive, work especially well with little training data, and do not benefit from common deep learning regularization tricks.","feed_headline":"Temporal convolutional networks equal Volterra series","feed_subtitle":"If right, TCNs are universal approximators for fading-memory systems and inherit classical system identification theory.","key_machinery":"The central object is the basic TCN, understood as a stack of causal FIR filters with pointwise ReLU nonlinearities. Two identities carry the argument: the Taylor expansion of the activation, which turns a TDNN layer into an infinite-degree Volterra series, and the shift commutation $q^{-1}\\sigma(z[k])=\\sigma(q^{-1}z[k])=\\sigma(z[k-1])$, which lets static nonlinear blocks be moved past delays so that stacked layers collapse into a parallel Wiener model. These two moves are what turn the architectural resemblance between TCNs and old block-oriented models into a stated model-class equivalence.","core_discovery":"The central claim is that a multi-layer TCN without dilations and skip connections is formally equivalent to a Volterra series of infinite degree and to a parallel Wiener model. The argument runs through the time-delay neural network: the TCN is the TDNN with zero padding and ReLU activations, and the TDNN was previously shown to compute Volterra kernels via Taylor expansions of its activation functions. The paper adds the observation that because time delays commute with static nonlinearities, the layers of a basic TCN can be squeezed together into a parallel Wiener model with one linear output layer. From this it concludes that TCN, Volterra, and parallel Wiener structures are all universal approximators for the same class of systems, while noting that training difficulty and basis-function suitability still differ.","pith_inferences":["The ReLU gap is repairable: replacing ReLU with a smooth activation such as softplus, or proving approximation directly with piecewise-linear bases, would put the Volterra equivalence on solid footing without changing the architecture's practical behavior.","If the equivalence holds, frequency-domain tools for block-oriented models, such as best linear approximation and nonlinear distortion analysis, become available for debugging and certifying trained TCNs, a path the paper leaves implicit.","The absence of any benefit from dilation on these benchmarks is likely problem-dependent; systems with genuinely long memory, such as switched or nearly periodic dynamics, are where dilated TCNs should be retested."],"forward_implications":["A basic TCN can approximate any time-invariant system with fading memory to within any desired accuracy, just as Volterra series can.","Because causal convolutions are effectively NARX models, TCN estimates are expected to be biased when the noise is colored; the paper identifies this as the likely source of the limitations seen in its experiments.","On the benchmarks tested, dropout, batch normalization, and dilated convolutions did not improve TCN accuracy, while depth helped only on the F-16 problem.","TCNs gave the best results among the compared architectures when training data were scarce, which suggests they are a sensible default in small-data identification problems.","The equivalence to parallel Wiener models means structural and frequency-domain results for block-oriented systems can be carried over to TCNs trained by deep learning."],"supporting_citations":[{"why":"Defines the TCN architecture (causal dilated convolutions, residual blocks) that the paper interprets as stacked nonlinear ARX models.","marker":"[25]"},{"why":"Defines the time-delay neural network that the TCN is said to match except for zero padding and ReLU activations.","marker":"[42]"},{"why":"Shows how to compute Volterra kernels from a TDNN by Taylor-expanding its sigmoid activations; this is the key equivalence the paper inherits.","marker":"[43]"},{"why":"Establishes that Volterra series are universal approximators for time-invariant fading-memory systems, the class for which TCNs are claimed to be universal.","marker":"[49]"},{"why":"Defines block-oriented model structures such as Wiener and Hammerstein systems used in the parallel-Wiener equivalence.","marker":"[44]"},{"why":"Surveys identification of block-oriented nonlinear systems, grounding the terminology and theory the paper connects to TCNs.","marker":"[45]"},{"why":"Shows how sandwich/block models can be squeezed together, supporting the argument that stacked TCN layers collapse into a parallel Wiener model.","marker":"[48]"},{"why":"Provides the Silverbox benchmark dataset used for the second experimental comparison.","marker":"[38]"},{"why":"Provides the F-16 ground vibration benchmark dataset used for the third experimental comparison.","marker":"[39]"}],"fun_headline_variants":["TCNs are Volterra series in disguise","TCN = Volterra series: new equivalence proof","Deep conv nets match classic models exactly","TCNs inherit Volterra theory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing the sigmoid activations of the time-delay neural network with ReLU activations preserves the Volterra equivalence, even though the equivalence is obtained by Taylor-expanding a smooth activation and ReLU has no Taylor expansion at its kink.","fun_headline_variants_meta":{"raw":{"variants":["TCNs are Volterra series in disguise","TCN = Volterra series: new equivalence proof","Deep conv nets match classic models exactly","TCNs inherit Volterra theory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1140,"prompt_tokens":790,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":296}},"tokens_in":406,"tokens_out":350,"duration_ms":3585,"temperature":1.0,"reasoning_tokens":296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:08:18.534451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-construct a one-layer ReLU TCN with output $\\hat{y}[k]=\\max(0,u[k]-u[k-1])$ and compare it with a Volterra series: a Volterra series is a polynomial in delayed inputs, so no convergent Volterra series can reproduce the kink along $u[k]=u[k-1]$, and demonstrating this failure would settle that the equivalence as stated needs qualification about domain or approximation topology.","supporting_citations":[{"cited_title":"An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Mod- eling,","cited_arxiv_id":null,"evidence_quote":"Defines the TCN architecture (causal dilated convolutions, residual blocks) that the paper interprets as stacked nonlinear ARX models."},{"cited_title":"Phoneme recognition using time-delay neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the time-delay neural network that the TCN is said to match except for zero padding and ReLU activations."},{"cited_title":"Calculation of the V olterra kernels of non-linear dynamic systems using an artificial neural network,","cited_arxiv_id":null,"evidence_quote":"Shows how to compute Volterra kernels from a TDNN by Taylor-expanding its sigmoid activations; this is the key equivalence the paper inherits."},{"cited_title":"Fading memory and the problem of approx- imating nonlinear operators with V olterra series,","cited_arxiv_id":null,"evidence_quote":"Establishes that Volterra series are universal approximators for time-invariant fading-memory systems, the class for which TCNs are claimed to be universal."},{"cited_title":"Giri and E.-W","cited_arxiv_id":null,"evidence_quote":"Defines block-oriented model structures such as Wiener and Hammerstein systems used in the parallel-Wiener equivalence."},{"cited_title":"Identification of block-oriented nonlinear systems starting from linear approximations: A survey,","cited_arxiv_id":null,"evidence_quote":"Surveys identification of block-oriented nonlinear systems, grounding the terminology and theory the paper connects to TCNs."},{"cited_title":"On representation and approximation of nonlinear systems,","cited_arxiv_id":null,"evidence_quote":"Shows how sandwich/block models can be squeezed together, supporting the argument that stacked TCN layers collapse into a parallel Wiener model."},{"cited_title":"Three free data sets for development and benchmarking in nonlinear system identification,","cited_arxiv_id":null,"evidence_quote":"Provides the Silverbox benchmark dataset used for the second experimental comparison."},{"cited_title":"F-16 aircraft benchmark based on ground vibration test data,","cited_arxiv_id":null,"evidence_quote":"Provides the F-16 ground vibration benchmark dataset used for the third experimental comparison."}],"review_version":1}