Pith. sign in

REVIEW 2 major objections 4 minor 73 references

Deep Convolutional Networks in System Identification

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The basic temporal convolutional network is equivalent to Volterra series and parallel Wiener models, making it a universal approximator for time-invariant systems with fading memory.

desk verdict A useful empirical bridge paper whose headline Volterra-equivalence claim is not actually proven for ReLU; worth refereeing but needs a major revision. read the letter →

arxiv 1909.01730 v2 pith:VF5GIF5W submitted 2019-09-04 eess.SY cs.LGcs.NEcs.SYstat.ML

Carl Andersson , Antônio H. Ribeiro , Koen Tiels , Niklas Wahlström , Thomas B. Schön This is my paper · ORCID
classification eess.SYcs.LGcs.NEcs.SYstat.ML
keywords temporalconvolutionalnetworksystemidentificationVolterraseriesblock-orientedmodelsparallelWienermodelnonlinearARXfadingmemorydeeplearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to connect two communities by showing that the basic temporal convolutional network (TCN) is not a new model class at all: it is the same as the classical Volterra series and parallel Wiener models. If that equivalence holds, TCNs inherit universal approximation for time-invariant systems with fading memory, and system identification theory about block-oriented models applies to modern deep sequence networks. The paper then tests TCNs against multilayer perceptrons and LSTMs on a nonlinear toy problem, the Silverbox circuit benchmark, and an F-16 ground vibration dataset, reporting that TCNs are competitive, work especially well with little training data, and do not benefit from common deep learning regularization tricks.

What carries the argument

The central object is the basic TCN, understood as a stack of causal FIR filters with pointwise ReLU nonlinearities. Two identities carry the argument: the Taylor expansion of the activation, which turns a TDNN layer into an infinite-degree Volterra series, and the shift commutation $q^{-1}\sigma(z[k])=\sigma(q^{-1}z[k])=\sigma(z[k-1])$, which lets static nonlinear blocks be moved past delays so that stacked layers collapse into a parallel Wiener model. These two moves are what turn the architectural resemblance between TCNs and old block-oriented models into a stated model-class equivalence.

What would settle it

Hand-construct a one-layer ReLU TCN with output $\hat{y}[k]=\max(0,u[k]-u[k-1])$ and compare it with a Volterra series: a Volterra series is a polynomial in delayed inputs, so no convergent Volterra series can reproduce the kink along $u[k]=u[k-1]$, and demonstrating this failure would settle that the equivalence as stated needs qualification about domain or approximation topology.

Watch

Extended reading notes

Core claim

The central claim is that a multi-layer TCN without dilations and skip connections is formally equivalent to a Volterra series of infinite degree and to a parallel Wiener model. The argument runs through the time-delay neural network: the TCN is the TDNN with zero padding and ReLU activations, and the TDNN was previously shown to compute Volterra kernels via Taylor expansions of its activation functions. The paper adds the observation that because time delays commute with static nonlinearities, the layers of a basic TCN can be squeezed together into a parallel Wiener model with one linear output layer. From this it concludes that TCN, Volterra, and parallel Wiener structures are all universal approximators for the same class of systems, while noting that training difficulty and basis-function suitability still differ.

Load-bearing premise

The load-bearing premise is that replacing the sigmoid activations of the time-delay neural network with ReLU activations preserves the Volterra equivalence, even though the equivalence is obtained by Taylor-expanding a smooth activation and ReLU has no Taylor expansion at its kink.

Editorial extensions

If this is right

  • A basic TCN can approximate any time-invariant system with fading memory to within any desired accuracy, just as Volterra series can.
  • Because causal convolutions are effectively NARX models, TCN estimates are expected to be biased when the noise is colored; the paper identifies this as the likely source of the limitations seen in its experiments.
  • On the benchmarks tested, dropout, batch normalization, and dilated convolutions did not improve TCN accuracy, while depth helped only on the F-16 problem.
  • TCNs gave the best results among the compared architectures when training data were scarce, which suggests they are a sensible default in small-data identification problems.
  • The equivalence to parallel Wiener models means structural and frequency-domain results for block-oriented systems can be carried over to TCNs trained by deep learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ReLU gap is repairable: replacing ReLU with a smooth activation such as softplus, or proving approximation directly with piecewise-linear bases, would put the Volterra equivalence on solid footing without changing the architecture's practical behavior.
  • If the equivalence holds, frequency-domain tools for block-oriented models, such as best linear approximation and nonlinear distortion analysis, become available for debugging and certifying trained TCNs, a path the paper leaves implicit.
  • The absence of any benefit from dilation on these benchmarks is likely problem-dependent; systems with genuinely long memory, such as switched or nearly periodic dynamics, are where dilated TCNs should be retested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper aims to build a bridge between deep learning and classical nonlinear system identification by studying temporal convolutional networks (TCNs). It describes the TCN as a stack of nonlinear ARX models, discusses residual blocks, and then makes theoretical connections: Section III-A claims that the basic TCN architecture (without dilation and skip connections) is equivalent to an infinite-degree Volterra series, transferring an earlier result for time-delay neural networks (TDNNs) with sigmoid activations; Section III-B argues that a multi-layer TCN can be reorganized as a parallel Wiener model; Section III-C concludes that basic TCNs, Volterra series, and parallel Wiener models are all universal approximators for time-invariant systems with fading memory. The experimental part evaluates TCN, MLP, and LSTM models on a nonlinear toy problem, the Silverbox benchmark, and an F-16 ground vibration dataset, with separate one-step-ahead and free-run simulation metrics. The code is made publicly available.

Significance. If the theoretical equivalence were rigorously established, the paper would provide a valuable conceptual link between modern deep sequence models and classical system identification structures, offering an interpretation of TCN hyperparameters (depth, kernel size, dilation) in terms of Volterra kernels and block-oriented models. The empirical study is honest and useful: the authors clearly distinguish training, validation, and test data, report both one-step-ahead and free-run errors, compare against established benchmarks, and openly document that standard deep learning techniques such as dropout and batch normalization often do not help in these identification tasks. The reproducible code is a further strength. However, the central theoretical claim is currently not supported for ReLU-based TCNs, because the transfer from the sigmoid-TDNN result in [43] to ReLU activations is made without proof and is in fact invalid as stated. Since the universal-approximation conclusion rests on this equivalence, the theoretical contribution needs substantial revision before the paper can be accepted.

major comments (2)
  1. [Section III-A] The equivalence between the basic TCN and an infinite-degree Volterra series is not established for ReLU activations. The paper states that the TCN is 'essentially the same' as the TDNN in [42] 'except for the zero padding and the use of ReLU activations instead of sigmoids', and then invokes [43] to conclude equivalence to a Volterra series. However, the proof in [43] explicitly constructs Volterra kernels by Taylor-expanding the activation function. That construction is valid for analytic activations such as sigmoids but not for ReLU, which is not analytic: its Taylor series around any bias b>0 is the linear function x, its Taylor series around b<0 is the zero function, and it is not differentiable at b=0. Consequently, the infinite-degree Volterra series obtained by the [43] construction for a ReLU neuron y[k]=ReLU(w u[k]+b) is either w u[k]+b or 0 locally, and cannot reproduce the actual ReLU output on the region where w u[k]+b changes sign. The paper provides no alternative proof that a ReLU-based TCN belongs to the class of Volterra series, nor a precise statement in terms of uniform approximation on compact sets.
  2. [Section III-C] The headline conclusion that 'The basic TCN architecture is equivalent to Volterra series and parallel Wiener models. They are thus all universal approximators for time-invariant systems with fading memory' depends directly on the Volterra equivalence discussed in Section III-A. Since that equivalence is not proven for ReLU activations, the universal-approximation claim is not supported by the argument given in the paper. The conclusion may be salvageable through a different route, for example by citing general universal-approximation results for neural networks with memory or by proving that ReLU networks can approximate fading-memory operators, but the current manuscript does not provide such a proof. This is a load-bearing gap in the central theoretical contribution.
minor comments (4)
  1. [Appendix B] In the F-16 subsection, the sentence about MLP training time is duplicated: 'Training the network with this configuration took 4 minutes. Training the network with this configuration took approximately 5 minutes.' One of the two sentences should be removed.
  2. [Tables II-IV] The experimental tables report single runs without confidence intervals or multiple-seed variation. Given that neural network training is stochastic, reporting the mean and spread over several seeds would strengthen the comparison, especially where the differences between TCN, MLP, and LSTM are small.
  3. [Table III] The comparison of RMSE values across many published methods is informative but should be read cautiously, because different methods use different sample ranges and experimental setups; the paper already partly acknowledges this by selecting 'no extrapolation' regions, but a brief sentence warning that the table is not a strictly controlled comparison would help the reader.
  4. [Section III-B] The phrase 'squeezed down to a parallel Wiener model' might be misinterpreted as implying that the number of parameters is preserved. As the authors note, the layers become 'less but larger', so the transformation is a structural reorganization rather than a parameter-preserving reduction; a more precise wording would avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the theoretical equivalences rest on external prior work, and the experiments use independent benchmark data with held-out test sets.

full rationale

The paper's derivation chain was checked for circular reductions. Section III-A claims that the basic TCN is equivalent to an infinite-degree Volterra series. This is explicitly transferred from the time-delay neural network result of Wray and Green [43], which is external prior work, and the paper quotes the key mechanism: 'The key ingredient is to use a Taylor series expansion of the activation functions sigma...' The claim is not derived from the paper's own fitted parameters or from a self-citation; it is a citation to an independent published derivation. Section III-B's connection to parallel Wiener models is a structural observation based on the definitions of TCN layers as linear dilational convolutions followed by static nonlinearities, together with the fact that time delays commute with static nonlinear functions; this is a direct equivalence by construction rather than a circular renaming. Section III-C's universal approximation conclusion cites Boyd and Chua [49], an external fading-memory result, and does not depend on the present paper's own conclusions. The experimental sections evaluate TCN, MLP, and LSTM models on the external Silverbox and F-16 benchmark datasets with explicit train/validation/test separation, and no test-set fitting is performed. The reported hyperparameter choices are selected by validation and are not the quantities being predicted. There are some self-citations, most notably to the coauthor's survey [45] and to Tiels [63], but these are used as background or as benchmark comparisons, not as load-bearing justification for the paper's central equivalence claims. A substantive correctness concern is that the Volterra-equivalence argument in [43] uses Taylor expansion of analytic activation functions, whereas TCNs use ReLU, which is not analytic; however, this is a validity gap in the transfer of an external result, not circular reasoning. The paper does not define its conclusion in terms of its premises, does not fit parameters and then rename them as predictions, and does not rely on a self-citation chain to force its result. Therefore no significant circularity is present.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The theoretical claims rely on standard approximation theorems and two prior results that are adapted to TCNs. The main unstated assumption is the transfer of the Volterra equivalence to ReLU activations, which is the most fragile step. No new entities are introduced.

free parameters (1)
  • Network hyperparameters (layers, kernel size, dropout, dilation, normalization) = varies by experiment, see Tables V and Appendix B
    Selected by grid search on validation data for each benchmark; these are model selection choices, not fitted constants in the theoretical claim.
assumptions (4)
  • standard math Neural networks are universal approximators (Hornik et al. 1989).
    Used in Section III-C to argue TCNs, Volterra series, and parallel Wiener models are all universal approximators for fading-memory systems.
  • domain assumption Fading-memory systems can be approximated by Volterra series (Boyd and Chua 1985).
    Used in Section III-C to conclude TCNs are universal approximators for fading-memory time-invariant systems.
  • ad hoc to paper The equivalence between TDNNs (sigmoid activations) and infinite-degree Volterra series, proven by Wray and Green [43], carries over to TCNs with ReLU activations.
    The paper does not prove this transfer; ReLU is not analytic, so the Taylor expansion argument in [43] does not directly apply. This is an unstated assumption.
  • domain assumption Time delays commute with the static nonlinear blocks in a convolutional layer, allowing layers to be squeezed into a parallel Wiener model.
    Used in Section III-B. The argument is valid for time-invariant convolutional layers because delays commute with the memoryless nonlinearity, but it is stated without a formal proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Convolutional Networks in System Identification." pith.science (2026). https://pith.science/paper/VF5GIF5W

@misc{pith2026190901730,
  author       = {Pith},
  title        = {Pith review of: Deep Convolutional Networks in System Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VF5GIF5W}},
  note         = {Machine review of arXiv:1909.01730}
}
read the original abstract

Recent developments within deep learning are relevant for nonlinear system identification problems. In this paper, we establish connections between the deep learning and the system identification communities. It has recently been shown that convolutional architectures are at least as capable as recurrent architectures when it comes to sequence modeling tasks. Inspired by these results we explore the explicit relationships between the recently proposed temporal convolutional network (TCN) and two classic system identification model structures; Volterra series and block-oriented models. We end the paper with an experimental study where we provide results on two real-world problems, the well-known Silverbox dataset and a newer dataset originating from ground vibration experiments on an F-16 fighter aircraft.

Figures

Figures reproduced from arXiv: 1909.01730 by the authors.

Figure 1
Figure 1. Illustration of the temporal convolution network (TCN) with residual blocks. (a) Temporal convolutional network [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (Example 1) Displays 100 samples of the free-run simulation TCN model vs the simulation of the true system. The kernel size for the causal convolutions is 2, the dropout rate is 0, it has 5 convolutional layers and a dilation rate of 1. The training set has 20 batches of 100 samples and was generated with (5) for v and w white Gaussian noise with standard deviations σv = 0.3 and σw = 0.3. The validation set has 2 ba… view at source ↗
Figure 3
Figure 3. (Example 1) Box plots showing how different design choices affect the performance of the TCN for noise standard deviation σ = 0.3 and training data length N = 2 000. On the y-axis the one-step-ahead RMSE on the validation set is displayed, and on the x-axis we have: in (a) the presence or absence of dilations; in (b) the dropout rate {0.0, 0.3, 0.5, 0.8}; in (c) the number of residual blocks {1, 2, 4, 8}; and, in (d… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (Example 2) The true output and the prediction error of the TCN model in free-run simulation for the Silverbox data. The model needs to extrapolate approximately outside the region ±0.2 marked by the dashed lines. V. CONCLUSION AND FUTURE WORK In this paper we applied …
Figure 5
Figure 5. Figure 5: (Example 3) Box plot showing how different depths of the neural network affects the performance of the TCN. Should be interpreted in the same way as [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: (Example 3) In one-step-ahead prediction mode, all tested model structures perform similar. The error is close to the noise floor around the main resonance at 7.3 Hz. (plot only at excited frequencies in [4.7, 11] Hz; true output spectrum in black, noise distortion in …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 66 canonical work pages

  1. [43]

    Calculation of the V olterra kernels of non-linear dynamic systems using an artificial neural network,

    J. Wray and G. G. R. Green, “Calculation of the V olterra kernels of non-linear dynamic systems using an artificial neural network,” Biological Cybernetics, vol. 71, no. 3, pp. 187–195, jul 1994

    work page 1994

  2. [42]

    Phoneme recognition using time-delay neural networks,

    A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, “Phoneme recognition using time-delay neural networks,”IEEE Trans. Acoust., Speech, Signal Process. , vol. 37, no. 3, pp. 328–339, 1989

    work page 1989

  3. [1]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems NIPS , 2012

    work page 2012

  4. [2]

    Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,

    G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V . Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury, “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,”IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82–97, 2012

    work page 2012

  5. [3]

    Efficient estimation of word representations in vector space,

    T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv:1301.3781, Tech. Rep., 2013

    arXiv 2013

  6. [4]

    A logical calculus of the ideas im- manent in nervous activity,

    W. S. McCulloch and W. Pitts, “A logical calculus of the ideas im- manent in nervous activity,” The bulletin of mathematical biophysics , vol. 5, no. 4, pp. 115–133, 1943

    work page 1943

  7. [5]

    Maxout Networks,

    I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y . Ben- gio, “Maxout Networks,” in Proceedings of the 30th International Conference on Machine Learning , S. Dasgupta and D. McAllester, Eds. PMLR, Feb. 2013, pp. 1319–1327

    work page 2013

  8. [6]

    Dropout: A simple way to prevent neural networks from overfitting

    N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research , vol. 15, no. 1, pp. 1929–1958, 2014

    work page 1929

Show all 73 references
  1. [7]

    Regulariza- tion of Neural Networks using DropConnect,

    L. Wan, M. Zeiler, S. Zhang, Y . Le Cun, and R. Fergus, “Regulariza- tion of Neural Networks using DropConnect,” in Proceedings of the 30th International Conference on Machine Learning , S. Dasgupta and D. McAllester, Eds. PMLR, Feb. 2013, pp. 1058–1066

    2013

  2. [8]

    Very Deep Convolutional Networks for Large-Scale Image Recognition,

    K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,”arXiv:1409.1556 [cs], Sep. 2014

    2014 arXiv

  3. [9]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015, pp. 1–9

    2015

  4. [10]

    Deep Residual Learning for Image Recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” inProc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778

    2016

  5. [11]

    Adam: A Method for Stochastic Optimiza- tion,

    D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimiza- tion,” in Proceedings of the 3rd International Conference for Learning Representations (ICLR), Dec. 2014

    2014

  6. [12]

    Optimization Methods for Large-Scale Machine Learning,

    L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization Methods for Large-Scale Machine Learning,” SIAM Review , vol. 60, no. 2, pp. 223–311, Jan. 2018

    2018

  7. [13]

    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,

    S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” in Proceed- ings of the 32nd International Conference on Machine Learning . PMLR, Jun. 2015, pp. 448–456

    2015

  8. [14]

    Rectifier nonlinearities improve neural network acoustic models,

    A. L. Maas, A. Y . Hannun, and A. Y . Ng, “Rectifier nonlinearities improve neural network acoustic models,” in In ICML Workshop on Deep Learning for Audio, Speech and Language Processing , 2013

    2013

  9. [15]

    On rectified linear units for speech processing,

    M. D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q. V . Le, P. Nguyen, A. Senior, V . Vanhoucke, J. Dean, and G. E. Hinton, “On rectified linear units for speech processing,” in 2013 IEEE Int. Conf. Acoustics, Speech and Signal Processing , May 2013, pp. 3517–3521

    2013

  10. [16]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proc. IEEE Int. Conf. Computer Vision , 2015, pp. 1026–1034

    2015

  11. [17]

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,

    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y . Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man ´e, R. Monga, S. Moore, D. Murray, C. Olah, M...

    2015

  12. [18]

    Automatic differen- tiation in PyTorch,

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differen- tiation in PyTorch,” 2017

    2017

  13. [19]

    Long Short-Term Memory,

    S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neu- ral Computation, 1997

    1997

  14. [20]

    Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks,

    C. Gulcehre, K. Cho, R. Pascanu, and Y . Bengio, “Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks,” 2014

    2014

  15. [21]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, Nov. 1998

    1998

  16. [22]

    Imagenet classification with deep convolutional neural networks,

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems , 2012, pp. 1097–1105

    2012

  17. [23]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2015, pp. 3431–3440

    2015

  18. [24]

    You only look once: Unified, real-time object detection,

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2016, pp. 779–788

    2016

  19. [25]

    An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Mod- eling,

    S. Bai, J. Z. Kolter, and V . Koltun, “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Mod- eling,” 2018

    2018

  20. [26]

    WaveNet: A Generative Model for Raw Audio,

    A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A Generative Model for Raw Audio,” arXiv:1609.03499 [cs], Sep. 2016

    2016 arXiv

  21. [27]

    Language mod- eling with gated convolutional networks,

    Y . N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language mod- eling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 933–941

    2017

  22. [28]

    Neural Machine Translation in Linear Time,

    N. Kalchbrenner, L. Espeholt, K. Simonyan, A. van den Oord, A. Graves, and K. Kavukcuoglu, “Neural Machine Translation in Linear Time,” arXiv:1610.10099 [cs], Oct. 2016

    2016 arXiv

  23. [29]

    A Convolutional Encoder Model for Neural Machine Translation,

    J. Gehring, M. Auli, D. Grangier, and Y . Dauphin, “A Convolutional Encoder Model for Neural Machine Translation,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, 2017, pp. 123–135

    2017

  24. [30]

    Identification and control of dynamical systems using neural networks,

    K. S. Narendra and K. Parthasarathy, “Identification and control of dynamical systems using neural networks,” IEEE Trans. Neural Netw., vol. 1, no. 1, pp. 4–27, 1990

    1990

  25. [31]

    Non-linear system identifica- tion using neural networks,

    S. Chen, S. A. Billings, and P. M. Grant, “Non-linear system identifica- tion using neural networks,” International Journal of Control, vol. 51, no. 6, pp. 1191–1214, 1990

    1990

  26. [32]

    Nonlinear black-box modeling in system identification: a unified overview,

    J. Sj ¨oberg, Q. Zhang, L. Ljung, A. Benveniste, B. Delyon, P.-Y . Glorennec, H. Hjalmarsson, and A. Juditsky, “Nonlinear black-box modeling in system identification: a unified overview,” Automatica, 1995

    1995

  27. [33]

    Improved learning algorithm for two-layer neural networks for identification of nonlinear systems,

    J. A. Vargas, W. Pedrycz, and E. M. Hemerly, “Improved learning algorithm for two-layer neural networks for identification of nonlinear systems,” Neurocomputing, vol. 329, pp. 86–96, Feb. 2019

    2019

  28. [34]

    Learning Nonlinear State-Space Models Using Deep Autoencoders,

    D. Masti and A. Bemporad, “Learning Nonlinear State-Space Models Using Deep Autoencoders,” in 2018 IEEE Conf. Decision and Control (CDC), 17, pp. 3862–3867

    2018

  29. [35]

    Stable Kalman filter and neural network for the chaotic systems identification,

    J. de Jes ´us Rubio, “Stable Kalman filter and neural network for the chaotic systems identification,” Journal of the Franklin Institute , vol. 354, no. 16, pp. 7444–7462, Nov. 2017

    2017

  30. [36]

    Com- parative study of neural networks for dynamic nonlinear systems identification,

    R. Kumar, S. Srivastava, J. R. P. Gupta, and A. Mohindru, “Com- parative study of neural networks for dynamic nonlinear systems identification,” Soft Computing, vol. 23, no. 1, pp. 101–114, Jan. 2019

    2019

  31. [37]

    Generalized Hybrid Constructive Learning Algorithm for Multioutput RBF Networks,

    X. Qian, H. Huang, X. Chen, and T. Huang, “Generalized Hybrid Constructive Learning Algorithm for Multioutput RBF Networks,” IEEE Trans. Cybern. , vol. 47, no. 11, pp. 3634–3648, Nov. 2017

    2017

  32. [38]

    Three free data sets for development and benchmarking in nonlinear system identification,

    T. Wigren and J. Schoukens, “Three free data sets for development and benchmarking in nonlinear system identification,” in 2013 European Control Conference (ECC), 2013

    2013

  33. [39]

    F-16 aircraft benchmark based on ground vibration test data,

    M. Schoukens and J.-P. No ¨el, “F-16 aircraft benchmark based on ground vibration test data,” in Workshop on Nonlinear System Identi- fication Benchmarks, Brussels, Belgium, 2017

    2017

  34. [40]

    Multilayer feedforward networks are universal approximators,

    K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, jan 1989

    1989

  35. [41]

    Schetzen, The Volterra & Wiener Theories of Nonlinear Systems

    M. Schetzen, The Volterra & Wiener Theories of Nonlinear Systems . Malabar, Florida: Krieger Publishing Company, 2006

    2006

  36. [44]

    Giri and E.-W

    F. Giri and E.-W. Bai, Eds., Block-oriented Nonlinear System Identi- fication. Springer London, 2010

    2010

  37. [45]

    Identification of block-oriented nonlinear systems starting from linear approximations: A survey,

    M. Schoukens and K. Tiels, “Identification of block-oriented nonlinear systems starting from linear approximations: A survey,” Automatica, vol. 85, pp. 272–292, nov 2017

    2017

  38. [46]

    Generalised Hammerstein–Wiener system estimation and a benchmark application,

    A. Wills and B. Ninness, “Generalised Hammerstein–Wiener system estimation and a benchmark application,” Control Engineering Prac- tice, vol. 20, no. 11, pp. 1097–1108, nov 2012

    2012

  39. [47]

    Parametric identification of parallel Wiener-Hammerstein systems,

    M. Schoukens, A. Marconato, R. Pintelon, G. Vandersteen, and Y . Rolain, “Parametric identification of parallel Wiener-Hammerstein systems,” Automatica, vol. 51, pp. 111–122, Jan. 2015

    2015

  40. [48]

    On representation and approximation of nonlinear systems,

    G. Palm, “On representation and approximation of nonlinear systems,” Biological Cybernetics, vol. 34, no. 1, pp. 49–52, 1979

    1979

  41. [49]

    Fading memory and the problem of approx- imating nonlinear operators with V olterra series,

    S. Boyd and L. O. Chua, “Fading memory and the problem of approx- imating nonlinear operators with V olterra series,” IEEE Transactions on Circuits and Systems , vol. CAS-32, no. 11, pp. 1150–1161, 1985

    1985

  42. [50]

    Identification of local linear state-space models: The Silver-box case study,

    V . Verdult, “Identification of local linear state-space models: The Silver-box case study,” IFAC Proceedings Volumes, vol. 37, no. 13, pp. 393–398, sep 2004

    2004

  43. [51]

    Identification of the Silverbox benchmark using nonlinear state-space models,

    A. Marconato, J. Sj ¨oberg, J. Suykens, and J. Schoukens, “Identification of the Silverbox benchmark using nonlinear state-space models,” IFAC Proceedings Volumes, vol. 45, no. 16, pp. 632–637, jul 2012

    2012

  44. [52]

    Wiener system identification with generalized orthonormal basis functions,

    K. Tiels, “Wiener system identification with generalized orthonormal basis functions,” Ph.D. dissertation, Vrije Universiteit Brussel, 2015

    2015

  45. [53]

    On direct identification of physical parameters in non-linear models,

    H. Hjalmarsson and J. Schoukens, “On direct identification of physical parameters in non-linear models,” IFAC Proceedings Volumes, vol. 37, no. 13, pp. 375–380, sep 2004

    2004

  46. [54]

    Fast identification of systems with nonlinear feedback,

    J. Paduart, G. Horv ´ath, and J. Schoukens, “Fast identification of systems with nonlinear feedback,” IFAC Proceedings Volumes, vol. 37, no. 13, pp. 381–385, sep 2004

    2004

  47. [55]

    Estimation of grey box and black box models for non-linear circuit data,

    L. Ljung, Q. Zhang, P. Lindskog, and A. Juditski, “Estimation of grey box and black box models for non-linear circuit data,” IFAC Proceedings Volumes, vol. 37, no. 13, pp. 399–404, sep 2004

    2004

  48. [56]

    A comparative study of LS-SVM’s applied to the Silver Box identification problem,

    M. Espinoza, K. Pelckmans, L. Hoegaerts, J. A. Suykens, and B. De Moor, “A comparative study of LS-SVM’s applied to the Silver Box identification problem,” IFAC Proceedings Volumes , vol. 37, no. 13, pp. 369–374, sep 2004

    2004

  49. [57]

    Identification of nonlinear systems using polynomial nonlinear state space models,

    J. Paduart, “Identification of nonlinear systems using polynomial nonlinear state space models,” Ph.D. dissertation, Vrije Universiteit Brussel, 2008

    2008

  50. [58]

    Identification of systems with localised nonlinearity: From state-space to block- structured models,

    A. Van Mulders, J. Schoukens, and L. Vanbeylen, “Identification of systems with localised nonlinearity: From state-space to block- structured models,” Automatica, vol. 49, no. 5, pp. 1392–1396, 2013

    2013

  51. [59]

    Kernel based par- tially linear models and nonlinear identification,

    M. Espinoza, J. A. Suykens, and B. De Moor, “Kernel based par- tially linear models and nonlinear identification,” IEEE Trans. Autom. Control, vol. 50, no. 10, pp. 1602–1606, oct 2005

    2005

  52. [60]

    Modelling of a slightly nonlinear system: a neural network approach,

    L. Sragner, J. Schoukens, and G. Horv ´ath, “Modelling of a slightly nonlinear system: a neural network approach,” IFAC Proceedings Volumes, vol. 37, no. 13, pp. 387–392, sep 2004

    2004

  53. [61]

    Identification of piecewise affine LFR models of interconnected systems,

    E. Pepona, S. Paoletti, A. Garulli, and P. Date, “Identification of piecewise affine LFR models of interconnected systems,” IEEE Trans. Control Syst. Technol., vol. 19, no. 1, pp. 148–155, jan 2011

    2011

  54. [62]

    Extended fuzzy logic: Sets and systems,

    F. Sabahi and M. R. Akbarzadeh-T, “Extended fuzzy logic: Sets and systems,” IEEE Trans. Fuzzy Syst. , vol. 24, no. 3, pp. 530–543, 2016

    2016

  55. [63]

    Polynomial nonlinear state-space modeling of the F-16 aircraft benchmark,

    K. Tiels, “Polynomial nonlinear state-space modeling of the F-16 aircraft benchmark,” in Workshop on Nonlinear System Identification Benchmarks, Brussels, Belgium, 2017

    2017

  56. [64]

    Pintelon and J

    R. Pintelon and J. Schoukens, System identification: A frequency domain approach, 2nd ed. Wiley-IEEE Press, 2012

    2012

  57. [65]

    Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,

    T. Salimans and D. P. Kingma, “Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,” 2016

    2016

  58. [66]

    On the momentum term in gradient descent learning algorithms,

    N. Qian, “On the momentum term in gradient descent learning algorithms,” Neural networks, vol. 12, no. 1, pp. 145–151, 1999

    1999

  59. [67]

    Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude,

    T. Tieleman and G. Hinton, “Lecture 6.5—RmsProp: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural Networks for Machine Learning, 2012. APPENDIX A. Neural networks common practices

    2012

  60. [68]

    A number of other regularization techniques have also appeared more specialized to neural networks

    Regularization: Similar to other approaches within sys- tem identification, L2- and L1-regularization are commonly used to reduce the flexibility of a model and hence avoid overfitting. A number of other regularization techniques have also appeared more specialized to neural netw...

  61. [69]

    The purpose of this is to avoid early saturation of the activation function and assuring that values in the proceeding layers are within the same dynamic range

    Batch Normalization: Before training a neural net- work, the inputs are commonly normalized by subtracting the mean and dividing by the variance. The purpose of this is to avoid early saturation of the activation function and assuring that values in the proceeding layers are w...

  62. [70]

    At each iteration only a random subset of the training data is used to com- pute the gradient and update the parameters

    Optimization Algorithms: Neural networks are trained using gradient-based optimization methods. At each iteration only a random subset of the training data is used to com- pute the gradient and update the parameters. This is called mini-batch gradient descent and is a crucial ...

  63. [71]

    Nonlinear toy problem: We used grid search for finding the hyperparameters. In each possible training configuration, we have trained the TCN for all possible combinations of: number of hidden layers in {16, 32, 64, 128, 256}; dropout rate in {0.0, 0.3, 0.5, 0.8 }; number of resi...

  64. [72]

    For LSTM and MLP, dropout was disabled this way

    Silverbox: Some hyperparameters were just experi- enced with manually to find good values and did not effect the results in any major fashion. For LSTM and MLP, dropout was disabled this way. For the TCN, the kernel size was set to 2 after initial experimentation. The number of...

  65. [73]

    F-16 ground vibration test: Again, we used grid search for finding the hyperparameters. In each possible training configuration, we have trained the TCN for all possible combinations of: number of hidden layers in {16, 32, 64, 128}; dropout rate in{0.0, 0.3, 0.5, 0.8}; number of...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.