Pith. sign in

REVIEW 2 major objections 4 minor 61 references

The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that Transformer self-attention is a particular instance of a broader affinity-matrix framework introduced by Infinite Feature Selection, not a structurally new mechanism.

desk verdict Punchline: the paper's historical/conceptual reframing is reasonable and honestly hedged, but the 'superset/special case' claim is overbroad as stated; a 'structural analogy' framing would make it defensible. read the letter →

arxiv 2507.14560 v2 pith:UTOL5HBA submitted 2025-07-19 cs.LG cs.CV

Giorgio Roffo This is my paper · ORCID
classification cs.LGcs.CV MSC 05C5015A1868T0568T0768R10
keywords self-attentionaffinitymatrixinfinitefeatureselectionTransformerpairwiserelevanceattentionmechanismgraph-basedlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that Transformer self-attention is not a standalone invention but a particular instance of a much older computational pattern: using a pairwise affinity matrix $A$ to decide how much each element should influence every other. Its central claim is that Infinite Feature Selection (Inf-FS), a 2015 feature-ranking method, is the broader paradigm, and that self-attention arises when $A$ is learned from token similarities and applied in a single hop rather than through an infinite path sum. If the claim holds, the affinity matrix becomes a common mathematical core that unifies feature selection, sequence modeling, vision, and graph learning, and design choices from one field can transfer to the others. The paper does not claim Inf-FS invented attention; it claims Inf-FS introduced the structural template that attention later operationalized.

What carries the argument

The carrying object is the pairwise affinity matrix $A\in\mathbb{R}^{N\times N}$ together with the geometric power-series identity $S=\sum_{k\geq 1}\alpha^k A^k=(I-\alpha A)^{-1}-I$, which sums contributions of all paths in the affinity graph. In Inf-FS this yields per-feature importance by multi-hop propagation; in self-attention, the same $A$ appears as $QK^\top/\sqrt{d_k}$ and is applied once through a row-wise softmax and value mixing. The power series is what lets the paper call Inf-FS the general case: truncating the series at one hop and replacing the fixed affinity by learned token similarities is, in the paper's reading, exactly the Transformer operation.

What would settle it

One concrete check is to run a single self-attention layer and an Inf-FS one-hop computation on the same token embeddings: Inf-FS with one hop returns a scalar score per element, while attention returns a vector per token obtained by mixing value vectors, so if the value projection is essential to task performance, attention is not reproduced by the Inf-FS operation and the containment claim reduces to a shared design motif.

Watch

Extended reading notes

Core claim

The discovery claimed is a lineage and containment relationship: Inf-FS's scoring function $S=(I-\alpha A)^{-1}-I$ over a fully connected feature graph and Transformer attention's $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ are two realizations of one affinity-based aggregation scheme. Inf-FS allows $A$ to be handcrafted or learned and propagates relevance over paths of arbitrary length; self-attention computes $A$ dynamically as scaled dot products and applies it in one hop, with depth coming from stacked layers. Because the Transformer's $A$ is a special way of parameterizing the same pairwise matrix, the paper concludes that self-attention can be viewed as a particular instantiation of the broader Inf-FS paradigm.

Load-bearing premise

The whole containment argument rests on the idea that sharing a pairwise matrix $A$ is the defining feature of these computations, even though self-attention adds value projections, row-wise softmax, and vector outputs that Inf-FS does not have.

Editorial extensions

If this is right

  • If self-attention is a special case of Inf-FS, then results about convergence, normalization, and multi-hop propagation from feature selection can be imported directly into attention design.
  • Attention's single-hop restriction becomes a truncation choice rather than a distinct mechanism, so expanding the Inf-FS power series inside a neural network would yield a principled multi-hop attention layer with closed-form aggregation.
  • The affinity-matrix reading gives a unified vocabulary for comparing attention across NLP, vision, and graphs, potentially simplifying transfer of design tricks among those domains.
  • Learned per-instance affinity in Inf-FS extensions implies that dynamic, input-dependent weighting usually credited to attention also existed in feature-selection work before the Transformer.
  • If the containment is taken formally, any improvement to Inf-FS's construction or normalization of $A$ could be applied to attention layers without changing the overall architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 'superset' wording is structural rather than algebraic: self-attention outputs a vector per token through value projections, while Inf-FS outputs a scalar per feature, so the containment is exact only if the value path is treated as incidental; I read the intended claim as a shared design pattern rather than formal set inclusion.
  • A testable extension would be to train a single-hop learned-affinity feature reweighting module and compare it head-to-head with an attention head on the same inputs; if performance and learned patterns match, the lineage claim becomes empirically meaningful rather than purely historical.
  • The same structural lens would also make kernel smoothing, PageRank, and bilateral filtering 'special cases' of Inf-FS, which suggests the framework is best used as an organizing perspective rather than a uniqueness claim about any one ancestor.
  • Attention's softmax is a row-stochastic normalization, whereas Inf-FS uses a fixed $\alpha$ decay; a direct bridge would be to interpret softmax as a data-dependent $\alpha$ or to show that $\alpha$ can be replaced by a row-wise softmax of the infinite series, which the paper does not prove.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper argues that Transformer self-attention is a modern instantiation of a broader principle of pairwise affinity-based computation, with Infinite Feature Selection (Inf-FS) identified as a foundational framework. It presents a historical timeline across feature selection, NLP, computer vision, and graph learning, and includes a structural comparison table and a discussion of differentiable feature selection gates. The central claim, stated in the abstract, body, and conclusion, is that self-attention is a special case of Inf-FS, making Inf-FS a superset framework.

Significance. If the formal containment claim were established, the paper would provide a unifying mathematical perspective on self-attention and feature selection. The manuscript is valuable as a conceptual and historical survey: it correctly states the Inf-FS power-series formula, restates the scaled dot-product attention formula, and points to relevant prior work, including the Non-local Neural Networks paper's own acknowledgment that self-attention is a special case of non-local operations. The paper is also explicit in disclaiming that Inf-FS introduced self-attention in its modern form. However, the headline 'superset/special case' claim is currently not derived, and the manuscript oscillates between a historical-narrative claim and a formal-mathematical claim. The fix is within scope: either supply a reduction or explicitly reframe the contribution as conceptual lineage.

major comments (2)
  1. [Section 2, paragraph on one-hop Inf-FS] The claimed bridge to self-attention is only an analogy. The text says that limiting Inf-FS to path length 1 gives a weighted degree sum and that this is 'structurally analogous' to a single self-attention layer, but one-hop Inf-FS yields a scalar score per feature, not a vector, and uses no softmax or value projections. The shared symbol A is not sufficient to embed the attention computation into Inf-FS. If the one-hop equivalence is intended as a formal statement, the authors need to construct a concrete mapping from the attention update z_i = Σ_j softmax(row)_ij V_j to an Inf-FS-style aggregation, or explicitly state that the relationship is at the level of shared mathematical motivation.
  2. [Structural Comparison table, Learnability and Pairwise Score rows] The table overstates the historical record in ways that affect the argument. The Learnability row says Inf-FS 'is a general paradigm' whose matrix A 'can be handcrafted or learned (e.g., ICCV 2017, TPAMI 2020)', but the original ICCV 2015 Inf-FS used a fixed statistical affinity; the learnable extension is a later contribution. The Pairwise Score row says Inf-FS scores 'can be dynamic, recomputed for each input sequence,' but the original method computes a static dataset-level affinity. These conflations make the 'superset' claim appear stronger than the cited papers support and should be corrected or clearly qualified.
minor comments (4)
  1. [Structural Comparison table, Learnability row] Typo: 'Subset of of the infFS formulation' should read 'Subset of the Inf-FS formulation.'
  2. [Inf-FS vs. Self-Attention: Structural Comparison] The sentence 'The table above summarizes these analogies' appears before the table, which is placed at the end of the document after the references; either move the table above that sentence or change the pointer.
  3. [References] Several references are duplicated: Vaswani et al. appears as [5] and [61], and Bahdanau et al. appears as [6] and [45]; also AFS appears as [27] and [34]/[59]. Please consolidate the bibliography.
  4. [Introduction, timeline entry for 2014] The timeline cites the Attentional Neural Network as [19], but the body text later cites it as [38]; ensure consistent citation numbering.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'Inf-FS is a superset' claim is asserted by defining the Inf-FS framework so broadly that any use of an affinity matrix A is included; the paper never reduces softmax attention to the Inf-FS scoring equation.

  1. self definitional [Section 'Inf-FS vs. Self-Attention: Structural Comparison'; echoed in Conclusion]
    "From a methodological perspective, Inf-FS offers a general framework for defining and using A, which can include the specific case of self-attention when A is constructed through learned dot-product similarity."

    The sentence defines the 'general framework' as any use of an affinity matrix A, then declares self-attention an instance because it constructs A by dot-product. The Conclusion restates this as 'This flexibility makes Inf-FS a superset framework—one that includes as a special case modern attention mechanisms.' The containment is therefore true by stipulation, not by reduction. The paper's own structural-comparison table records incompatible maps: Inf-FS produces a single relevance score per feature via S=(I−αA)^{-1}−I, while self-attention produces per-token vectors Z=softmax(QK^T/√d)V, with softmax instead of a fixed α and one-hop rather than infinite paths.

full rationale

The paper's historical and survey content is not circular: it assembles an independently checkable timeline of affinity-matrix methods, cites peer-reviewed prior work for Inf-FS, and explicitly disclaims that Inf-FS invented self-attention. There are no fitted parameters, no predictions from data, and no uniqueness theorems imported from the authors. The circularity is confined to the central methodological claim that self-attention is a special case of Inf-FS. That claim is made true only by broadening the meaning of 'Inf-FS' to 'a general framework for defining and using A,' and the paper's own table concedes the equation-level differences between the two computations. Because the central containment result reduces to a definitional move, the score is 6; it is not higher because the surrounding lineage argument rests on external, falsifiable references rather than on a self-citation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are introduced in this paper. The alpha decay parameter appears in the Inf-FS equation but is inherited from the cited method, not fitted here. The axioms listed are the mathematical and interpretive assumptions the central claim relies on.

assumptions (3)
  • standard math The Neumann series S = sum_{k>=1} (alpha A)^k equals (I - alpha A)^-1 - I for 0 < alpha < 1/rho(A).
    Invoked in Section 2 and in the timeline equation to define Inf-FS scoring.
  • domain assumption A softmax-normalized dot product of queries and keys constitutes an affinity matrix that plays the same role as Inf-FS's A.
    This is the interpretative bridge that lets the paper map self-attention onto Inf-FS; it is presented as natural rather than proved.
  • ad hoc to paper A structural similarity between two pairwise computations is sufficient to call one a superset of the other.
    The paper's central thesis depends on this inference, but no formal subsumption is provided; the equations show different outputs for the two methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention." pith.science (2026). https://pith.science/paper/UTOL5HBA

@misc{pith2026250714560,
  author       = {Pith},
  title        = {Pith review of: The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UTOL5HBA}},
  note         = {Machine review of arXiv:2507.14560}
}
read the original abstract

The self-attention mechanism, now central to deep learning architectures such as Transformers, is a modern instance of a more general computational principle: learning and using pairwise affinity matrices to control how information flows through a model. This paper traces the conceptual origins of self-attention across multiple domains, including computer vision, natural language processing, and graph learning, through their shared reliance on an affinity matrix, denoted as A. We highlight Infinite Feature Selection (Inf-FS) as a foundational approach that generalizes the idea of affinity-based weighting. Unlike the fixed dot-product structure used in Transformers, Inf-FS defines A either through domain knowledge or by learning, and computes feature relevance through multi-hop propagation over the affinity graph. From this perspective, self-attention can be seen as a special case of Inf-FS: it uses a single-hop affinity computation where A is dynamically built from token similarities. We argue that the underlying structure, reasoning over pairwise relationships, is preserved across both approaches, and the key differences lie in how the affinity matrix is defined and applied. By situating self-attention within the broader paradigm of affinity-based computation, we unify several strands of machine learning research and highlight a common mathematical foundation that underpins diverse models and tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 60 canonical work pages

  1. [1]

    Roffo, S

    G. Roffo, S. Melzi, and M. Cristani,Infinite feature selection, ICCV, 2015

    work page 2015

  2. [2]

    Roffo and S

    G. Roffo and S. Melzi,Infinite latent feature selection: A probabilistic latent graph-based ranking approach, ICCV, 2017

    work page 2017

  3. [3]

    Roffo, S

    G. Roffo, S. Melzi, and M. Cristani,Infinite feature selection, IEEE TPAMI, 2021

    work page 2021

  4. [4]

    Roffo,Inf-FS Codebase and extensions, GitHub repository:https://github.com/ giorgioroffo/inf-FS, accessed 2025

    G. Roffo,Inf-FS Codebase and extensions, GitHub repository:https://github.com/ giorgioroffo/inf-FS, accessed 2025

    work page 2025

  5. [5]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin,Attention is all you need, NeurIPS, 2017

    work page 2017

  6. [6]

    Bahdanau, K

    D. Bahdanau, K. Cho, and Y. Bengio,Neural machine translation by jointly learning to align and translate, ICLR, 2015

    work page 2015

  7. [7]

    Tomasi and R

    C. Tomasi and R. Manduchi,Bilateral filtering for gray and color images, ICCV, 1998

    work page 1998

  8. [8]

    Buades, B

    A. Buades, B. Coll, and J.-M. Morel,A non-local algorithm for image denoising, CVPR, 2005

    work page 2005

Show all 61 references
  1. [9]

    L. Page, S. Brin, R. Motwani, and T. Winograd,The PageRank citation ranking: Bringing order to the web, Technical Report, Stanford InfoLab, 1998

    1998

  2. [10]

    Roffo and S

    G. Roffo and S. Melzi,Features selection via eigenvector centrality, New Frontiers in Mining Complex Patterns ECML/PKDD, 2016

    2016

  3. [11]

    Roffo,Ranking to learn and learning to rank: On the role of ranking in pattern recognition applications, PhD Thesis, University of Verona

    G. Roffo,Ranking to learn and learning to rank: On the role of ranking in pattern recognition applications, PhD Thesis, University of Verona

  4. [12]

    Roffo and S

    G. Roffo and S. Melzi,Online feature selection for visual tracking, BMVC, 2016

    2016

  5. [13]

    Obertino, G

    S. Obertino, G. Roffo, C. Granziera, and G. Menegaz,Infinite feature selection on shore- based biomarkers reveals connectivity modulation after stroke, PRNI, 2016

    2016

  6. [14]

    Roffo,Exploring advanced large language models with llmsuite, arXiv preprint arXiv:2407.12036, 2024

    G. Roffo,Exploring advanced large language models with llmsuite, arXiv preprint arXiv:2407.12036, 2024

    2024 arXiv

  7. [15]

    Roffo, C

    G. Roffo, C. Biffi, P. Salvagnini, and A. Cherubini,Feature selection gates with gradient routing for endoscopic image computing, MICCAI, 2024. 20 AFFINITY-BASED COMPUTATION

    2024

  8. [16]

    Roffo, C

    G. Roffo, C. Biffi, P. Salvagnini, and A. Cherubini,Hard-attention gates with gradient routing for endoscopic image computing, arXiv preprint arXiv:2407.04400, 2024

    2024 arXiv

  9. [17]

    Roffo,Feature selection library: A widely applicable MATLAB library for feature selec- tion, arXiv preprint arXiv:1607.01327, 2016

    G. Roffo,Feature selection library: A widely applicable MATLAB library for feature selec- tion, arXiv preprint arXiv:1607.01327, 2016

    2016 arXiv

  10. [18]

    Roffo,Feature Selection Library (MATLAB Toolbox)

    G. Roffo,Feature Selection Library (MATLAB Toolbox)

  11. [19]

    Q. Wang, J. Zhang, S. Song, and Z. Zhang,Attentional Neural Network: Feature Selection Using Cognitive Feedback, NeurIPS, 2014

    2014

  12. [20]

    Cheng, L

    J. Cheng, L. Dong, and M. Lapata,Long short-term memory-networks for machine reading, ACL, 2016

    2016

  13. [21]

    Paulus, C

    R. Paulus, C. Xiong, and R. Socher,A deep reinforced model for abstractive summarization, ICLR, 2018

    2018

  14. [22]

    Parikh, O

    A. Parikh, O. T¨ ackstr¨ om, D. Das, and J. Uszkoreit,A decomposable attention model for natural language inference, EMNLP, 2016

    2016

  15. [23]

    Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio,A structured self-attentive sentence embedding, ICLR, 2017

    2017

  16. [24]

    Santoro, D

    A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, P. Battaglia, and T. Lillicrap,A simple neural network module for relational reasoning, NeurIPS, 2017

    2017

  17. [25]

    J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh,Set transformer: A framework for attention-based permutation-invariant neural networks, ICML, 2019

    2019

  18. [26]

    K. Roy, M. S. Charikar, and A. Singh,Unsupervised feature selection using attention-based neural networks, arXiv:1811.03846, 2018

    2018 arXiv

  19. [27]

    J. Gui, T. Liu, Z. Sun, D. Tao, and T. Tan,AFS: An attention-based mechanism for supervised feature selection, AAAI, 2019

    2019

  20. [28]

    A. Abid, M. Balin, and J. Zou,Concrete autoencoders for differentiable feature selection and reconstruction, ICML, 2019

    2019

  21. [29]

    Gui et al.,Feature selection based on structured sparsity: A comprehensive study, AAAI, 2019

    J. Gui et al.,Feature selection based on structured sparsity: A comprehensive study, AAAI, 2019

    2019

  22. [30]

    X. Wang, R. Girshick, A. Gupta, and K. He,Non-local neural networks, CVPR, 2018

    2018

  23. [31]

    Veliˇ ckovi´ c, G

    P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Li` o, and Y. Bengio,Graph attention networks, ICLR, 2018

    2018

  24. [32]

    Kitaev, L

    N. Kitaev, L. Kaiser, and A. Levskaya,Reformer: The efficient transformer, ICLR, 2020

    2020

  25. [33]

    S. Wang, B. Li, M. Khabsa, H. Fang, and H. Ma,Linformer: Self-attention with linear complexity, arXiv:2006.04768, 2020

    2006 arXiv

  26. [34]

    AFS: An Attention-Based Mechanism for Supervised Feature Selection,

    J. Gui, T. Liu, Z. Sun, D. Tao, and T. Tan, “AFS: An Attention-Based Mechanism for Supervised Feature Selection,” inProceedings of the AAAI Conference on Artificial Intel- ligence, vol. 33, no. 1, pp. 3705–3713, 2019

    2019

  27. [35]

    Sequential Self-Attention for Progressive Feature Selec- tion,

    T. Zhao, Q. Kong, and Y. Wang, “Sequential Self-Attention for Progressive Feature Selec- tion,” inInternational Conference on Learning Representations (ICLR), 2023

    2023

  28. [36]

    Choromanski, V

    K. Choromanski, V. Likhosherstov, D. Dohan, et al.,Rethinking attention with performers, ICLR, 2021

    2021

  29. [37]

    Z. Shen, M. Zhang, J. Sun, et al.,Efficient attention: Attention with linear complexities, W ACV, 2021

    2021

  30. [38]

    X. Wang, R. Girshick, A. Gupta, and K. He,An attentional neural network for image classification, NeurIPS, 2014

    2014

  31. [39]

    Ramsauer, B

    H. Ramsauer, B. Sch¨ afl, J. Lehner, et al.,Hopfield networks is all you need, ICLR, 2021

    2021

  32. [40]

    Huang, X

    Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu,CCNet: Criss-cross attention for semantic segmentation, ICCV, 2019

    2019

  33. [41]

    J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu,Dual attention network for scene segmentation, CVPR, 2019

    2019

  34. [42]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al.,An image is worth 16x16 words: Trans- formers for image recognition at scale, ICLR, 2021

    2021

  35. [43]

    Z. Liu, Y. Lin, Y. Li, F. Zhou, and X. Sun,Learning Node Representations with Latent Graphs, NeurIPS, 2019. THE ROOTS OF SELF-ATTENTION 21

    2019

  36. [44]

    Chandrashekar, M

    R. Chandrashekar, M. Huang, and Q. Liu,A survey on feature selection methods, arXiv:2205.03466, 2022

    2022 arXiv

  37. [45]

    Bahdanau, K

    D. Bahdanau, K. Cho, and Y. Bengio,Neural Machine Translation by Jointly Learning to Align and Translate, ICLR, 2015

    2015

  38. [46]

    On Spectral Clustering: Analysis and an algorithm,

    A. Y. Ng, M. I. Jordan, and Y. Weiss, “On Spectral Clustering: Analysis and an algorithm,” in *Advances in Neural Information Processing Systems (NeurIPS)*, 2002

    2002

  39. [47]

    A tutorial on spectral clustering,

    U. von Luxburg, “A tutorial on spectral clustering,” *Statistics and Computing*, vol. 17, no. 4, pp. 395–416, 2007

    2007

  40. [48]

    The structure and function of complex networks,

    M. E. J. Newman, “The structure and function of complex networks,” *SIAM Review*, vol. 45, no. 2, pp. 167–256, 2003

    2003

  41. [49]

    Power and centrality: A family of measures,

    P. Bonacich, “Power and centrality: A family of measures,” *American Journal of Sociol- ogy*, vol. 92, no. 5, pp. 1170–1182, 1987

    1987

  42. [50]

    Learning with local and global consistency,

    D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Sch¨ olkopf, “Learning with local and global consistency,” in *NeurIPS*, 2004

    2004

  43. [51]

    Infinite Feature Selection,

    G. Roffo, S. Melzi, and M. Cristani, “Infinite Feature Selection,” in *IEEE International Conference on Computer Vision (ICCV)*, 2015

    2015

  44. [52]

    Some Experiments on the Recognition of Speech, with One and with Two Ears,

    E. C. Cherry, “Some Experiments on the Recognition of Speech, with One and with Two Ears,”Journal of the Acoustic Society of America, vol. 25, no. 5, pp. 975–979, 1953

    1953

  45. [53]

    D. E. Broadbent,Perception and Communication. Pergamon Press, 1958

    1958

  46. [54]

    The Information Available in Brief Visual Presentations,

    G. Sperling, “The Information Available in Brief Visual Presentations,”Psychological Monographs: General and Applied, vol. 74, no. 11, pp. 1–29, 1960

    1960

  47. [55]

    Learning Internal Representations by Error Propagation,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning Internal Representations by Error Propagation,” inParallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, MIT Press, 1986, pp. 318–362. [See sigma-pi units, p. 329–330]

    1986

  48. [56]

    Learning, Invariance, and Generalization in High-Order Neural Networks,

    C. L. Giles and T. Maxwell, “Learning, Invariance, and Generalization in High-Order Neural Networks,”Applied Optics, vol. 26, no. 23, pp. 4972–4978, 1987

    1987

  49. [57]

    Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks,

    J. Schmidhuber, “Learning to Control Fast-Weight Memories: An Alternative to Dynamic Recurrent Networks,”Neural Computation, vol. 4, no. 1, pp. 131–139, 1992

    1992

  50. [58]

    Infinite Feature Selection,

    G. Roffo, S. Melzi, and M. Cristani, “Infinite Feature Selection,” *IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)*, vol. 43, no. 7, pp. 2436–2450, 2021

    2021

  51. [59]

    AFS: An Attention-Based Mechanism for Supervised Feature Selection,

    J. Gui, T. Liu, Z. Sun, D. Tao, and T. Tan, “AFS: An Attention-Based Mechanism for Supervised Feature Selection,” in *AAAI Conference on Artificial Intelligence*, 2019

    2019

  52. [60]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in *International Conference on Learning Representations (ICLR)*, 2015

    2015

  53. [61]

    Attention is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is All You Need,” in *NeurIPS*, 2017. Email address:giorgio.roffo@gmail.com 22 AFFINITY-BASED COMPUTATION Aspect Infinite Feature Selection (ICCV 2015) Self-Attention (Transform- ers, 2017) Underlying Graph Fully-connected ...

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.