Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

Lilian Hollard; Lucas Mohimont; Luiz-Angelo Steffenel; Nathalie Gaveau

REVIEW 4 major objections 4 minor 39 references

Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that sub-1.5M-parameter vision models built on inverted bottlenecks scale poorly because they depend on feature-map interference, and that suppressing this dependence through architectural design improves scaling.

desk verdict A genuinely interesting empirical hypothesis about interference in low-parameter bottlenecks, undermined by a confounded intervention and missing reproducibility details. read the letter →

arxiv 2507.15798 v1 pith:OBM7A5YQ submitted 2025-07-21 cs.CV

Lilian Hollard , Lucas Mohimont , Nathalie Gaveau , Luiz-Angelo Steffenel This is my paper

classification cs.CV

keywords superpositioninterferencelow-parameterneuralnetworksbottleneckarchitecturessuperlinearactivationSoLUmechanisticinterpretabilityscaling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain why many low-parameter vision networks (under 1.5M parameters) scale much worse than expected, and it pins the cause on feature-map interference, a form of superposition in which one neuron encodes several features at once. The authors insert a superlinear activation, the SoLU module, into the residual stream of different bottleneck architectures and measure how much accuracy drops. Inverted bottlenecks (MobileNetv2, MobileNetv3, and MobileNeXt with Batch Norm) lose roughly half their accuracy, while architectures whose depthwise convolution stays in the input channel dimension lose only a few points. From this they conclude that interference dependence is what limits scaling and that reducing it improves accuracy in the very low-parameter regime. As a proof of concept, they build the NoDepth Bottleneck, which combines interference-free design elements and shows better scaling than MobileNetv2 and MobileNetv3 on ImageNet.

What carries the argument

The argument is carried by two instruments. The first is the SoLU module, defined as $y = \ln(x \cdot \mathrm{softmax}(x))$ applied to the residual stream just before the skip connection; because it is superlinear and superadditive, it suppresses co-activation of polysemantic directions, so inserting it is a probe that reveals whether an architecture depends on interference. The second is the feature-alignment score $O_{ij} = \sum_k \hat X_{ik}\hat X_{kj}^T$ over L2-normalized channel feature maps, which quantifies how aligned feature directions are, with values near 0 meaning orthogonal and values near 1 meaning fully superimposed. The paper uses these probes inside a standardized vessel model, inspired by ConvNeXt, that isolates the bottleneck block so different architectures can be compared fairly. The NoDepth Bottleneck combines the design elements these probes identify: depthwise convolution in the input channel dimension, Layer Norm instead of Batch Norm, and a bypass that preserves the first expansion's dimensions.

What would settle it

An experiment that would settle this: train an inverted-bottleneck model with SoLU inserted but with optimization controls, such as a lower learning rate, gradient clipping, or a normalization change that does not alter feature packing, and check whether the roughly 50% accuracy drop persists. If it disappears, the drop is an optimization artifact rather than evidence of interference dependence.

Watch

Extended reading notes

Core claim

The central discovery is a dichotomy among bottleneck architectures at very low parameter counts. Inverted bottlenecks and sandglass blocks with Batch Norm encode features by packing multiple directions into the same high-dimensional space; this interference is effective at larger scales but breaks down under roughly 1.5M parameters, producing poor scaling. Architectures such as the ConvNeXt-like bottleneck, MobileNeXt with Layer Norm, and the proposed NoDepth Bottleneck are interference-free: they keep depthwise convolution in the input channel dimension, use Layer Norm instead of Batch Norm, and preserve a bypass of the first expansion, so suppressing interference with SoLU barely changes accuracy. Feature-alignment statistics confirm the split: interference-dependent models produce feature maps that SoLU pushes into suboptimal spaces, while interference-free models maintain their representational structure. The paper validates the dichotomy on CIFAR-10, CIFAR-100, and ImageNet, and reports that NoDepth scales better than MobileNetv2 and MobileNetv3 in the low-parameter range, though it is explicitly a proof of concept and not a new state of the art.

Load-bearing premise

The load-bearing premise is that the large accuracy drops seen when SoLU is inserted are caused specifically by removing feature interference, and not by side effects of replacing the activation and normalization behavior, such as optimization instability from a superlinear function combined with Batch Norm.

Editorial extensions

If this is right

Below roughly 1.5M parameters, an inverted-bottleneck block with Batch Norm and an expansion ratio of at least 2 should be expected to underperform simpler depthwise-separable blocks at the same parameter budget.
Inserting SoLU into the residual stream is a cheap diagnostic: a large accuracy drop flags an interference-dependent architecture, while a small drop flags one that is likely to scale well.
Architecture choices, such as keeping depthwise convolution at the input dimension and using Layer Norm after expansion, matter more than the choice of superlinear activation for low-parameter scaling.
The NoDepth Bottleneck provides a template: interference-free design can beat MobileNetv2 and MobileNetv3 scaling on ImageNet without claiming state-of-the-art accuracy.
Models like MobileNetv1, which use depthwise separable convolutions without an expansion bottleneck, inherently avoid the interference dependence that hurts inverted-bottleneck scaling at very low parameter counts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If interference dependence is the mechanism, then a practical screening test for architecture search would be to run each candidate block with and without SoLU at the target parameter count and discard candidates that lose more than a few points.
The same lens could be applied to other low-parameter regimes, such as quantization or pruning, where packing features into fewer effective dimensions may create interference that hurts scaling.
The paper explicitly notes that NoDepth is computationally expensive and not a new state of the art; an efficient variant that retains the interference-free properties would be a natural next step.
An alternative explanation the paper does not rule out is that SoLU's superlinearity interacts with Batch Norm's statistics during training; a test that controls optimization dynamics would determine whether 'interference' is a representational or a training-dynamics phenomenon.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The thing to know: the paper has a real empirical observation hiding behind a causal story the experiments don't fully support. Swapping SoLU into the residual stream wrecks MobileNetv2/v3 and BatchNorm MobileNeXt but leaves ConvNeXt-like and LayerNorm variants nearly untouched, and the authors connect this to the known low-parameter scaling gap where MobileNetv1 beats later MobileNets. That is worth looking at.

What's genuinely new: bringing SoLU, a Transformer interpretability tool, into convolutional bottlenecks as an intervention, plus a unified vessel comparison across Inverted, Sandglass, and ConvNeXt bottlenecks. The NoDepth block is a reasonable proof of concept, and the authors are appropriately modest about it — they call it a proof of concept and say it doesn't set a new SOTA. The feature-alignment analysis (Oij) is a nice visual complement, though descriptive rather than causal.

Where it gets soft. The central attribution — that SoLU's accuracy collapse comes specifically from removing feature interference — is not identified. SoLU as used here is a compound intervention: it inserts a superlinear softmax-gated function and simultaneously changes normalization at the residual stream (the ln in Eq. 2.2 is LayerNorm). Substituting LayerNorm into Inverted Bottlenecks gives a 17% drop, but there is no LN-only control without SoLU, and no control with a different superlinear activation like SiLU. So the ~50% drop could be optimization instability or norm mismatch, not interference removal. The stress-test note has this right, and it's load-bearing: the architecture taxonomy, the Oij story, and the NoDepth rationale all lean on that step.

Empirical hygiene is also thin: no error bars or seed counts, no code or data, the CIFAR-100 result is asserted but never shown, and the ImageNet validation is a single figure with no training protocol. The statement that 'varying learning rates or optimizers do not alter our core findings' is made without supporting runs.

Who is this for? Someone working on ultra-low-parameter vision architectures or mechanistic interpretability of CNNs could get useful hypotheses and a set of interesting ablation directions. It is not a settled mechanistic explanation. With code, seeds, and the missing controls it could become a solid contribution. I would send it to peer review because the question is genuinely interesting and the authors are not overclaiming, but I would ask for major revision, not acceptance as is.

Referee Report

4 major / 4 minor

Summary. The paper studies why low-parameter vision models built on Inverted and Sandglass bottlenecks scale worse than architectures such as ConvNeXt-like blocks. It proposes that the cause is reliance on feature-map interference (superposition), and it uses the SoLU activation, inserted in the residual stream as in Eq. (2.2), as an intervention to remove interference. The authors report accuracy drops of roughly 50% for MobileNetv2/v3-style bottlenecks and 42% for MobileNeXt with Batch Norm, versus much smaller drops for ConvNeXt-like, LayerNorm-based, and their proposed NoDepth Bottleneck. They also analyze feature-map alignment with an interference index Oij and present ImageNet results for NoDepth as a proof of concept. The central claim is that reducing interference improves scaling in the sub-1.5M-parameter regime.

Significance. If the interference attribution were established, the paper would offer a useful mechanistic design heuristic for low-parameter vision architectures and a plausible explanation for the well-documented scaling gap of MobileNetv2/v3 at very low parameter counts. The paper has two genuine strengths: it applies a mechanistic-interpretability intervention (SoLU) to convolutional bottlenecks in a unified experimental 'vessel' that isolates the bottleneck block, and it proposes a concrete proof-of-concept architecture. However, the central causal step is confounded, the quantitative evidence lacks reproducibility details, and the NoDepth validation is a built demonstration rather than an independent test. The paper does not provide code, exact accuracy tables, or training protocols, which limits verification. With additional controls and fuller reporting, the contribution could become a solid empirical study; currently the mechanistic interpretation is underdetermined.

major comments (4)

[Section 4.1, Eqs. (2.1)-(2.2)] The SoLU intervention changes two things simultaneously: it inserts the superlinear function x * softmax(x) and, because the paper defines ln as Layer Norm, it replaces the normalization behavior at the residual-stream insertion point. The large accuracy drop of the Inverted Bottleneck is therefore not uniquely attributable to removal of feature-map interference; it could equally reflect optimization instability from a superlinear activation applied to BatchNorm-trained features, or a mismatch between BatchNorm statistics and LayerNorm at that location. The reported substitute experiment that replaces Batch Norm with Layer Norm (17% drop) does not isolate SoLU, because no LayerNorm-only (without SoLU) baseline is reported and no alternative superlinear activation (e.g., SiLU) is tested. Since the architecture taxonomy, the Oij feature-alignment narrative, and the NoDepth design rationale all depend on this causal step, the interference attribution is underdetermined. Please add a matched LayerNorm-only control and a SiLU-with-LayerNorm control, or otherwise show that the accuracy drop is specific to interference removal.
[Section 4.1, Figs. 5-6] The central quantitative evidence consists of accuracy drops of roughly 50%, 42%, 17%, 6%, and 4%, but no error bars, number of seeds, or exact accuracy tables are reported. The statement in Section 3.0.1 that CIFAR-100 'consistently demonstrated the same behavior' is not supported by any CIFAR-100 result in the manuscript. The ImageNet comparison in Figure 11 omits the training protocol (epochs, batch size, resolution, optimizer settings, and parameter counts of the compared variants). Without this information, the scaling claims in Sections 4.3 and 5 cannot be verified.
[Section 4.2, Eq. (4.1)] The interference index Oij is not well defined as written. If Xhat is a normalized feature map, the expression sum_k Xhat_ik * Xhat^T_kj is ambiguous about which axes are contracted and how a scalar between 0 and 1 is obtained from a matrix product. Figures 7 and 8 describe qualitatively that feature directions become 'more aligned' or 'dissimilar,' but no quantitative definition of the reported mean angles or of the layer-wise aggregation is provided. This weakens the feature-alignment evidence that supports the proposed taxonomy.
[Section 4.3] The NoDepth Bottleneck is constructed directly from the paper's own conclusions (depthwise convolution at the input dimension, Layer Norm, and a bypass that retains the first-expansion dimensions), so the ImageNet comparison in Figure 11 is a built demonstration rather than an independent test of the interference hypothesis. The paper does acknowledge that this is a proof of concept, but the wording 'prove its theoretical scaling superiority' and 'empirically validates our results' overstates what a single hand-designed architecture can establish. An external prediction, such as an a priori rule that identifies which existing architectures will be SoLU-sensitive, or a test on an architecture not designed from these insights, would strengthen the claim.

minor comments (4)

[Eq. (3.2)] The parentheses in y = F(x, {Wi}) + x = Wout(sigma(Win(ln(Wdw(x)))) + x are unbalanced; the intended formula is likely Wout(sigma(Win(ln(Wdw(x))))) + x.
[Eqs. (2.1)-(2.2)] The notation 'ln' is nonstandard: the text says 'ln denotes Layer Norm,' but the symbol normally denotes the natural logarithm. Using 'LN(.)' would avoid confusion.
[Table 1] The table caption does not identify the width multiplier or variant used for each model row (for example, MobileNetv1 0.5 versus 1.3), and the source of the top-1 accuracies is not given.
[Figure 11] The axis labels and legend entries are not legible in the provided figure; please add a table with exact accuracies and parameter counts for each curve.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SoLU is an external intervention, the architecture taxonomy is empirical, and NoDepth is validated on ImageNet as an external benchmark.

full rationale

The paper's central chain is: (1) insert the external SoLU module (Eq. 2.2) into bottleneck residual streams; (2) observe accuracy drops that differ by architecture (Figs. 5-6); (3) measure feature alignment with Oij (Eq. 4.1) to support the interpretation that the drops track interference; (4) design NoDepth from the resulting design intuitions and test it on ImageNet (Fig. 11). None of these steps reduces to its own input by construction. SoLU is taken from Elhage et al. [5] and is not fitted here; the accuracy drops are empirical outputs, not definitions. The interference measure Oij is defined independently of the accuracy comparison, so the paper does not equate 'interference' with 'the thing SoLU removes' at the definitional level; it provides feature-map evidence (Figs. 7-8, 10) connecting the intervention to the construct. The NoDepth section is explicitly a proof of concept: the model is built from insights 'uncovered throughout this paper' and then evaluated on ImageNet, a dataset not used to select the design; this is a prospective external check, not a fitted parameter renamed as a prediction. The only self-citations ([12], [13], [18]) occur in background material on agriculture and fog computing and are not load-bearing for the interference claim. The main weakness of the paper—that the SoLU intervention also changes normalization and activation behavior, so the causal attribution to interference may be confounded—is a correctness risk, not a circularity, because the paper's conclusions do not follow by definition or by self-citation. Therefore no circular step meets the evidence bar.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The central claims rest on four unproven premises: SoLU is a valid interference probe, Oij measures interference, SoLU sensitivity is causally due to interference, and the vessel design isolates bottlenecks. The NoDepth design adds hand-chosen hyperparameters. These are not fatal, but they bound the strength of the conclusion.

free parameters (2)

NoDepth architecture hyperparameters = not reported
Expansion ratio, channel widths, depth, Layer Norm placement, and permutation/concatenation structure are chosen by the authors from CIFAR-10 insights; no ablation or search is reported.
SoLU integration position in residual stream = before residual connection
The placement of SoLU is a design choice that is not derived; all conclusions depend on this specific integration point.

assumptions (4)

domain assumption SoLU superlinear activation reduces feature interference and is a valid probe for superposition reliance.
Inherited from Elhage et al. [5] and applied to convolutional bottlenecks; the paper's causal interpretation relies on it.
domain assumption Cosine similarity Oij of normalized feature maps measures interference, with 0 meaning orthogonal and 1 meaning fully aligned.
Defined in Eq 4.1; no validation is given that this metric tracks superposition in trained CNNs.
ad hoc to paper Accuracy drop under SoLU is caused by removal of interference rather than by optimization or normalization side effects.
Central to Section 4.1; alternative explanations are not controlled.
domain assumption The unified vessel architecture isolates bottleneck behavior.
Assumes that stem and downsampling modules do not confound comparisons between bottlenecks.

invented entities (2)

NoDepth Bottleneck independent evidence
purpose: Proof-of-concept low-parameter module combining depthwise convolution in input space, Layer Norm, and a dimension-preserving bypass.
The architecture yields ImageNet scaling behavior that can be independently tested, though exact numbers are only in Figure 11.
Interference index Oij
purpose: Quantifies feature alignment along the channel axis to diagnose superposition and interference.
A new measurement construct introduced by Eq 4.1; no external validation that it corresponds to superposition.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models." pith.science (2026). https://pith.science/paper/OBM7A5YQ

@misc{pith2026250715798,
  author       = {Pith},
  title        = {Pith review of: Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBM7A5YQ}},
  note         = {Machine review of arXiv:2507.15798}
}

read the original abstract

The paper investigates the performance of state-of-the-art low-parameter deep neural networks for computer vision, focusing on bottleneck architectures and their behavior using superlinear activation functions. We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics. Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters). We identify key design elements that reduce interference by examining various bottleneck architectures, leading to a more efficient neural network. Consequently, we propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset. These findings contribute to more efficient and scalable neural networks for the low-parameter range and advance the understanding of bottlenecks in computer vision. https://caiac.pubpub.org/pub/3dh6rsel

Figures

Figures reproduced from arXiv: 2507.15798 by the authors.

**Figure 1.** Inverted Bottleneck scaling issue When examining ultra-low-parameter neural networks (ranging from 0.5M to 1.5M parameters), an important pattern is often overlooked: scaling. It is commonly assumed that MobileNetv2 and v3 progressively improve the parameter-to-accuracy ratio. While this holds true at larger scales, MobileNetv1 outperforms both at lower parameter counts. For instance, with 1.5M parameters, MobileNe… view at source ↗

**Figure 2.** 2D visualization of feature alignment and interference: Oij ranges from 0 for orthogonal features and 1 for high superposition, leading to unintended activations. *Interferences Superposition and Interferences. In neural networks, superposition occurs when a single neuron or group of neurons simultaneously represent multiple distinct features, a phenomenon known as polysemanticity—where one neuron encodes multiple t… view at source ↗

**Figure 3.** Our proposition of the SoLU module integration into residual space We propose integrating the SoLU function2 , as described in equation 2.1, into various low-parameter state-of-the-art architectures ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: The base vessel containing each specific architecture, inspired by ConvNeXt [31]. We use a depth repetition for each identity bloc of [3,3,9,3]. Directly comparing different bottlenecks using their base models would be ineffective, as each architecture has unique chara…

**Figure 5.** Figure 5: Accuracy difference between the base architecture of the Inverted Bottlenecks (v2 and v3) and their SoLU implementation. However, when analyzing ConvNeXt and MobileNeXt (with Layer Norm) in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Accuracy difference between the base architecture of the ConvNeXt-like Bottleneck and Sandglass Bottleneck and their SoLU implementation. To explore whether the base architecture inherently mitigates interference, we simulate several configurations: a standard bottlen…

**Figure 7.** Figure 7: Mean features angles between architecture dependent on small interference (MobileNetv3) and interference-free architecture (ConvNeXt). MobileNetv3 exhibits a 50% accuracy drop in its baseline form with Batch Norm. The model’s feature maps consist mostly of orthogonal v…

**Figure 8.** Figure 8: MobileNeXt features direction specificities. utilizes high-dimensional space by aligning features in similar directions—over 50% of layers exhibit this behavior. However, SoLU disrupts the architecture’s reliance on such interference-based representations. Initially, …

**Figure 9.** Figure 9: Our proposed architecture, inspired by Inverted Bottlenecks [25, 27] and GhostNets [37, 38] [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

**Figure 11.** Figure 11: NoDepth Bottleneck with sota bottleneck architectures 5. Conclusion Our study examined the mechanistic behavior of bottleneck architectures and the impact of superlinear activation functions, specifically the SoLU module, focusing on low-parameter models for computer …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 28 canonical work pages

[1]

Cloud computing and emerg- ing IT platforms: Vision, hype, and reality for delivering computing as the 5th utility

R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic. “Cloud computing and emerg- ing IT platforms: Vision, hype, and reality for delivering computing as the 5th utility”. In: Future Generation Computer Systems25.6 (June 2009), pp. 599–616.issn: 0167-739X. doi: 10.1016/j.future.2008.12.001

work page doi:10.1016/j.future.2008.12.001 2009
[2]

Toy models of superposition

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. “Toy models of superposition”. In: arXiv preprint arXiv:2209.10652 (2022)

arXiv 2022
[3]

Learning Multiple Layers of Features from Tiny Images

A. Krizhevsky. “Learning Multiple Layers of Features from Tiny Images”. In: 2009

work page 2009
[4]

Imagenet: A large-scale hierar- chical image database

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. “Imagenet: A large-scale hierar- chical image database”. In:2009 IEEE conference on computer vision and pattern recognition. Ieee. 2009, pp. 248–255.doi: 10.1109/CVPR.2009.5206848

arXiv 2009
[5]

Softmax Linear Units

N. Elhage et al. “Softmax Linear Units”. In:Transformer Circuits Thread(2022). url: https: //transformer-circuits.pub/2022/solu/index.html

work page 2022
[6]

Fake news detection on social media using geometric deep learning

F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein. “Fake news detection on social media using geometric deep learning”. In:arXiv preprint arXiv:1902.06673(2019)

arXiv 2019
[7]

Deep learning for social media analysis in crises situations

M. B. Lazreg, M. Goodwin, and O.-C. Granmo. “Deep learning for social media analysis in crises situations”. In:The 29th Annual Workshop of the Swedish Artificial Intelligence Society (SAIS) 2–3 June 2016, Malmö, Sweden. 2016, p. 31

work page 2016
[8]

A deep learning approach to drone moni- toring

Y. Chen, P. Aggarwal, J. Choi, and C.-C. J. Kuo. “A deep learning approach to drone moni- toring”. In:2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE. 2017, pp. 686–691

work page 2017

Show all 39 references

[9]

Real-time drone detection using deep learn- ing approach

M. Wu, W. Xie, X. Shi, P. Shao, and Z. Shi. “Real-time drone detection using deep learn- ing approach”. In: Machine Learning and Intelligent Communications: Third International Conference, MLICOM 2018, Hangzhou, China, July 6-8, 2018, Proceedings 3. Springer. 2018, pp. 22–32

2018
[10]

Overview of deep learning in medical imaging

K. Suzuki. “Overview of deep learning in medical imaging”. In: Radiological physics and technology 10.3 (2017), pp. 257–273

2017
[11]

An overview of deep learning in medical imaging

A. Anaya-Isaza, L. Mera-Jiménez, and M. Zequera-Diaz. “An overview of deep learning in medical imaging”. In:Informatics in medicine unlocked26 (2021), p. 100723

2021
[12]

AccurateYieldPredictionusingDeepLearning: challenges and recent developments on smart-viticulture

L.Mohimont,L.Hollard,andL.A.Steffenel.“AccurateYieldPredictionusingDeepLearning: challenges and recent developments on smart-viticulture”. In:rd International Workshop on Information Systems Engineering for Smarter Life (ISESL). 2023

2023
[13]

Applying Knowledge Distillation on Pre-Trained Model for Early Grapevine Detection

L. Hollard and L. Mohimont. “Applying Knowledge Distillation on Pre-Trained Model for Early Grapevine Detection”. In:Workshop Proceedings of the 19th International Conference on Intelligent Environments (IE2023). IOS Press. 2023, pp. 149–156

2023
[14]

Generative adversarial networks: introduction and outlook

K. Wang, C. Gou, Y. Duan, Y. Lin, X. Zheng, and F.-Y. Wang. “Generative adversarial networks: introduction and outlook”. In:IEEE/CAA Journal of Automatica Sinica4.4 (2017), pp. 588–598

2017
[15]

Generative adversarial networks

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. “Generative adversarial networks”. In:Communications of the ACM 63.11 (2020), pp. 139–144

2020
[16]

Convolutional networks for images, speech, and time series

Y. LeCun, Y. Bengio, et al. “Convolutional networks for images, speech, and time series”. In: The handbook of brain theory and neural networks3361.10 (1995), p. 1995

1995
[17]

Attention is all you need

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention is all you need”. In:Advances in neural information processing systems 30 (2017)

2017
[18]

Improvingtheperformanceoffogcomputingthroughtheuseofdatalocality

L.A.Steffenel.“Improvingtheperformanceoffogcomputingthroughtheuseofdatalocality”. In: 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE. 2018, pp. 217–224.doi: 10.1109/CAHPC.2018.8645879

2018
[19]

Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing

Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang. “Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing”. In:Proceedings of the IEEE107.8 (Aug. 2019), pp. 1738–1762.doi: 10.1109/JPROC.2019.2918951. 12

2019
[20]

Deep Learning for Edge Computing Applications: A State-of-the-Art Survey

F. Wang, M. Zhang, X. Wang, X. Ma, and J. Liu. “Deep Learning for Edge Computing Applications: A State-of-the-Art Survey”. In:IEEE Access8 (2020). Conference Name: IEEE Access, pp. 58322–58336.doi: 10.1109/ACCESS.2020.2982411

2020
[21]

Edge AI: a survey

R. Singh and S. S. Gill. “Edge AI: a survey”. In: Internet of Things and Cyber-Physical Systems 3 (2023), pp. 71–92

2023
[22]

Edge AI: A taxonomy, systematic review and future directions

S. S. Gill, M. Golec, J. Hu, M. Xu, J. Du, H. Wu, G. K. Walia, S. S. Murugesan, B. Ali, M. Kumar, et al. “Edge AI: A taxonomy, systematic review and future directions”. In:Cluster Computing 28.1 (2025), pp. 1–53

2025
[23]

MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

S. Mehta and M. Rastegari. “MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer”. In:International Conference on Learning Representations. 2022. url: https://openreview.net/forum?id=vh-0sUt8HlG

2022
[24]

Mobilevitv3:Mobile-friendlyvisiontransformerwithsimple and effective fusion of local, global and input features

S.N.WadekarandA.Chaurasia.“Mobilevitv3:Mobile-friendlyvisiontransformerwithsimple and effective fusion of local, global and input features”. In:arXiv preprint arXiv:2209.15159 (Oct. 2022). doi: 10.48550/arXiv.2209.15159

2022 doi arXiv
[25]

Mobilenets: Efficient convolutional neural networks for mobile vision applications

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. “Mobilenets: Efficient convolutional neural networks for mobile vision applications”. In: arXiv preprint arXiv:1704.04861(2017)

2017 arXiv
[26]

Mobilenetv2: Inverted residuals and linear bottlenecks

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. “Mobilenetv2: Inverted residuals and linear bottlenecks”. In:Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 4510–4520

2018
[27]

Searching for mobilenetv3

A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, et al. “Searching for mobilenetv3”. In:Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 1314–1324

2019
[28]

EfficientNet: Rethinking Model Scaling for Convolutional Neural Net- works

M. Tan and Q. Le. “EfficientNet: Rethinking Model Scaling for Convolutional Neural Net- works”. en. In:Proceedings of the 36th International Conference on Machine Learning. PMLR, May 2019, pp. 6105–6114.url: https://proceedings.mlr.press/v97/tan19a.html (visited on 12/18/2023)

2019
[29]

EfficientNetV2: Smaller Models and Faster Training

M. Tan and Q. Le. “EfficientNetV2: Smaller Models and Faster Training”. en. In:Proceedings of the 38th International Conference on Machine Learning. PMLR, July 2021, pp. 10096– 10106. url: https://proceedings.mlr.press/v139/tan21a.html (visited on 12/18/2023)

2021
[30]

Mehta and M

S. Mehta and M. Rastegari. Separable Self-attention for Mobile Vision Transformers. June
[31]

A convnet for the 2020s

Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. “A convnet for the 2020s”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 11976–11986

2022
[32]

Rethinking bottleneck structure for effi- cient mobile network design

D. Zhou, Q. Hou, Y. Chen, J. Feng, and S. Yan. “Rethinking bottleneck structure for effi- cient mobile network design”. In:Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer. 2020, pp. 680–697

2020
[33]

Efficientvit: Memory efficient vision transformer with cascaded group attention

X. Liu, H. Peng, N. Zheng, Y. Yang, H. Hu, and Y. Yuan. “Efficientvit: Memory efficient vision transformer with cascaded group attention”. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 14420–14430

2023
[34]

Sparse attention with linear units

B. Zhang, I. Titov, and R. Sennrich. “Sparse attention with linear units”. In:arXiv preprint arXiv:2104.07012 (2021)

2021 arXiv
[35]

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

A. M. Saxe, J. L. McClelland, and S. Ganguli. “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks”. In:arXiv preprint arXiv:1312.6120(2013)

2013 arXiv
[36]

Levit: a visiontransformerinconvnet’sclothingforfasterinference

B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze. “Levit: a visiontransformerinconvnet’sclothingforfasterinference”.In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 12259–12269

2021
[37]

Ghostnet: More features from cheap operations

K. Han, Y. Wang, Q. Tian, J. Guo, C. Xu, and C. Xu. “Ghostnet: More features from cheap operations”. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 1580–1589

2020
[38]

Repghost: a hardware-efficient ghost module via re-parameterization

C. Chen, Z. Guo, H. Zeng, P. Xiong, and J. Dong. “Repghost: a hardware-efficient ghost module via re-parameterization”. In:arXiv preprint arXiv:2211.06088(2022)

2022 arXiv
[2022]

doi: 10.48550/arXiv.2206.02680

doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Cloud computing and emerg- ing IT platforms: Vision, hype, and reality for delivering computing as the 5th utility

R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic. “Cloud computing and emerg- ing IT platforms: Vision, hype, and reality for delivering computing as the 5th utility”. In: Future Generation Computer Systems25.6 (June 2009), pp. 599–616.issn: 0167-739X. doi: 10.1016/j.future.2008.12.001

work page doi:10.1016/j.future.2008.12.001 2009

[2] [2]

Toy models of superposition

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. “Toy models of superposition”. In: arXiv preprint arXiv:2209.10652 (2022)

arXiv 2022

[3] [3]

Learning Multiple Layers of Features from Tiny Images

A. Krizhevsky. “Learning Multiple Layers of Features from Tiny Images”. In: 2009

work page 2009

[4] [4]

Imagenet: A large-scale hierar- chical image database

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. “Imagenet: A large-scale hierar- chical image database”. In:2009 IEEE conference on computer vision and pattern recognition. Ieee. 2009, pp. 248–255.doi: 10.1109/CVPR.2009.5206848

arXiv 2009

[5] [5]

Softmax Linear Units

N. Elhage et al. “Softmax Linear Units”. In:Transformer Circuits Thread(2022). url: https: //transformer-circuits.pub/2022/solu/index.html

work page 2022

[6] [6]

Fake news detection on social media using geometric deep learning

F. Monti, F. Frasca, D. Eynard, D. Mannion, and M. M. Bronstein. “Fake news detection on social media using geometric deep learning”. In:arXiv preprint arXiv:1902.06673(2019)

arXiv 2019

[7] [7]

Deep learning for social media analysis in crises situations

M. B. Lazreg, M. Goodwin, and O.-C. Granmo. “Deep learning for social media analysis in crises situations”. In:The 29th Annual Workshop of the Swedish Artificial Intelligence Society (SAIS) 2–3 June 2016, Malmö, Sweden. 2016, p. 31

work page 2016

[8] [8]

A deep learning approach to drone moni- toring

Y. Chen, P. Aggarwal, J. Choi, and C.-C. J. Kuo. “A deep learning approach to drone moni- toring”. In:2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). IEEE. 2017, pp. 686–691

work page 2017