Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification

Michal Byra; Stanislaw Janik

REVIEW 3 major objections 4 minor 47 references

Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mixture-of-experts transformer reads labels hidden in network weights

desk verdict Solid empirical advance in INR weight-space classification with a well-motivated hierarchical MoE, but the large-scale SOTA claims rest on a couple of unreplicated results. read the letter →

arxiv 2607.29463 v1 pith:5IUGSPX6 submitted 2026-07-31 cs.CV

Stanislaw Janik , Michal Byra This is my paper · ORCID

classification cs.CV

keywords implicitneuralrepresentationsweight-spacelearningmixture-of-expertsmeta-learningimageclassificationSIRENexplainabilitynetworkpruning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Implicit neural representations (INRs) turn each image into a small coordinate-based network, so image classification can be done on network weights instead of pixels. This paper argues that the right way to read those weights is with a hierarchical mixture-of-experts transformer: first route whole INR layers to specialists, then route individual neurons to specialists conditioned on the layer choice. Trained jointly with a meta-learned SIREN initializer, the scheme reaches reported accuracies of 99.06% on MNIST, 65.01% on CIFAR-10, 62.96% on Imagenette, and 26.73% on ImageNet-1K, ahead of earlier weight-space classifiers. The same architecture comes with a gradient-based attribution method, Grad-CAG, and a pruning analysis intended to show that class-relevant structure concentrates in INR subcircuits. A sympathetic reader would take the paper's core contribution to be the demonstration that conditional, layer-aware computation rather than a larger dense transformer is what makes INR weight-space classification scale.

What carries the argument

The load-bearing object is the HMoE block: a two-stage sparse mixture of experts that replaces the ordinary feed-forward network in each Transformer layer. Stage one pools the tokens belonging to each INR layer, produces a layer-level routing vector via a softmax over expert scores, and processes the whole layer's tokens with the selected top-k experts; stage two routes each individual token using an attention-like score that combines the token's own projection with a learned projection of the stage-one layer routing vector. Around this sits an input transform the paper fixes at $\$\lambda$(\phi-\$\theta$)$ with $\$\lambda$=500$, and a meta-learning loop (an inner reconstruction fit, an outer objective mixing reconstruction, classification, and expert-balancing losses) that updates the shared initializer $\$\theta$$ and the per-step learning-rate schedule. Grad-CAG, computed as $|\phi_j \, \partial c_i / \partial \phi_j|$ per weight, provides the attribution scores used for pruning and spatial explanation.

What would settle it

Train the full pipeline but set the classification weight in the meta-learning objective to zero, so the INR weight space is not shaped by the classifier; if HMoE accuracy stays at the reported level, the claim that meta-shaping the weight distribution is load-bearing would be falsified. A related check is to sweep the scaling factor $\$\lambda$$ over orders of magnitude and see whether accuracy collapses away from 500.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that the learned weight deltas $\$\lambda$(\phi-\$\theta$)$ produced by a short inner-loop fit of a shared SIREN contain enough class-discriminative structure for a Transformer classifier, and that a hierarchical Mixture-of-Experts block extracts that structure better than a standard block. The block splits the weight token stream by INR layer, sends each whole layer to a top-1 expert, then lets individual tokens choose experts using the layer-level routing as a conditioning signal. Because the cross-entropy loss backpropagates through the classifier into the meta-initialization and per-step learning rates, the INR weight space itself is shaped so that class differences become readable. Evidence in its support includes gains over both dense and parameter-matched baselines, an ablation in which removing the layer-wise stage lowers accuracy, and routing statistics showing stable layer-expert specialization without collapse.

Load-bearing premise

The load-bearing premise is that the per-image change in fitted weights from the shared initialization, after one fixed scaling of 500, reliably separates classes and does not discard the information the classifier needs.

Editorial extensions

If this is right

With 10 blocks and sparse top-1 routing, HMoE-MWT reports higher accuracy than the earlier 20-block dense model on CIFAR-10, so architectural structure rather than model width is the primary lever.
HMoE-MWT beats the parameter-matched dense baseline: at 1.1M classifier parameters it reports 61.47% on Imagenette versus 56.78%, and at 11M parameters it reports 62.52% versus 59.92%, indicating the gain is not just from extra capacity.
Replacing the layer-wise stage with a token-wise MoE drops accuracy, so grouping tokens by INR layer is a functional part of the method.
Grad-CAG attribution maps score 0.35 on a 200-mask pointing-game test, versus 0.22 for magnitude pruning and 0.15 for random, indicating class-relevant weights have a spatial signature that can be localized.
The framework extends to high resolution with 26.73% top-1 on ImageNet-1K, suggesting weight-space learning can operate beyond toy-scale domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the layer-wise experts specialize by INR layer, a dense transformer with layer-specific positional embeddings may recover part of the HMoE gain, which would isolate the contribution of routing from the contribution of layer information.
The fixed $\lambda=500$ scaling of $\phi-\theta$ is an untested hyperparameter; a calibration sweep across a range of meta-initialization scales could improve robustness at ImageNet-1K resolution.
Grad-CAG pruning that preserves coarse object identity could be repurposed as a no-pixel-access audit tool for detecting dataset biases, since the tench-fishing-net error appears as a spatial attribution.
Applying the same HMoE tokenization to other INR families such as audio wavefields or 3D NeRFs is stated as future work, but the layer-frequency correspondence motivating the design suggests the gain should transfer if those INRs also show layer specialization.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you follow weight-space learning. The core idea is a two-stage hierarchical Mixture-of-Experts transformer that routes INR weight tokens first by layer, then by token, trained end-to-end with a meta-learned SIREN. It works: on MNIST, Fashion-MNIST, CIFAR-10 and Imagenette it beats the previous MWT baselines, and the parameter-matched comparison on Imagenette (61.47 vs 56.78 at 1.1M; 62.52 vs 59.92 at 11M) shows the architecture itself buys the gains, not just extra parameters. The ablation on CIFAR-10 also supports the layer-wise routing component. The paper also introduces Grad-CAG, a weight-space attribution method, and shows it can localize objects better than magnitude or random pruning. That is honest, useful work.

Soft spots are mostly about the headline claims at scale. The ImageNet-1K number (26.73%) is a single run with no error bar, at 1% coordinate subsampling with a PSNR of only 22.06 dB; the 2.6-point gap over MWT-L could easily be seed noise. Similarly, the 69.11% CIFAR-10 result with 20 blocks and augmentations appears only in a sentence, with no table and no configuration details, so it cannot be audited. The input scaling lambda=500 is never ablated, which matters because the classifier sees lambda(phi-theta). The code link is given but no commit hash or released data, so reproducibility is not yet verifiable. The ablation table has a garbled row (the default HMoE-MWT row appears to be mangled), and the ablations use an 80/20 split of CIFAR-10, so they are not directly comparable to the main numbers. The pointing-game score of 0.35 is modest, though better than the baselines.

None of these are fatal. The core contribution is the hierarchical MoE, and the evidence for that is solid on the mid-scale benchmarks. The fix is straightforward: add error bars or at least repeated runs for ImageNet, put the 20-block CIFAR-10 result in a table with full config, ablate lambda, and clean up the code release. This deserves serious peer review with those revisions.

Referee Report

3 major / 4 minor

Summary. The paper proposes a hierarchical Mixture-of-Experts (HMoE) Transformer for classifying implicit neural representations (INRs) of images. It extends the Meta Weight Transformer (MWT) framework by replacing the standard feed-forward blocks with a two-stage sparse MoE: a layer-wise stage that routes all tokens originating from a given INR layer to a selected expert, and a token-wise stage that performs fine-grained routing conditioned on the layer-level gating. The classifier operates on λ(ϕ−θ), the scaled difference between the per-image adapted INR parameters and the shared meta-initialization. The authors report state-of-the-art accuracy among weight-space classifiers on MNIST, Fashion-MNIST, CIFAR-10, Imagenette, and ImageNet-1K, including parameter-matched comparisons with MWT on Imagenette. They also introduce Grad-CAG, a gradient-weighted attribution score for INR weights, and use it for structured pruning and for generating spatial attribution maps, with a pointing-game evaluation. An ablation study on CIFAR-10 indicates that the combined hierarchical MoE outperforms either the layer-wise or token-wise stage alone.

Significance. If the empirical claims hold, this is a meaningful advance in weight-space learning: the hierarchical MoE is a natural inductive bias for the layer-structured weight space of INRs, and the paper provides credible evidence of architectural advantage via parameter-matched Imagenette comparisons against a strong dense baseline. The release of code, the reporting of means and standard deviations on multiple benchmarks, and the detailed ablations are strengths. The attribution and pruning analyses, including the pointing-game evaluation, are a useful first step toward interpretability in weight-space classification. However, the broad 'state-of-the-art across low- to high-resolution datasets' claim is stronger than the evidence: the ImageNet-1K result is a single unreplicated run, and the 20-block CIFAR-10 SOTA number is not tabulated or fully specified. The fixed input scaling λ=500 and its use in baselines also needs clarification for a fair comparison.

major comments (3)

[§4.1, Table 4] The ImageNet-1K result is reported as a single run (Table 4: HMoE-MWT-L 26.73% vs MWT-L 24.11%) with no standard deviation, at 0.01 coordinate subsampling and a reconstruction PSNR of 22.06 dB. Given the heavy subsampling and the 2.6-point gap, this number could be within run-to-run variance; the claim of state-of-the-art performance on high-resolution ImageNet-1K should be supported by multiple seeds or explicitly labeled as a preliminary single run.
[§4.1, paragraph following Table 1] The claimed 69.11% CIFAR-10 accuracy for the 20-block HMoE-MWT with augmentations is not tabulated, and its configuration (exact augmentations, loss weights, training schedule, number of runs) is not specified. Since this number is used to claim a new SOTA and to argue that architectural design beats model scaling, it must be reported with full settings and variance in a table, or removed from the headline claims.
[§3.2 and Supplementary 'Additional training details'] The classifier input is λ(ϕ−θ) with λ fixed at 500, but no ablation or sensitivity analysis is provided, and the paper does not state whether the MWT baselines also use this scaling. If MWT does not, the architecture comparisons in Tables 1–3 may partly reflect a difference in input normalization rather than the MoE design; please ablate λ and run a matched baseline with identical scaling.

minor comments (4)

[§4.3, Table 5] The default-configuration row appears to have concatenated numbers ('1 4 1 465.0131.71 11M'), making the token-expert count and the accuracy/PSNR values ambiguous; please reformat so each column is distinct.
[§4.3] The text says the first three rows of Table 5 correspond to replacing the layer-wise MoE with a standard token-wise MoE, but the first row has the layer-wise MoE active and token-wise disabled; the row grouping and the wording should be reconciled.
[§4.3 and Supplementary Table 1] The main-text ablation (Table 5) does not state that some ablations are trained on 80% of CIFAR-10; the supplementary caption says this. Please state the data split in the main text, since the default accuracy in Table 5 (65.01%) differs from the 80%-split default (63.14%) in the supplementary.
[§4.1, Table 1] Table 1 compares HMoE-WT (11M classifier) with WT (1.1M classifier) without parameter matching; the claim that HMoE-WT 'substantially outperforms' WT should point to the parameter-matched results in Table 3 to avoid overstating the comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's core evidence is external benchmark evaluation and standard end-to-end training, with no derivation that reduces to its own inputs.

full rationale

This paper makes no first-principles derivation claim; its central evidence is external benchmark accuracy (MNIST, Fashion-MNIST, CIFAR-10, Imagenette, ImageNet-1K) compared against published baselines, including parameter-matched comparisons. The meta-learning scheme that injects the classification loss into the INR initialization is inherited transparently from prior work [14] as a cited external framework, not a self-citation, and it is standard end-to-end supervised training; the reported test accuracies are measured on held-out splits and are not equal to any training objective by construction. The lambda(phi - theta) input representation and the lambda=500 scaling are explicit design choices, not fitted quantities later relabeled as predictions. The Grad-CAG attribution is a direct gradient-times-input saliency score applied to weight space, and the pruning and localization analyses are empirical evaluations on the trained model; neither step renames a known result as a new derivation nor imports a uniqueness theorem. Concerns about unreplicated ImageNet runs and the untabulated 20-block CIFAR-10 figure are reproducibility and reporting risks, not circularity. Therefore no load-bearing circular step is identifiable from the paper's equations, training protocol, or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. The axioms listed reflect the domain assumptions that the INR representation, the meta-learning delta encoding, the layer-wise grouping, the permutation handling, and the gradient-based attribution are all valid premises. Several hyperparameters are hand-set or ablation-tuned, most notably lambda=500 which scales the classifier input.

free parameters (6)

lambda (input scale) = 500
Scaling factor for the classifier input lambda(phi-theta); set by hand, no ablation in main text.
w_cls (classification loss weight) = 0.01
Weight of cross-entropy loss in outer loop; chosen via ablation on CIFAR-10.
w_balance (balancing loss weight) = 0.1
Weight of the MoE load-balancing loss; chosen via ablation.
number of experts and TopK = E_L=4, E_T=4, k=1
Sparse routing configuration; ablated in Table 5.
inner-loop steps = 6 low-res, 4 high-res
Number of inner-loop gradient updates; hyperparameter choice.
coordinate subsampling rate = 1.0 low-res, 0.1 Imagenette, 0.01 ImageNet
Fraction of pixel coordinates used to fit each INR; affects accuracy and cost.

assumptions (5)

domain assumption Gradient-based meta-learning (MAML/MetaSGD) converges to a shared initialization that enables fast adaptation to individual images.
The outer-loop update in Algorithm 1 assumes that a single theta and per-step learning rates alpha can serve all images in the dataset, and that inner-loop SGD with L_rec yields a good per-image representation.
ad hoc to paper The scaled difference lambda(phi - theta) preserves class-discriminative information while removing image-agnostic components.
The choice to feed phi-theta instead of phi to the classifier is a modeling decision; it is the key input representation and its sufficiency is not theoretically justified.
domain assumption INR layers encode coherent functional and frequency structure such that routing all tokens in a layer through the same experts is beneficial.
Motivates the layer-wise MoE; supported empirically by the ablation, but assumed in the architecture design.
domain assumption The learned positional encoding plus Transformer attention can cope with the permutation symmetry of INR neurons, i.e., the meta-learning produces a consistent neuron ordering or the model learns invariance.
No explicit permutation equivariance is enforced; the paper relies on the common initialization producing consistent ordering across images.
domain assumption Grad-CAG importance scores (first-order gradient times weight) identify functionally important weights for classification.
The attribution method is a heuristic extension of Grad-CAM; its validity is evaluated via pruning and pointing game, but not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification." pith.science (2026). https://pith.science/paper/5IUGSPX6

@misc{pith2026260729463,
  author       = {Pith},
  title        = {Pith review of: Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IUGSPX6}},
  note         = {Machine review of arXiv:2607.29463}
}

read the original abstract

Implicit Neural Representations (INRs) encode signals as the weights of a coordinate-based neural network and have recently been proposed as an alternative domain for downstream learning. While promising, classification directly in weight space remains challenging due to the high dimensionality and complex structure of INR parameters. Furthermore, the way discriminative information is distributed across INR weights remains poorly understood. We propose a hierarchical Mixture-of-Experts (HMoE) Transformer that processes INR weights using conditional computation aligned with the structure of the underlying implicit network. Coupled with a meta-learning framework that shapes INR parameters for downstream tasks, our model achieves state-of-the-art accuracy across standard benchmarks, ranging from low-resolution datasets to high-resolution ImageNet-1K. To gain insight into how INRs encode discriminative information, we develop weight-space attribution and pruning methods that identify parameters most relevant for classification. These analyses reveal how class-specific structure emerges within INR layers and support the suitability of MoE architectures for weight-space learning. Our approach advances both the performance and interpretability of weight-space classifiers.

Figures

Figures reproduced from arXiv: 2607.29463 by the authors.

**Figure 1.** Overview of the proposed framework. End-to-end training jointly optimizes the meta-INR and the hierarchical Mixture-of-Experts (HMoE) classifier through reconstruction and classification losses (left). After training, our attribution pipeline computes weight-space importance scores, identifies relevant computational subcircuits, and explains classification decisions via graph-based attribution (right). 2.3 Mixture… view at source ↗

**Figure 2.** The proposed hierarchical Mixture-of-Experts (HMoE) Transformer block. The INR weights are tokenized according to their originating INR layer and processed by two MoE stages: a Layer MoE, which routes tokens based on layer-level structure, and a Token MoE, which performs fine-grained expert routing across all tokens [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** UMAP visualization of attribution-graph embeddings for HMoE-MWT (left) and HMoE-WT (right). Each point corresponds to an Imagenette sample and is colored by its ground-truth class. The meta-learned model produces smooth, continuous manifolds that remain consistent across training and validation splits, suggesting that meta-learning encourages distributed, reusable subcircuits shared across classes [PITH_FULL_IMAGE… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Functional pruning analysis of INR weight importance. PSNR curves show reconstruction degradation under different pruning strategies. Removing high-attribution or large-magnitude weights quickly destroys the reconstruction, while pruning lowattribution weights preser…

**Figure 5.** Figure 5: Attribution maps obtained via weight pruning. The first three examples show accurate and semantically meaningful attributions. The remaining three examples illustrate a typical failure mode of the dataset: the tench class is spuriously associated with the presence of …

**Figure 1.** Figure 1: Expert routing analysis over the full Imagenette validation dataset. Layerwise MoE (left) shows clear layer-expert specialization with dominant experts per INR layer validating that different INR layers encode functionally distinct information. Token-wise MoE (right)…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 21 canonical work pages

[1]

In: The Eleventh International Conference on Learning Rep- resentations (2023)

Ainsworth, S., Hayase, J., Srinivasa, S.: Git re-basin: Merging models modulo per- mutation symmetries. In: The Eleventh International Conference on Learning Rep- resentations (2023)

work page 2023
[2]

PloS one10(7), e0130140 (2015)

Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one10(7), e0130140 (2015)

2015
[3]

arXiv preprint arXiv:2302.03130 (2023)

Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J.R., Kim, H.: Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130 (2023)

arXiv 2023
[4]

IEEE Transactions on Knowledge and Data Engineering (2025)

Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., Huang, J.: A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering (2025)

2025
[5]

arXiv preprint arXiv:2302.05438 (2023)

De Luigi, L., Cardace, A., Spezialetti, R., Ramirez, P.Z., Salti, S., Di Stefano, L.: Deep learning on implicit neural representations of shapes. arXiv preprint arXiv:2302.05438 (2023)

arXiv 2023
[6]

In: 2009 IEEE conference on computer vision and pattern recognition

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

2009
[7]

In: International Conference on Machine Learning

Dupont, E., Kim, H., Eslami, S.A., Rezende, D.J., Rosenbaum, D.: From data to functa: Your data point is a function and you can treat it like one. In: International Conference on Machine Learning. pp. 5694–5725. PMLR (2022)

work page 2022
[8]

arXiv preprint arXiv:2411.03688 (2024)

Essakine, A., Cheng, Y., Cheng, C.W., Zhang, L., Deng, Z., Zhu, L., Schönlieb, C.B., Aviles-Rivero, A.I.: Where do we stand with implicit neural representations? a technical and performance survey. arXiv preprint arXiv:2411.03688 (2024)

arXiv 2024

Show all 47 references

[9]

In: International conference on machine learning

Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning. pp. 1126–1135. PMLR (2017) 16 S. Janik and M. Byra

2017
[10]

In: Proceedings of the IEEE/CVF international conference on computer vision

Fong, R., Patrick, M., Vedaldi, A.: Understanding deep networks via extremal perturbations and smooth masks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2950–2958 (2019)

2019
[11]

In: Proceedings of the IEEE international conference on computer vision

Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the IEEE international conference on computer vision. pp. 3429–3437 (2017)

2017
[12]

arXiv preprint arXiv:1803.03635 (2018)

Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)

2018 arXiv
[13]

In: International Conference on Machine Learning

Frankle, J., Dziugaite, G.K., Roy, D., Carbin, M.: Linear mode connectivity and the lottery ticket hypothesis. In: International Conference on Machine Learning. pp. 3259–3269. PMLR (2020)

2020
[14]

In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence

Gielisse, A., van Gemert, J.: End-to-end implicit neural representations for classi- fication. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence. pp. 18728–18737 (2025)

2025
[15]

Advances in Neural Information Processing Systems35, 11893–11905 (2022)

Godfrey, C., Brown, D., Emerson, T., Kvinge, H.: On the symmetries of deep learning models and their internal representations. Advances in Neural Information Processing Systems35, 11893–11905 (2022)

2022
[16]

In: International Conference on Machine Learning

Grigsby, E., Lindsey, K., Rolnick, D.: Hidden symmetries of relu networks. In: International Conference on Machine Learning. pp. 11734–11760. PMLR (2023)

2023
[17]

Advances in neural information processing systems28 (2015)

Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. Advances in neural information processing systems28 (2015)

2015
[18]

arXiv preprint arXiv:2408.05451 (2024)

Hänni, K., Mendel, J., Vaintrob, D., Chan, L.: Mathematical models of computa- tion in superposition. arXiv preprint arXiv:2408.05451 (2024)

2024 arXiv
[19]

Information11(2), 108 (2020)

Howard, J., Gugger, S.: Fastai: a layered api for deep learning. Information11(2), 108 (2020)

2020
[20]

In: The Thirteenth International Conference on Learning Representations (2025)

Kahana, J., Horwitz, E., Shuval, I., Hoshen, Y.: Deep linear probe generators for weight space learning. In: The Thirteenth International Conference on Learning Representations (2025)

2025
[21]

arXiv preprint arXiv:2406.10685 (2024)

Kalogeropoulos, I., Bouritsas, G., Panagakis, Y.: Scale equivariant graph metanet- works. arXiv preprint arXiv:2406.10685 (2024)

2024 arXiv
[22]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Kim, C., Lee, D., Kim, S., Cho, M., Han, W.S.: Generalizable implicit neural representations via instance pattern composers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11808–11817 (2023)

2023
[23]

In: The Twelfth International Conference on Learning Repre- sentations (2024)

Kofinas,M.,Knyazev,B.,Zhang,Y.,Chen,Y.,Burghouts,G.J.,Gavves,E.,Snoek, C.G., Zhang, D.W.: Graph neural networks for learning equivariant representations of neural networks. In: The Twelfth International Conference on Learning Repre- sentations (2024)

2024
[24]

Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

2009
[25]

Proceedings of the IEEE86(11), 2278–2324 (1998)

LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE86(11), 2278–2324 (1998)

1998
[26]

arXiv preprint arXiv:2312.04501 (2023)

Lim, D., Maron, H., Law, M.T., Lorraine, J., Lucas, J.: Graph metanetworks for processing diverse neural architectures. arXiv preprint arXiv:2312.04501 (2023)

2023 arXiv
[27]

arXiv preprint arXiv:1711.05101 (2017)

Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[28]

arXiv preprint arXiv:2406.17438 (2024)

Ma, Q., Paudel, D.P., Konukoglu, E., Van Gool, L.: Implicit-zoo: A large-scale dataset of neural implicit functions for 2d images and 3d scenes. arXiv preprint arXiv:2406.17438 (2024)

2024
[29]

arXiv preprint arXiv:1802.03426 (2018) Weight-space mixture-of-experts 17

McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018) Weight-space mixture-of-experts 17

2018 arXiv
[30]

In: International Conference on Machine Learning

Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., Maron, H.: Equiv- ariant architectures for learning in deep weight spaces. In: International Conference on Machine Learning. pp. 25790–25816. PMLR (2023)

2023
[31]

Advances in neural information processing systems29(2016)

Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., Clune, J.: Synthesizing the pre- ferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems29(2016)

2016
[32]

Distill5(3), e00024–001 (2020)

Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., Carter, S.: Zoom in: An introduction to circuits. Distill5(3), e00024–001 (2020)

2020
[33]

In: Proceedings of the British Machine Vision Conference (BMVC) (2018)

Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models. In: Proceedings of the British Machine Vision Conference (BMVC) (2018)

2018
[34]

IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

Ramirez, P.Z., De Luigi, L., Sirocchi, D., Cardace, A., Spezialetti, R., Ballerini, F., Salti, S., Di Stefano, L.: Deep learning on object-centric 3d neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

2024
[35]

Advances in Neural Information Processing Systems34, 8583–8595 (2021)

Riquelme, C., Puigcerver, J., Mustafa, B., Neumann, M., Jenatton, R., Su- sano Pinto, A., Keysers, D., Houlsby, N.: Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems34, 8583–8595 (2021)

2021
[36]

In: Proceedings of the IEEE international conference on computer vision

Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad- cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)

2017
[37]

arXiv preprint arXiv:1701.06538 (2017)

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)

2017 arXiv
[38]

arXiv preprint arXiv:2303.07226 (2023)

Shen, S., Yao, Z., Li, C., Darrell, T., Keutzer, K., He, Y.: Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 (2023)

2023 arXiv
[39]

arXiv preprint arXiv:1312.6034 (2013)

Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional net- works: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)

2013 arXiv
[40]

Advances in neural information processing systems33, 7462–7473 (2020)

Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Advances in neural information processing systems33, 7462–7473 (2020)

2020
[41]

arXiv preprint arXiv:1706.03825 (2017)

Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M.: Smoothgrad: re- moving noise by adding noise. arXiv preprint arXiv:1706.03825 (2017)

2017 arXiv
[42]

Advances in neural informa- tion processing systems33, 7537–7547 (2020)

Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Sing- hal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural informa- tion processing systems33, 753...

2020
[43]

arXiv preprint arXiv:1708.07747 (2017)

Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

2017 arXiv
[44]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Yüce, G., Ortiz-Jiménez, G., Besbinar, B., Frossard, P.: A structured dictionary perspective on implicit neural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19228–19238 (2022)

2022
[45]

Advances in neural information processing systems36(2024)

Zhou,A.,Yang,K.,Burns,K.,Cardace,A.,Jiang,Y.,Sokota,S.,Kolter,J.Z.,Finn, C.: Permutation equivariant neural functionals. Advances in neural information processing systems36(2024)

2024
[46]

Zhou, A., Yang, K., Jiang, Y., Burns, K., Xu, W., Sokota, S., Kolter, J.Z., Finn, C.: Neural functional transformers. Advances in neural information processing systems 36, 77485–77502 (2023) Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification Stan...

2023
[128]

For the inner-loop optimization, we use SGD without momentum, and we optimize the inner-loop learning rates using AdamW with a learning rate of 0.01

The classifier is trained with a learning rate of 0.0001 and weight decay of 0.0001. For the inner-loop optimization, we use SGD without momentum, and we optimize the inner-loop learning rates using AdamW with a learning rate of 0.01. For the SIREN network, we setω= 10.0for MN...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

In: The Eleventh International Conference on Learning Rep- resentations (2023)

Ainsworth, S., Hayase, J., Srinivasa, S.: Git re-basin: Merging models modulo per- mutation symmetries. In: The Eleventh International Conference on Learning Rep- resentations (2023)

work page 2023

[2] [2]

PloS one10(7), e0130140 (2015)

Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one10(7), e0130140 (2015)

2015

[3] [3]

arXiv preprint arXiv:2302.03130 (2023)

Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J.R., Kim, H.: Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130 (2023)

arXiv 2023

[4] [4]

IEEE Transactions on Knowledge and Data Engineering (2025)

Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., Huang, J.: A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering (2025)

2025

[5] [5]

arXiv preprint arXiv:2302.05438 (2023)

De Luigi, L., Cardace, A., Spezialetti, R., Ramirez, P.Z., Salti, S., Di Stefano, L.: Deep learning on implicit neural representations of shapes. arXiv preprint arXiv:2302.05438 (2023)

arXiv 2023

[6] [6]

In: 2009 IEEE conference on computer vision and pattern recognition

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large- scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

2009

[7] [7]

In: International Conference on Machine Learning

Dupont, E., Kim, H., Eslami, S.A., Rezende, D.J., Rosenbaum, D.: From data to functa: Your data point is a function and you can treat it like one. In: International Conference on Machine Learning. pp. 5694–5725. PMLR (2022)

work page 2022

[8] [8]

arXiv preprint arXiv:2411.03688 (2024)

Essakine, A., Cheng, Y., Cheng, C.W., Zhang, L., Deng, Z., Zhu, L., Schönlieb, C.B., Aviles-Rivero, A.I.: Where do we stand with implicit neural representations? a technical and performance survey. arXiv preprint arXiv:2411.03688 (2024)

arXiv 2024