Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones

Andrey Zhmoginov; Jihwan Lee; Mark Sandler

REVIEW 5 major objections 5 minor 44 references

Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that task-conditioned weight projection can turn one large transformer into specialized small transformers that outperform universal conditional models of the same size.

desk verdict A solid, clearly-written empirical study of task-conditioned weight projection for shrinking transformers, with a real but underdetermined central mechanism and some missing controls that should be fixed in revision. read the letter →

arxiv 2506.05641 v1 pith:LHUHLGQ4 submitted 2025-06-06 cs.LG cs.CL

Andrey Zhmoginov , Jihwan Lee , Mark Sandler This is my paper

classification cs.LGcs.CL

keywords task-conditionedweightprojectionmodelcompressionhypernetworkssmalltransformergenerationautoregressiveimagemodelingzero-shotgeneralizationmulti-tasklearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large transformers are expensive to run and carry more general knowledge than most individual tasks need. This paper tries to turn that redundancy into an advantage by generating a small, task-specialized transformer directly from the weights of a large one. The method maps each layer's weight matrix of the large model to a smaller matrix using task-dependent projection matrices, and it co-trains the source model so that many different specialized models can be carved out of one shared store. On image-generation benchmarks the generated small models typically beat universal conditional models of the same size, sometimes by a wide margin, which suggests a route to cheap, personalized deployment of foundation-model knowledge.

What carries the argument

The central object is the task-dependent projection operator $\rho_s(t; M_1)$. It maps the weight matrix $W_1$ of each layer in the large model to a smaller matrix $W_s = P(t) W_1 Q(t)$, where $P(t)$ and $Q(t)$ are sparse matrices whose diagonal entries are emitted by a shallow MLP fed with the task identifier $t$; this amounts to generating the rows and columns of the small weight matrix as linear combinations of $2^k$ rows and columns of the source matrix. The projection operator is learned end-to-end by optimizing the loss of the projected model on task-conditioned data while continuing to train $M_1$, with the source loss added so the foundation model is not degraded. This mechanism is what lets a single large model serve as a shared knowledge store from which specialized submodels of several sizes can be carved out in one forward pass.

What would settle it

Take the trained projection operator for $P_{1/8}$ and evaluate it on a task identifier with scale component $2.0$, as the paper does in Figure 4, then compare the model's loss against an unconditional model of the same size; if the projected model fails to beat the unconditional baseline on such out-of-range identifiers, then the generated weights are not specializing beyond the training distribution.

Watch

Extended reading notes

Core claim

The central claim is that task-conditioned weight projection is an effective one-shot way to produce small specialized transformers from a single large source model. The projected model is defined by $W_s = P(t) W_1 Q(t)$ and $b_s = P(t) b_1$, where $P$ and $Q$ are generated by shallow MLPs from the task identifier $t$, and the whole system—source plus projection operators—is trained on the sum of the projected loss and the source loss. In experiments on SYNTH MNIST and ImageNet, the projected models typically outperform universal conditional models of the same architecture and size; on SYNTH MNIST, the smallest projected model roughly matches a conditional model four times larger, and on ImageNet a projected model with one quarter the embedding size roughly matches a $1.6\times$ larger conditional model.

Load-bearing premise

The load-bearing premise is that a task identifier seen after training is mapped by the projection operator to weights that genuinely specialize to that task, not to weights memorized from the training-task set; the zero-shot experiments in Appendix D.3 show this assumption is only partially met, especially for the smallest projected models.

Editorial extensions

If this is right

On SYNTH MNIST, a projected model with one eighth the embedding size reaches loss 0.57, close to the 0.58 loss of a conditional model four times larger, so small generated models can substitute for much larger universal conditional ones on narrow distributions.
On ImageNet, the projected $P_{1/4}$ roughly matches a $1.6\times$ larger conditional model and $P_{1/8}$ roughly matches a $2.5\times$ larger conditional model, so the advantage persists on realistic, noisier image distributions but shrinks.
A single source model can be jointly trained with projection operators for all sizes $s \in \{1/2, 1/4, 1/8\}$ without degrading any of them, yielding a one-model family that spans different compute budgets.
Training data for some tasks improves projected models on related but different tasks, indicating the source model acts as a shared parameter store across tasks.
Zero-shot generalization is partial: when the task-identifier scale moves from $[0,1]$ to $2$, the largest projected model $P_{1/2}$ degrades less than the conditional baseline, while $P_{1/4}$ and $P_{1/8}$ degrade sharply, showing that the smallest projected models risk memorizing the training-task range.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the projection map from task identifiers to weights is smooth, so interpolating between task IDs may synthesize models for tasks that were never labeled; Figure 4 suggests this is reliable only near the training distribution.
Beyond the paper: since $W_s$ is linear in $W_1$, the family of achievable small models is limited to linear combinations of source rows and columns; a natural test is to insert a nonlinearity between two projection steps and see whether task-tailored expressiveness improves.
Beyond the paper: the advantage over universal conditional models should shrink as task conditionals become broad; a direct comparison on a heterogeneous language benchmark would test whether the specialization margin is a general property or an artifact of narrow image distributions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is genuinely useful: a single large transformer, kept intact, can be paired with cheap task-conditioned diagonal projection operators that generate small specialized transformers at multiple sizes. The experiments on image modeling show that the resulting projected models beat universal conditional models of the same size, and the gap grows at smaller sizes (roughly a 4x parameter advantage on SYNTH MNIST, 1.6-2.5x on ImageNet). The paper is honest about its scope, and the cross-task knowledge transfer results are a nice bonus. The claim that the source model's performance is preserved is also well supported. So the recipe works, and that is the main contribution.

Where the paper is soft, though, is in explaining why it works. The stress-test concern is legitimate: there is no ablation that removes task conditioning (e.g., a constant task identifier) while keeping everything else fixed. The projection operator could be doing little more than distilling the co-trained source model's knowledge into a smaller architecture, and the task-conditioning might be a minor embellishment. The paper's own Appendix D.1 shows that a larger source model improves projected-model loss, which is exactly the kind of distillation signal that would explain the gains without any task-specific projection. So the central mechanism is underdetermined, even if the empirical payoff is real. This is not a fatal flaw, but it needs to be addressed before the paper's interpretation can be taken at face value.

Other soft spots are more routine. The ImageNet s=1/2 comparison is within statistical noise (2.211 vs 2.210 with errors around 0.001-0.002). SYNTH MNIST results come without error bars, so I cannot tell how much of the reported advantage is stable. The scaling estimates that convert loss differences into '1.6x' and '2.5x' larger models are not backed by an explicit fit, making them rough rather than rigorous. The zero-shot generalization experiment in Appendix D.3 is commendable, but it shows P1/8 degrades sharply outside the training range of t_scale, so the method's generality to unseen tasks is fragile at the smallest sizes. Finally, no code or data is released, which limits independent validation.

That said, the central empirical result is not a mirage: the projected models do beat same-size universal conditional baselines, and the improvement is consistent across at least two datasets and several sizes. With the missing ablation and a few clean-ups, this could be a solid contribution to the hypernetwork/compression literature.

My recommendation: send it to peer review. A serious referee should be asked to focus on the task-conditioning ablation, error bars, and the scaling analysis.

— a colleague

Referee Report

5 major / 5 minor

Summary. The paper proposes 'projectable models': given a large Transformer M1 and a task identifier t, learned task-conditioned projection operators P(t), Q(t) generate the weights of smaller Transformers Ps(t) via low-rank row/column combinations of M1's weight matrices. The source model and projectors are trained jointly with the objective in Eq. (2). On SYNTH MNIST and 64x64 ImageNet autoregressive image modeling, the projected models are compared against universal conditional GPT-2 baselines of the same architecture and size. The authors report that projected models achieve lower losses, with P1/8 roughly matching a conditional model four times larger on SYNTH MNIST and smaller margins on ImageNet, and they include additional experiments on source-model size, cross-task transfer, and zero-shot generalization.

Significance. If the reported improvements replicate, the method is a meaningful empirical advance: it provides a single source model that can emit task-specialized small transformers at inference time without per-task fine-tuning, and the projector is parameter-efficient in its input dependence. The paper includes several useful controls beyond the main tables, such as source-size comparison (Table 3), cross-task transfer (Table 4), and a zero-shot scale experiment (Figure 4). The main caveat is that the central mechanism is underdetermined by the current experiments, and some reported margins are within or near the reported noise; the next version should provide the missing controls and error analysis.

major comments (5)

[§3.3.2, Eq. (2)] The central claim that task-conditioned projection produces specialized models is not isolated by an ablation that removes task conditioning. The comparison to universal conditional models tilde{M}_s tests whether conditioning on t helps, but it does not test whether the projector's dependence on t is what matters: a projector trained with a constant or randomized task id, with the same co-training of M1, could show similar gains if the improvement is largely distillation from the tuned source. Please add this control; otherwise the mechanism remains underdetermined.
[Table 2] On ImageNet the s=1/2 entry shows P_s=2.210 versus tilde{M}_s=2.211 with reported statistical errors of about 0.002 and 0.001 respectively; this difference is not significant and should be presented as parity, not as evidence of improvement. The SYNTH MNIST table reports no error bars or seeds, so the headline 'four times larger' matching claim is not yet quantified.
[§3.3.2] The statements that P_{1/4} and P_{1/8} roughly match 1.6x and 2.5x larger conditional models are made without showing the scaling fit or the uncertainty in these factors. Please report the loss-versus-model-size curve used for the interpolation, including confidence intervals, or soften the claims to direct losses.
[§3.3.2 and Table 1] The comparison understates the total parameter cost of the projected approach. For s=1/8 the projector has roughly 1.9M parameters while the generated model has about 1.3M parameters, and the source model is co-trained; the '4x smaller' statement compares only generated-model parameters. Please report projector overhead and discuss when the overhead is amortized.
[Appendix D.3] The zero-shot experiment is a load-bearing check on specialization, and it shows that P_{1/8} degrades sharply when t_scale leaves the training range, as the authors acknowledge. This should be quantified (e.g., loss at t_scale=2 relative to the in-range value) and moved into the main text, because it tempers the claim that the smallest generated models specialize rather than overfit the training task distribution.

minor comments (5)

[Appendix D.3] There is a typo: 'it's zero-shot generalization capability' should be 'its zero-shot generalization capability'.
[Figure 4] The panels would benefit from a legend or caption text explaining which curves correspond to the projected model and the conditional baseline; currently the reader must infer this.
[Appendix D.3] The symbol t is used for the full task identifier and also for the scale component t_scale; please disambiguate the notation.
[Reproducibility] No code or checkpoints are released; given the sensitivity to hyperparameters such as the projector inner dimension r and the source-loss weight w_src, releasing code and training details would substantially aid reproducibility.
[Table 3] The caption states that these are training losses, while the main tables report validation losses; please clarify whether the source-size comparison was also evaluated on a validation split.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported comparisons are empirical measurements, not derivations from fitted constants or self-citations.

full rationale

The paper's central comparison (Ps vs ~Ms in Tables 1-2) is an empirical measurement: both model families are trained with explicit objectives (Eq. 2 for the projected models; standard conditional autoregressive training for the baselines) and then evaluated on the same loss. No reported number is derived from a fitted parameter or from the projection definition by construction. The projection form Ws = P W1 Q is an architectural choice, not a theorem, and the paper does not claim to derive performance from it. The zero-shot experiment (Figure 4) is a genuine extrapolation to out-of-range t_scale values and shows degradation for smaller models, which is an honest falsifiable result. References are to external prior work (Ha et al., Chen et al., Radford et al.); there are no self-citations, uniqueness theorems, or imported ansatze that carry the argument. The absence of a task-agnostic projection control is a possible confound but not a circularity: it does not make any central claim equivalent to its own inputs by construction. Therefore the paper is self-contained against the circularity patterns considered here.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical and does not introduce invented quantities. The key load-bearing assumptions are the adequacy of the diagonal projection form and the generalization of the projection operator to new tasks; both are validated only indirectly through loss measurements.

free parameters (2)

projector inner dimension r = 8 (experiments also use 4-32)
Hyperparameter controlling the MLP width that generates the diagonal entries of P and Q; chosen by the authors without an ablation justifying the value.
source-loss weight w_src = not reported
Equation (2) includes the multiplier w_src that balances the source model loss and the projected model loss; the paper never reports the numerical value used, so the result depends on an undisclosed setting.

assumptions (3)

ad hoc to paper The task-conditional weight manifold theta(t) is smooth and can be approximated by the diagonal projection W_s = P W_1 Q (Eq. 1).
Appendix B motivates the manifold, but the projection form is a design choice, not derived from first principles.
domain assumption Task identifiers t, as real-valued vectors, define well-separated task distributions p(x|t) suitable for training.
Stated in Section 2.2 and used to train both conditional baselines and projection operators; for ImageNet, SimCLRv2 embeddings are assumed to capture task semantics.
domain assumption Co-training M1 with the projection loss does not damage M1's universal performance.
The paper reports M1's loss after co-training matches a standalone model (Section 3.3.2, Table 1), but this is verified only on the same training distribution.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones." pith.science (2026). https://pith.science/paper/LHUHLGQ4

@misc{pith2026250605641,
  author       = {Pith},
  title        = {Pith review of: Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHUHLGQ4}},
  note         = {Machine review of arXiv:2506.05641}
}

read the original abstract

Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models.

Figures

Figures reproduced from arXiv: 2506.05641 by the authors.

**Figure 2.** Projection of the source model weight W1 into the projected model weight Ws. Here t is the task identifier and P(t), Q(t) are “projection matrices” whose diagonal elements are generated using shallow MLPs from t. 2.2. Task-Dependent Matrix Generators Our primary goal is generating small task-specific Transformer models, which requires that the generated weight matrices Ws depend on the task. In the following, we … view at source ↗

**Figure 3.** Examples of 32 × 32 synthetic images generated from the task ID t encoding the background texture (1 of 3 types of the texture, scale, rotation, colors, distortion, etc.) and the overlaid MNIST digit. distribution of images with a given embedding t. 3.2. Experimental Setup: Model Architecture In our SYNTHMNIST experiments, all 32 × 32 RGB input images were quantized by separately mapping each pixel into one of 512 d… view at source ↗

**Figure 4.** Average loss as a function of the scale component tscale (plotted on the x axis) of the task identifier t with the following projected models: (a) P1/2, (b) P1/4, (c) P1/8. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 17 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Hyperstyle: Stylegan inversion with hypernetworks for real image editing

Alaluf, Y., Tov, O., Mokady, R., Gal, R., and Bermano, A. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pp.\ 18511--18521, 2022

work page 2022
[3]

Generative pretraining from pixels

Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pp.\ 1691--1703. PMLR , 2020 a

work page 2020
[4]

Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12,...

work page 2020
[5]

Streamlining redundant layers to compress large language models

Chen, X., Hu, Y., and Zhang, J. Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403.19135, 2024

arXiv 2024
[6]

Deb, B., Zheng, G., and Awadallah, A. H. Boosting natural language generation from instructions with meta-learning. arXiv preprint arXiv:2210.11617, 2022

work page Pith review arXiv 2022
[7]

Imagenet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei - Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA , pp.\ 248--255. IEEE Computer Society, 2009. doi:10.1109/CVPR.2009.5206848

arXiv 2009
[8]

M., Tran, A

Dinh, T. M., Tran, A. T., Nguyen, R., and Hua, B.-S. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11389--11398, 2022

work page 2022

Show all 44 references

[9]

M., and Le, Q

Ha, D., Dai, A. M., and Le, Q. V. Hypernetworks. CoRR, abs/1609.09106, 2016. URL http://arxiv.org/abs/1609.09106

2016 arXiv
[10]

Hyperprompt: Prompt-based task-conditioning of transformers

He, Y., Zheng, S., Tay, Y., Gupta, J., Du, Y., Aribandi, V., Zhao, Z., Li, Y., Chen, Z., Metzler, D., et al. Hyperprompt: Prompt-based task-conditioning of transformers. In International conference on machine learning, pp.\ 8678--8690. PMLR, 2022

2022
[11]

and Peters, M

Ivison, H. and Peters, M. E. Hyperdecoders: Instance-specific decoders for multi-task nlp. arXiv preprint arXiv:2203.08304, 2022

2022 arXiv
[12]

Hint: Hypernetwork instruction tuning for efficient zero-& few-shot generalisation

Ivison, H., Bhagia, A., Wang, Y., Hajishirzi, H., and Peters, M. Hint: Hypernetwork instruction tuning for efficient zero-& few-shot generalisation. arXiv preprint arXiv:2212.10315, 2022

2022 arXiv
[13]

Scaling up gans for text-to-image synthesis

Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10124--10134, 2023

2023
[14]

W., and Romero Soriano, A

Knyazev, B., Drozdzal, M., Taylor, G. W., and Romero Soriano, A. Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems, 34: 0 29433--29448, 2021

2021
[15]

Mend: Meta demonstration distillation for efficient and effective in-context learning

Li, Y., Ma, X., Lu, S., Lee, K., Liu, X., and Guo, C. Mend: Meta demonstration distillation for efficient and effective in-context learning. arXiv preprint arXiv:2403.06914, 2024

2024 arXiv
[16]

Hart: Efficient adaptation via regularized autoregressive parameter generation

Liang, C., Karampatziakis, N., Zhao, T., and Chen, W. Hart: Efficient adaptation via regularized autoregressive parameter generation. 2023

2023
[17]

Weight distillation: Transferring the knowledge in neural network parameters

Lin, Y., Li, Y., Wang, Z., Li, B., Du, Q., Xiao, T., and Zhu, J. Weight distillation: Transferring the knowledge in neural network parameters. arXiv preprint arXiv:2009.09152, 2020

2009 arXiv
[18]

and Wolf, L

Littwin, G. and Wolf, L. Deep meta functionals for shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 1824--1833, 2019

2019
[19]

Llm-pruner: On the structural pruning of large language models

Ma, X., Fang, G., and Wang, X. Llm-pruner: On the structural pruning of large language models. CoRR, abs/2305.11627, 2023. doi:10.48550/ARXIV.2305.11627. URL https://doi.org/10.48550/arXiv.2305.11627

2023 doi arXiv
[20]

K., Ruder, S., Dehghani, M., and Henderson, J

Mahabadi, R. K., Ruder, S., Dehghani, M., and Henderson, J. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489, 2021

2021 arXiv
[21]

Learning to compress prompts with gist tokens

Mu, J., Li, X., and Goodman, N. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024

2024
[22]

Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation

Nirkin, Y., Wolf, L., and Hassner, T. Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 4061--4070, 2021

2021
[23]

Investigating the effectiveness of hypertuning via gisting

Phang, J. Investigating the effectiveness of hypertuning via gisting. arXiv preprint arXiv:2402.16817, 2024

2024 arXiv
[24]

Hypertuning: Toward adapting large language models without back-propagation

Phang, J., Mao, Y., He, P., and Chen, W. Hypertuning: Toward adapting large language models without back-propagation. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July...

2023
[25]

Language models are unsupervised multitask learners

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

2019
[26]

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

2020
[27]

High-resolution image synthesis with latent diffusion models

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

2022
[28]

Weight subcloning: direct initialization of transformers using larger pretrained ones

Samragh, M., Farajtabar, M., Mehta, S., Vemulapalli, R., Faghri, F., Naik, D., Tuzel, O., and Rastegari, M. Weight subcloning: direct initialization of transformers using larger pretrained ones. CoRR, abs/2312.09299, 2023. doi:10.48550/ARXIV.2312.09299. URL https://doi.org/10....

2023 doi arXiv
[29]

Implicit neural representations with periodic activation functions

Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 0 7462--7473, 2020

2020
[30]

K., Tabor, J., Trzci \'n ski, T., et al

Spurek, P., Kasymov, A., Mazur, M., Janik, D., Tadeja, S. K., Tabor, J., Trzci \'n ski, T., et al. HyperPocket: Generative point cloud completion . In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.\ 6848--6853. IEEE, 2022

2022
[31]

W., and Schwarz, J

Tack, J., Kim, J., Mitchell, E., Shin, J., Teh, Y. W., and Schwarz, J. R. Online adaptation of language models with a memory of amortized contexts. arXiv preprint arXiv:2403.04317, 2024

2024 arXiv
[32]

A survey on transformer compression

Tang, Y., Wang, Y., Guo, J., Tu, Z., Han, K., Hu, H., and Tao, D. A survey on transformer compression. arXiv preprint arXiv:2402.05964, 2024

2024 arXiv
[33]

Hypergrid transformers: Towards a single model for multiple tasks

Tay, Y., Zhao, Z., Bahri, D., Metzler, D., and Juan, D.-C. Hypergrid transformers: Towards a single model for multiple tasks. In International conference on learning representations, 2020

2020
[34]

Neural discrete representation learning

Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

2017
[35]

Example-based hypernetworks for multi-source adaptation to unseen domains

Volk, T., Ben-David, E., Amosy, O., Chechik, G., and Reichart, R. Example-based hypernetworks for multi-source adaptation to unseen domains. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023

2023
[36]

F., and Sacramento, J

Von Oswald, J., Henning, C., Grewe, B. F., and Sacramento, J. Continual learning with hypernetworks. arXiv preprint arXiv:1906.00695, 2019

1906 arXiv
[37]

Learngene: Inheriting condensed knowledge from the ancestry model to descendant models

Wang, Q., Yang, X., Lin, S., Wang, J., and Geng, X. Learngene: Inheriting condensed knowledge from the ancestry model to descendant models. arXiv preprint arXiv:2305.02279, 2023

2023 arXiv
[38]

Model compression and efficient inference for large language models: A survey

Wang, W., Chen, W., Luo, Y., Long, Y., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748, 2024

2024 arXiv
[39]

Sheared llama: Accelerating language model pre-training via structured pruning

Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694, 2023

2023 arXiv
[40]

Initializing models with larger ones

Xu, Z., Chen, Y., Vishniakov, K., Yin, Y., Shen, Z., Darrell, T., Liu, L., and Liu, Z. Initializing models with larger ones. arXiv preprint arXiv:2311.18823, 2023

2023 arXiv
[41]

and Ren, X

Ye, Q. and Ren, X. Learning to generate task-specific adapters from task description. arXiv preprint arXiv:2101.00420, 2021

2021 arXiv
[42]

Graph hypernetworks for neural architecture search

Zhang, C., Ren, M., and Urtasun, R. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018

2018 arXiv
[43]

Adding conditional control to text-to-image diffusion models

Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 3836--3847, 2023

2023
[44]

Hypermoe: Towards better mixture of experts via transferring among experts

Zhao, H., Qiu, Z., Wu, H., Wang, Z., He, Z., and Fu, J. Hypermoe: Towards better mixture of experts via transferring among experts. arXiv preprint arXiv:2402.12656, 2024

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Hyperstyle: Stylegan inversion with hypernetworks for real image editing

Alaluf, Y., Tov, O., Mokady, R., Gal, R., and Bermano, A. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pp.\ 18511--18521, 2022

work page 2022

[3] [3]

Generative pretraining from pixels

Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research, pp.\ 1691--1703. PMLR , 2020 a

work page 2020

[4] [4]

Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12,...

work page 2020

[5] [5]

Streamlining redundant layers to compress large language models

Chen, X., Hu, Y., and Zhang, J. Streamlining redundant layers to compress large language models. arXiv preprint arXiv:2403.19135, 2024

arXiv 2024

[6] [6]

Deb, B., Zheng, G., and Awadallah, A. H. Boosting natural language generation from instructions with meta-learning. arXiv preprint arXiv:2210.11617, 2022

work page Pith review arXiv 2022

[7] [7]

Imagenet: A large-scale hierarchical image database

Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei - Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA , pp.\ 248--255. IEEE Computer Society, 2009. doi:10.1109/CVPR.2009.5206848

arXiv 2009

[8] [8]

M., Tran, A

Dinh, T. M., Tran, A. T., Nguyen, R., and Hua, B.-S. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11389--11398, 2022

work page 2022