Pith. sign in

REVIEW 3 major objections 5 minor 5 cited by

Text-to-LoRA: Instant Transformer Adaption

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims a hypernetwork can generate task-specific LoRA adapters for unseen tasks from a text description alone, in a single forward pass.

desk verdict Solid empirical study with an over-stated zero-shot claim; the evaluation benchmarks sit close to the SNI training distribution, and the only clearly out-of-distribution tasks show no clear win over multi-task LoRA. read the letter →

arxiv 2506.06105 v2 pith:7JIWQ2WP submitted 2025-06-06 cs.LG cs.AI

Rujikorn Charakorn , Edoardo Cetin , Yujin Tang , Robert Tjarko Lange This is my paper · ORCID
classification cs.LGcs.AI
keywords hypernetworkLoRAzero-shotadaptationlargelanguagemodelsparameter-efficientfine-tuningtaskdescriptionsmulti-tasksupervisedadaptercompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Text-to-LoRA (T2L) tries to make LLM adaptation as cheap as typing a sentence. The authors claim a hypernetwork can read a natural-language description of a task and directly output the LoRA matrices that would normally come from hours of fine-tuning, in a single forward pass and without any gradient updates. Trained end-to-end with supervised fine-tuning on hundreds of SNI tasks, it beats a multi-task LoRA baseline on ten unseen benchmarks and compresses many adapters into one network. The paper is careful that zero-shot gains track how close the test task is to the training distribution: code-generation benchmarks, absent from SNI, show small gains over the base model. The intended payoff is that task adaptation becomes a text-prompt interaction rather than a data-and-GPU engineering job.

What carries the argument

The carrier of the argument is a hypernetwork $h_\theta$ whose input is a concatenation of a text embedding $f(z_i)$ (from a bidirectional task encoder), a learnable module embedding $E[m]$, and a layer embedding $E[l]$; optional A/B and rank embeddings are added in smaller variants. The output head writes the low-rank $A$ and $B$ matrices of a LoRA adapter for every targeted query/value projection across all layers, batching all layer/module inputs in one pass. Three architectures (L, M, S) trade output-head size and inductive bias against capacity. Training is either reconstruction of pre-trained adapters or direct supervised fine-tuning; the SFT objective is what yields zero-shot generalization, while reconstruction loses that ability because independently trained adapters for similar tasks are not close in parameter space.

What would settle it

Run T2L(SFT)L on task types that are absent from SNI, such as a held-out code-generation suite or a non-English QA suite, and compare zero-shot performance against the multi-task LoRA baseline using the paper's own protocol. The paper's numbers already show small gains on HumanEval (39.2 vs 37.2 base); if the same pattern appears broadly, the central claim must be restated as generalization within the SNI task neighborhood rather than to arbitrary tasks.

Watch

Extended reading notes

Core claim

The paper's central claim is that a hypernetwork — a network that outputs parameters for another network — can be trained to map a natural-language task description to the low-rank adapter matrices (LoRA) that specialize a frozen LLM to that task. T2L is trained with supervised fine-tuning across 479 tasks from the SNI distribution, with descriptions generated in bulk to encourage diversity. At test time, it produces the whole adapter in one forward pass. On ten unseen benchmarks, the largest variant averages 67.7% versus 66.3% for a multi-task LoRA baseline and 67.3% for the per-instance Hyperdecoders baseline, and it beats task-specific oracle LoRAs on a few individual benchmarks even though it undershoots their average. The paper also emphasizes compression: a single trained hypernetwork can reproduce the behavior of hundreds of individual adapters, and reconstruction-trained versions match oracle performance when the reconstruction error is low.

Load-bearing premise

T2L's zero-shot generality depends on the training distribution (SNI tasks with generated descriptions) adequately covering the kinds of tasks users will describe; the paper itself notes code generation is absent from training and that most benchmarks overlap with SNI task types.

Editorial extensions

If this is right

  • A user can adapt a frozen LLM to a new task from a task description alone, with no dataset collection and no backpropagation through the base model.
  • One trained hypernetwork can compress hundreds of LoRA adapters into a single set of weights and emit the right adapter on demand.
  • Zero-shot gains improve as the number of training tasks and the compute budget grow together, suggesting the approach scales with task diversity.
  • The same hypernetwork weights transfer to Llama-3.1-8B-Instruct and Gemma-2-2B-Instruct, so the mechanism is not tied to one base model.
  • Lossy reconstruction can act as regularization, so decoded adapters occasionally beat the task-specific LoRAs they were trained to reproduce.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the input is just a text embedding, the same hypernetwork recipe should extend to other parameter-efficient adapters such as prefixes, adapters, or direct activation modulation; the paper only tries LoRA, so this is an extrapolation.
  • The reconstruction-trained failure suggests a concrete prerequisite for weight-generating hypernetworks: the adapter library must be produced under a shared objective, or the mapping from task semantics to weights must be learned rather than assumed.
  • The sensitivity to description quality implies a deployment-time guard could filter or rewrite user descriptions before generating an adapter, since misaligned strings degrade performance sharply.
  • If T2L scales with task diversity, the natural next test is to include code generation and other absent task types in the training mixture and see whether the HumanEval and MBPP gaps close.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Text-to-LoRA (T2L), a hypernetwork that maps task-description text embeddings to LoRA adapters for a frozen base LLM. T2L can be trained either to reconstruct pre-trained LoRA weights or end-to-end by supervised fine-tuning on a multi-task SNI dataset. The central claims are (i) T2L compresses hundreds of LoRAs while roughly preserving task-specific LoRA performance, and (ii) an SFT-trained T2L produces useful LoRA adapters zero-shot for unseen benchmark tasks, outperforming a multi-task LoRA baseline and the Arrow Routing baseline. Experiments use Mistral-7B-Instruct, Llama-3.1-8B-Instruct, and Gemma-2-2B-Instruct across 10 benchmarks, with ablations on architecture, training scheme, description sources, and scaling.

Significance. If the central claims held in full, T2L would be a practically valuable contribution: it offers single-forward-pass adaptation with no per-task gradient training, lossy compression of LoRA libraries, and a demonstration that language descriptions alone can steer adapters. The paper is honest about many limitations (Section 7) and provides code, multiple base-model checks, and a useful analysis (Appendix D) of why reconstruction-trained hypernetworks fail to generalize. However, the zero-shot generalization claim is partially undermined by distribution overlap between the SNI training pool and most evaluation benchmarks, and the comparisons against Arrow Routing are indirect. The result is an interesting and well-executed empirical study, but the headline claims need stronger evidential support.

major comments (3)
  1. [Section 4.2, Table 2; Appendix J; footnote in Section 4] The claim of zero-shot generalization to 'entirely unseen tasks' is not supported by the evidence in Table 2, because most evaluation benchmarks have near analogues in the SNI training pool. As the paper's own footnote and Appendix J show, training includes task228_arc_answer_generation_easy, task380_boolq_yes_no_question, task033_winogrande_answer_generation, task081_piqa_wrong_answer_generation, task1286_openbookqa_question_answering, task1389_hellaswag_completion, and multiple arithmetic QA tasks such as task867_mawps_multiop_question_answering. For the two families the paper identifies as absent from training (HumanEval and MBPP), Table 2 shows T2L(SFT)L at 39.2 and 51.9 versus MT-LoRA at 39.6 and 48.1, i.e., one loss and one gain on small margins. The aggregate 67.7 vs 66.3 advantage is therefore dominated by tasks near the training distribution. The authors should either provide an explicit out-of-distribution analysis (e.g., code-only average, or per-task distribution-similarity plots) with variance estimates, or soften the 'unseen task' claim accordingly.
  2. [Section 4 and Table 2; Introduction contribution list] Table 2 and Section 4 compare T2L with Arrow Routing results taken from Ostapenko et al. (2024) and state that the comparison is indirect, using different LoRA adapters, training tasks, and likely different benchmark prompts. Despite this, the Introduction and contribution list claim T2L 'outperforms ... Arrow Routing'. Since Arrow Routing scores are not produced under the same evaluation protocol (e.g., GSM8K is N/A and HumanEval is 28.7), this claim is not substantiated. A direct re-implementation or a clearly caveated claim is needed.
  3. [Section 4.2, Tables 2, 3, 7, 8] No uncertainty quantification is reported anywhere in Section 4.2. T2L scores in Table 2 are averages over three generated LoRAs, but the individual values, standard deviations, or multiple training seeds are not reported; Tables 3, 7, and 8 are single-run numbers. Given that several head-to-head differences against MT-LoRA are on the order of one point (e.g., ArcE 88.9 vs 88.3, WG 64.2 vs 62.4, MBPP 51.9 vs 48.1), the claim of 'consistently' improving over MT-LoRA needs confidence intervals or a significance test to be credible.
minor comments (5)
  1. [Equation (4)] Equation (4) has a formatting typo: 'concat[f (zi), E[m], E[l]]]' has an extra closing bracket; the notation should be cleaned up.
  2. [Figure 1 (bottom right)] The horizontal axis label reads '489' but the text and Table 3 use 479 training tasks; this appears to be a typo.
  3. [Listing 4 (PIQA_TEMPLATE)] The PIQA_TEMPLATE string contains an unclosed quotation mark after 'more convenient or more common.'; the code listing should be corrected.
  4. [Section 5.4] The statement that reconstruction-trained T2L 'fails to generalize' is stronger than the evidence in Table 6 warrants, since reconstruction-trained T2L still improves over the base model (61.8 vs 55.8); a more precise wording such as 'generalizes substantially worse than SFT-trained T2L' would be more accurate.
  5. [Appendix K] The paragraph in Appendix K is confusing: it first says that computing the reconstruction loss in the full adaptation matrix space 'could generalize better' and then immediately says 'we empirically find that it does not outperform'; this should be rewritten for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: T2L's zero-shot claim is trained on SNI tasks and evaluated on held-out benchmarks; reconstruction results are explicitly labeled as compression.

full rationale

The claimed derivation chain is: (i) train the hypernetwork h_theta via the SFT loss of Eq. (5) on 479 SNI tasks; (ii) at inference, embed a task description and generate LoRA weights via Eqs. (3)-(4); (iii) measure on 10 held-out benchmarks in Table 2. No benchmark LoRA, benchmark label, or benchmark performance enters the SFT training objective, so the zero-shot prediction is not fitted to the evaluation quantity by construction. The reconstruction experiments in Table 1 are explicitly presented as distillation/compression: the paper states that 'the benchmark tasks are indirectly seen during training by T2L, as it learns to distill benchmark-specific LoRAs,' so matching the oracle adapters there is a compression property, not a prediction. The paper's own footnote discloses that several benchmarks share domains with SNI tasks and that MBPP/HumanEval are different; this is an honest limitation on the strength of 'unseen,' not a circular step. The generation of benchmark task descriptions from five sampled QA pairs is a possible leakage/contamination risk, but it is not an equivalence-by-construction: T2L never sees the benchmark labels or adapters, and the description embedding is not fit to the evaluation metric. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling citation is present; comparisons use independent baselines such as MT-LoRA, Hyperdecoders, and Arrow Routing. A separate methodological caveat is that Table 2 reports the 'best model of each variant from our scaling experiment in Section 5.1,' i.e., model selection on the benchmarks, which can inflate zero-shot numbers, but that is a selection-bias concern rather than circular reasoning. Overall, the core derivation is self-contained and empirically falsifiable.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. The listed free parameters are standard hyperparameters and architecture choices that affect performance but are not fitted to a single target result. The core axioms are domain assumptions about the existence of a shared adaptation manifold and the informativeness of natural language descriptions for generating LoRAs.

free parameters (6)
  • LoRA rank r = 8
    Fixed for all LoRA adapters; determines low-rank dimension of adapters.
  • LoRA alpha = 16
    Scaling factor for LoRA update, from PEFT config in Listing 1.
  • Lora dropout = 0.05
    Dropout applied in LoRA layers.
  • NEFTune noise alpha = 5.0
    Noise level used during supervised fine-tuning of T2L.
  • Number of task descriptions per task = 128
    Descriptions sampled online during SFT; set based on Appendix N.
  • Hypernetwork output dimension = d_out=512, task encoder 64D, embeddings 32D, A/B 128D, rank 128D
    Architecture sizes chosen for L/M/S variants in Appendix F.
assumptions (5)
  • domain assumption A shared low-dimensional adaptation mechanism exists across tasks that a hypernetwork can capture.
    Stated as a hypothesis in Section 1: 'We hypothesize that different LoRA adapters share the same underlying adaptation mechanism.' This underpins the entire approach.
  • domain assumption Natural language task descriptions provide sufficient conditioning signal for generating useful LoRA adapters.
    Central to the method; evaluated in Section 5.3 where unaligned descriptions hurt performance.
  • domain assumption The text embedding model (gte-large-en-v1.5) captures task semantics relevant to LoRA generation.
    Used for all task embeddings; robustness checked with Mistral embeddings in Section 5.2.
  • domain assumption SNI task distribution supports zero-shot transfer to the selected benchmarks.
    Assumed in Section 4.2; the paper notes benchmarks share similarities with SNI, and the authors removed 10 contaminated datasets.
  • domain assumption L1 loss on LoRA weight entries is a suitable training objective for reconstruction.
    Section 3.2; Appendix D shows this leads to poor generalization for reconstruction-trained T2L.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Text-to-LoRA: Instant Transformer Adaption." pith.science (2026). https://pith.science/paper/7JIWQ2WP

@misc{pith2026250606105,
  author       = {Pith},
  title        = {Pith review of: Text-to-LoRA: Instant Transformer Adaption},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7JIWQ2WP}},
  note         = {Machine review of arXiv:2506.06105}
}
read the original abstract

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model. Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices. To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass. After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets. Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks. This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements. Our code is available at https://github.com/SakanaAI/text-to-lora

Figures

Figures reproduced from arXiv: 2506.06105 by the authors.

Figure 1
Figure 1. Left: Conceptual overview of T2L’s training routine. Given a set of task description embeddings, we train a hypernetwork to generate LoRA adaptation matrices (∆W) for various tasks. The weights of T2L are either optimized to distill pre-trained LoRA weights or via multi-task supervised fine-tuning on downstream tasks. Right, Top: Relative performance to the oracles on training SNI tasks with varying compression rati… view at source ↗
Figure 2
Figure 2. Overview of T2L architectural variations. The dashed box at the bottom shows the output size of a single forward pass of T2L. Blue boxes are trainable modules. Cyan boxes are trainable embedding layers. Components in dashed boxes are only used with their corresponding architectures. r is the rank of a LoRA adapter and d is the size of the input and the output dimension. where LSFT gives the supervised fine-tuning lo… view at source ↗
Figure 3
Figure 3. Relative performance and training reconstruction [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Qualitative examples of responses from applying LoRA generated by [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: 2D t-SNE projection of activations of T2L’s task encoder (left) and activations of the last MLP block (right) grouped by benchmark tasks (represented by colors). We probe T2L with unseen three task descriptions per benchmark. We can see activations clustering in both p…
Figure 6
Figure 6. Figure 6: Top row: Each plot shows the similarity between a benchmark LoRA adapter and 479 SNI-trained adapters in the weight space (y-axis) against their similarity in the task embedding space (x-axis). Bottom row: Each plot shows SNI-trained adapters’ performance relative to a…
Figure 7
Figure 7. Figure 7: Training tasks from Lots-of-LoRAs (based on the SNI dataset) used for training the Text-to-LoRA model. The [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Training tasks from Lots-of-LoRAs (based on the SNI dataset) used for training the Text-to-LoRA model. The [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Training tasks from Lots-of-LoRAs (based on the SNI dataset) used for training the Text-to-LoRA model. The [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Validation tasks used during the training of the Text-to-LoRA model. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Each plot shows the similarity between a benchmark LoRA adapter and 479 SNI-trained adapters in the [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: The prompt template used to query GPT-4o mini for task descriptions. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Examples of training descriptions from three SNI training tasks. [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Task descriptions of the benchmark tasks: boolq, gsm8k, and humaneval. [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: Task descriptions of the benchmark tasks: mbpp, winogrande, piqa [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Task descriptions of the benchmark tasks: hellaswag, arc [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: Task descriptions of the benchmark tasks: openbookqa [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: Random descriptions 27 [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: Zero-shot benchmark performance of SFT-trained [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KV-Skill: Forging Expertise in the Model's Native Language

    cs.LG 2026-08 conditional novelty 6.0 of 10

    Task knowledge can be compressed into an external factorized operator that a frozen LLM reads through a trained interface, and this representation outperforms text and matched soft-prefix, prefix, and LoRA baselines o...

  2. DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.

  3. Learning Only What Valid Adapters Can Express: Subspace-Constrained Adaptation Against Fine-Tuning Poisoning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Restricting LoRA fine-tuning to the subspace of 196 trusted adapters blocks label-inversion poisoning and provides a built-in OOD signal, at the cost of a plasticity ceiling on poorly-covered tasks.

  4. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

  5. GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models

    cs.CL 2025-11 reject novelty 5.0 of 10

    An LLM can memorize a knowledge graph into LoRA weights and answer relation/reasoning queries about it without graph context, but the evaluation partly trains on the test task.

Reference graph

Works this paper leans on

53 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [4]

    T., Vuorio, R., and Whiteson, S

    Beck, J., Jackson, M. T., Vuorio, R., and Whiteson, S. Hypernetworks in meta-reinforcement learning. In Conference on Robot Learning, pp.\ 1478--1487. PMLR, 2023

    work page 2023

  5. [5]

    LLM2V ec: Large language models are secretly powerful text encoders

    BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. LLM2V ec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=IW1PR7vEBf

    work page 2024

  6. [6]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  7. [7]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    1901

  8. [8]

    Compress then serve: Serving thousands of lora adapters with little overhead

    Br \"u el-Gabrielsson, R., Zhu, J., Bhardwaj, O., Choshen, L., Greenewald, K., Yurochkin, M., and Solomon, J. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., B...

    2021

  2. [10]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  3. [11]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  4. [12]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    H., and Zheng, G

    Deb, B., Awadallah, A. H., and Zheng, G. Boosting natural language generation from instructions with meta-learning. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 6792--6808, Abu...

    2022 doi

  6. [14]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  7. [15]

    Digre, K. B. and Brennan, K. Shedding light on photophobia. Journal of Neuro-ophthalmology, 32 0 (1): 0 68--81, 2012

    2012

  8. [16]

    A survey on in-context learning

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 1107--1128, 2024

    2024

  9. [17]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Gururangan, S., Marasovi \'c , A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020

    2004 arXiv

  11. [19]

    Ha, D., Dai, A., and Le, Q. V. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016

    2016 arXiv

  12. [20]

    Hyperprompt: Prompt-based task-conditioning of transformers

    He, Y., Zheng, S., Tay, Y., Gupta, J., Du, Y., Aribandi, V., Zhao, Z., Li, Y., Chen, Z., Metzler, D., et al. Hyperprompt: Prompt-based task-conditioning of transformers. In International conference on machine learning, pp.\ 8678--8690. PMLR, 2022

    2022

  13. [21]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  14. [22]

    and Peters, M

    Ivison, H. and Peters, M. E. Hyperdecoders: Instance-specific decoders for multi-task nlp. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 1715--1730, 2022

    2022

  15. [23]

    Ivison, H., Bhagia, A., Wang, Y., Hajishirzi, H., and Peters, M. E. Hint: Hypernetwork instruction tuning for efficient zero-and few-shot generalisation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11...

    2023

  16. [24]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  17. [25]

    B., Chandra, B., and Yejin, C

    Keisuke, S., Ronan, L. B., Chandra, B., and Yejin, C. Winogrande: An adversarial winograd schema challenge at scale. 2019

    2019

  18. [26]

    A single linear layer yields task-adapted low-rank matrices

    Kim, H., Sasaki, S., Hoshino, S., and Honda, U. A single linear layer yields task-adapted low-rank matrices. arXiv preprint arXiv:2403.14946, 2024

    2024 arXiv

  19. [27]

    J., Blankevoort, T., and Asano, Y

    Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. Ve RA : Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NjNfLdxr3A

    2024

  20. [28]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  21. [29]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019

    1910 arXiv

  22. [30]

    Towards general text embeddings with multi-stage contrastive learning

    Li, Z., Zhang, X., Zhang, Y., Long, D., Xie, P., and Zhang, M. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023

    2023 arXiv

  23. [31]

    S., Wang, Y., and Zhang, L

    Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=1qvx610Cu7

    2023

  24. [32]

    Hyperlora: Efficient cross-task generalization via constrained low-rank adapters generation

    Lv, C., Li, L., Zhang, S., Chen, G., Qi, F., Zhang, N., and Zheng, H.-T. Hyperlora: Efficient cross-task generalization via constrained low-rank adapters generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 16376--16393, 2024

    2024

  25. [33]

    K., Ruder, S., Dehghani, M., and Henderson, J

    Mahabadi, R. K., Ruder, S., Dehghani, M., and Henderson, J. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489, 2021

    2021 arXiv

  26. [34]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018

    2018

  27. [35]

    Learning to compress prompts with gist tokens

    Mu, J., Li, X., and Goodman, N. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [36]

    Hyperloader: Integrating hypernetwork-based lora and adapter layers into multi-task transformers for sequence labelling

    Ortiz-Barajas, J.-G., Gomez-Adorno, H., and Solorio, T. Hyperloader: Integrating hypernetwork-based lora and adapter layers into multi-task transformers for sequence labelling. arXiv preprint arXiv:2407.01411, 2024

    2024 arXiv

  29. [37]

    M., Charlin, L., Roux, N

    Ostapenko, O., Su, Z., Ponti, E. M., Charlin, L., Roux, N. L., Pereira, M., Caccia, L., and Sordoni, A. Towards modular llms by building and reusing a library of loras. arXiv preprint arXiv:2405.11157, 2024

    2024 arXiv

  30. [38]

    Hypertuning: Toward adapting large language models without back-propagation

    Phang, J., Mao, Y., He, P., and Chen, W. Hypertuning: Toward adapting large language models without back-propagation. In International Conference on Machine Learning, pp.\ 27854--27875. PMLR, 2023

    2023

  31. [39]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  32. [40]

    Discovering neural nets with low kolmogorov complexity and high generalization capability

    Schmidhuber, J. Discovering neural nets with low kolmogorov complexity and high generalization capability. Neural Networks, 10 0 (5): 0 857--873, 1997

    1997

  33. [41]

    Attention as a hypernetwork

    Schug, S., Kobayashi, S., Akram, Y., Sacramento, J., and Pascanu, R. Attention as a hypernetwork. arXiv preprint arXiv:2406.05816, 2024

    2024 arXiv

  34. [42]

    Stanley, K. O. and Miikkulainen, R. A taxonomy for artificial embryogeny. Artificial life, 9 0 (2): 0 93--130, 2003

    2003

  35. [43]

    W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D

    Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686, 2021

    2021 arXiv

  36. [44]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  37. [45]

    F., and Sacramento, J

    Von Oswald, J., Henning, C., Grewe, B. F., and Sacramento, J. Continual learning with hypernetworks. arXiv preprint arXiv:1906.00695, 2019

    1906 arXiv

  38. [46]

    S., Arunkumar, A., Stap, D., et al

    Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Emp...

    2022

  39. [47]

    Y., Guu, K., Yu, A

    Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  40. [48]

    H., Joiner, W

    Wurtz, R. H., Joiner, W. M., and Berman, R. A. Neuronal mechanisms for visual stability: progress and problems. Philosophical Transactions of the Royal Society B: Biological Sciences, 366 0 (1564): 0 492--503, 2011

    2011

  41. [49]

    Task-agnostic low-rank adapters for unseen E nglish dialects

    Xiao, Z., Held, W., Liu, Y., and Yang, D. Task-agnostic low-rank adapters for unseen E nglish dialects. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 7857--7870, Singapore, December 2...

    2023 doi

  42. [50]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  43. [51]

    Graph hypernetworks for neural architecture search

    Zhang, C., Ren, M., and Urtasun, R. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018

    2018 arXiv

  44. [52]

    mgte: Generalized long-context text representation and reranking models for multilingual text retrieval

    Zhang, X., Zhang, Y., Long, D., Xie, W., Dai, Z., Tang, J., Lin, H., Yang, B., Xie, P., Huang, F., et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arXiv:2407.19669, 2024

    2024 arXiv

  45. [53]

    Lora land: 310 fine-tuned llms that rival gpt-4, a technical report

    Zhao, J., Wang, T., Abid, W., Angus, G., Garg, A., Kinnison, J., Sherstinsky, A., Molino, P., Addair, T., and Rishi, D. Lora land: 310 fine-tuned llms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.00732, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.