Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Transformer-Squared: Self-adaptive LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Scaling only the singular values of a frozen LLM's weights, trained by reinforcement learning, adapts it to unseen tasks at inference time through a two-pass dispatch.

desk verdict SVF's RL-trained singular-value scaling is a clever new combination, but the paper overclaims 'consistently outperforms LoRA' and the single-run numbers lack the error bars needed to back it. read the letter →

arxiv 2501.06252 v3 pith:QWOSZJWJ submitted 2025-01-09 cs.LG cs.AIcs.CL

Qi Sun , Edoardo Cetin , Yujin Tang This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords self-adaptiveLLMssingularvaluefine-tuningSVFreinforcementlearningparameter-efficienttwo-passinferencecross-modeltransfermixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Transformer-Squared is built on one strong claim: an LLM trained on broad data already contains the directions it needs for new tasks inside its frozen weight matrices, so adapting it can be reduced to rescaling the singular values of those matrices. The paper proposes Singular Value Fine-tuning (SVF), which learns one small scaling vector per weight matrix by reinforcement learning, and then a two-pass inference scheme that identifies the incoming task and mixes pre-trained expert vectors to adapt the model on the spot. If the claim holds, task-specific adaptation no longer requires large low-rank fine-tuning modules: SVF reports better task performance than LoRA with orders of magnitude fewer trainable parameters, and the same expert vectors improve unseen benchmarks such as MATH, HumanEval, ARC-Challenge, and even visual question answering. This matters for the practical goal of language models that can be specialized cheaply and composed dynamically.

What carries the argument

The load-bearing object is the singular-value scaling vector $z$, one per fine-tuned weight matrix, defined by $W' = U \Sigma' V^\top$ with $\Sigma' = \Sigma \otimes \mathrm{diag}(z)$. It carries the argument because it reduces adaptation to a per-direction gain knob for each existing singular component, which is both extremely parameter-efficient and naturally compositional: expert vectors can be linearly mixed without re-training. The second piece of machinery is the two-pass dispatch, which turns this compositionality into self-adaptation by first classifying or searching for the right mixture of experts on held-out few-shot prompts and then generating the answer with the adapted model.

What would settle it

Give SVF a synthetic task whose correct solution requires a weight matrix that is not expressible as a rescaling of the base matrix's singular values, for example a rank-1 rotation into a new direction, and check whether any $z$-vector can recover the target behavior; if the best achievable accuracy stays at the base model's level, the fixed-subspace assumption is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the right parameter space for efficient LLM adaptation is the spectrum of each weight matrix, not a low-rank additive update. For every weight matrix $W = U \Sigma V^\top$, SVF learns a vector $z$ and replaces $\Sigma$ with $\Sigma \otimes \mathrm{diag}(z)$, leaving $U$ and $V$ frozen; the full-rank structure is preserved because only the singular values are rescaled. Trained end-to-end with a KL-regularized REINFORCE objective, these $z$-vectors become compact expert vectors, each specializing in a task such as math, code, or reasoning. At inference, Transformer-Squared runs the model twice: a dispatch step (prompt-based, classifier-expert, or few-shot CEM search) selects or interpolates a combination of experts, and the second pass answers the prompt with the adapted weights. The paper reports that this beats LoRA and other PEFT baselines on the training tasks, transfers to unseen tasks in both language and vision-language settings, and even transfers expert vectors across different LLM architectures when the singular-vector ordering is preserved.

Load-bearing premise

The load-bearing premise is that the fixed singular subspaces $U$ and $V$ of each pre-trained weight matrix already contain every direction a new task could need, so changing only the singular values is enough to elicit the right behavior.

Editorial extensions

If this is right

  • SVF makes it possible to specialize an LLM with only hundreds of labeled examples, because it optimizes task reward directly with RL rather than requiring solution text for next-token prediction.
  • The three dispatch strategies form a ladder: prompt-based, classifier-expert, and few-shot CEM mixing, with the paper reporting monotonic gains as more test-time information is used.
  • Because expert vectors are additive combinations of singular-value scalings, they can be stored and composed at inference, opening a route to modular LLMs that gain skills without retraining the base model.
  • Cross-model transfer of expert vectors, which the paper demonstrates from Llama-3-8B to Mistral-7B, implies that the effective skills learned by SVF may be reusable across architectures as long as singular vectors stay in a compatible order.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the boundary of its central assumption untested: if a task demands a weight-space direction that is not present in the frozen $U$ and $V$ subspaces, rescaling singular values cannot express the required change, so SVF's advantage should disappear on such tasks.
  • The surprising cross-model transfer suggests singular-vector ordering may encode a canonical, architecture-independent structure; a direct test would be to transfer experts between models of different scales or families and see whether ordered transfer still holds.
  • One implication for continual learning is that new experts can be added to the vector bank without touching old ones, and the CEM mixing coefficients could themselves be adapted online as the task distribution shifts; the paper does not demonstrate this, but the machinery is in place.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Transformer-Squared, a self-adaptation framework for LLMs. The core building block is Singular Value Fine-tuning (SVF), which fine-tunes only a per-weight-matrix scaling vector z that multiplies the singular values of each frozen weight matrix (W' = U Σ' V^T with Σ' = Σ ⊗ diag(z)). SVF vectors are trained with REINFORCE plus a KL penalty on the GSM8K, MBPP-Pro, ARC-Easy, and TextVQA tasks. At inference, Transformer-Squared runs a two-pass procedure: the first pass identifies task properties via one of three strategies (prompt engineering, a classification expert, or few-shot CEM-based interpolation of expert vectors), and the second pass generates the answer using the adapted weights. The method is evaluated on Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3-70B-Instruct, and a Llama-3-LLaVA-Next-8B VLM, on unseen tasks MATH, HumanEval, ARC-Challenge, and OKVQA. The paper claims that SVF consistently outperforms LoRA with orders of magnitude fewer parameters, and that Transformer-Squared improves over the base model on unseen tasks.

Significance. The core idea is interesting and timely: scaling only the singular values of pre-trained weights is a highly compact, compositional PEFT parameterization, and the two-pass self-adaptation mechanism is a principled way to combine expert vectors at inference. The paper ships its full source code, which is a concrete strength. The cross-model transfer result (Table 5) is surprising and, if reproducible, would be a valuable finding. However, the empirical evidence as presented is not yet sufficient to support the central 'consistently outperforms LoRA' claim: all numbers are single-run point estimates without error bars, several cells in the paper's own tables contradict the abstract's wording, and the few-shot adaptation protocol uses held-out test examples to fit the interpolation coefficients, making the 'unseen task' claim transductive rather than strictly zero-shot. The method's expressiveness boundary (fixed singular subspaces) is acknowledged as a limitation but not probed. With added statistical rigor and tempered claims, the paper could make a solid contribution.

major comments (5)
  1. [§4.2, Table 1; Abstract] The abstract and Section 1 state that the method 'consistently outperforms ubiquitous approaches such as LoRA', but Table 1 contradicts this: on Llama3-8B MBPP-Pro, LoRA scores 67.68 versus SVF's 66.67, and on Llama3-70B ARC-Easy, LoRA scores 88.55 versus SVF's 88.47; additionally, SVF does not improve over the base model on Llama3-70B MBPP-Pro (80.81, normalized 1.00) or ARC-Easy (88.47, normalized 0.99). The 'consistently' claim should be removed or replaced with a precise statement of where SVF wins, ties, or loses, and the comparison should be revisited once uncertainty estimates are available.
  2. [§4.2, Tables 1 and 2; Figure 5] Every performance number in Tables 1, 2, 5, and Figure 5 is a point estimate from a single run; no seeds, confidence intervals, or significance tests are reported. Several of the normalized gains are 1.01–1.04 (e.g., Llama3-8B GSM8K 79.15 vs 75.89, MATH 25.47 vs 24.54), which are small relative to the seed-to-seed variation typically observed in LLM fine-tuning. The LoRA baseline in Table 2 is described as 'using all checkpoints' and reporting the highest score, while SVF checkpoint selection is based on validation performance; this asymmetry could inflate the LoRA baseline but also makes the claimed SVF advantage hard to interpret. The authors should report mean and standard deviation over at least three seeds, or bootstrap confidence intervals over test examples, and specify the exact checkpoint-selection rule for every method.
  3. [§4.2, Table 2] The claim that the three adaptation strategies provide 'a clear monotonic trend' is not supported by the data. On Llama3-8B MATH, Prompt adaptation scores 25.22 while Cls-expert scores 25.18; on ARC-Challenge, Cls-expert (81.37) is below Prompt (81.74). More importantly, several adaptation strategies reduce performance relative to the base model: Mistral-7B MATH drops from 13.02 to 11.86 (Prompt) and 11.60 (Cls-expert), and Llama3-70B MATH drops from 40.64 to 40.44 (Prompt). The text says all strategies improve on Llama3-8B and 'at least two out of three tasks' for the other models, which is accurate, but the monotonicity claim and the sentence 'Transformer2 is able to push performance far further' in the introduction should be reconciled with these negative results.
  4. [§3.2 (C) and Appendix A.4] The few-shot adaptation strategy fits the CEM interpolation coefficients α using 10 examples held out from the target test set and then reports performance on the remaining test examples. This is a transductive evaluation protocol: the adaptation procedure has access to labeled examples from the target task's test distribution. The paper should clearly state this protocol in the main text, justify it (e.g., as a small calibration set scenario), and discuss why the LoRA baseline and the other Transformer-Squared strategies are not given the same access. As written, the 'unseen task' framing overstates the degree of generalization demonstrated by the few-shot results.
  5. [§3.1, §3.2, and §4.3 (Analysis 4, Table 5)] The expressiveness of SVF is bounded by the assumption that the frozen singular subspaces U and V already contain the directions needed for any target task; the paper does not probe this boundary, and the cross-model transfer result (Llama3-8B vectors applied to Mistral-7B) further assumes that singular-vector orderings are alignable across architectures with different widths and head counts. The paper itself notes this is surprising and only tests two similar architectures, which is appropriate, but the related claim that SVF 'technically provides more information than low-rank approaches' because it spans full rank is misleading: SVF has r scaling parameters but cannot change directions, whereas LoRA has (m+n)r' parameters that can. The two parameterizations have different expressive reach, so the parameter-count comparison in Section 3.2 should be reframed as a trade-off rather than a dominance claim.
minor comments (6)
  1. [Related Work, §2] The related-work list contains a bare '?' citation in the sentence listing LoRA extensions (after 'Cetoli, 2024'); this should be replaced with the intended reference or removed.
  2. [§3.1, Eq. (1)] The notation 'Σ' = Σ ⊗ diag(z)' is ambiguous because '⊗' conventionally denotes Kronecker product, while the intended operation appears to be elementwise multiplication of the diagonal of Σ with z; please clarify the notation or define a symbol such as '⊙'.
  3. [§4.2, Figure 5] Figure 5 reports VLM results (TextVQA and OKVQA) with no numerical values or error bars; the claim of a 39% improvement over the base model should be supported by a table with exact numbers and, ideally, repeated runs.
  4. [Throughout] The model name is written inconsistently as 'LLAMA 3' in tables and captions but 'Llama' in the text and figures; please standardize the capitalization.
  5. [Appendix C] The appendix is titled 'PCA on Llama 3 and Mistral', but the analysis actually computes the fraction of singular-value variance captured by the top r components; this is not PCA and should be relabeled as 'singular value spectrum' or similar.
  6. [§4.2, Table 3] Table 3 reports inference-time ratios using wall-clock seconds (e.g., MATH 42.64s first pass, 321.19s second pass); the text explains the asymptotic reasoning, but it would be clearer to also report the ratio in terms of generated tokens, since wall-clock time depends on hardware and batch size.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity: SVF/Transformer2 results are evaluated against external benchmarks and no prediction reduces to its fitted input by construction.

full rationale

The paper's central derivation is empirical rather than formal: SVF trains per-singular-value scaling vectors z with a REINFORCE objective (Eq. 1) on training splits of GSM8K, MBPP-Pro, and ARC-Easy, and reports accuracy on external test splits and on held-out tasks (MATH, HumanEval, ARC-Challenge, OKVQA). The reported numbers are not algebraically forced by the definition of W' = U Sigma' V^T, because the reward signal comes from task correctness, not from the parameterization itself. The few-shot adaptation strategy does fit the interpolation coefficients alpha on 10 examples held out from each target test set, and the paper transparently states this in Section 3.2(C) and Appendix A.4. This is a transductive, test-time adaptation procedure and it weakens the literal 'unseen task' wording for that one variant, but it is not a circular derivation: the remaining test examples are not the same examples used for fitting, and the prompt-based and classification-expert strategies do not use target-task labels at all. The only same-author citation (Akiba et al., 2024) appears in the conclusion as a suggested future direction and is not load-bearing. No uniqueness theorem, forced ansatz via self-citation, or renaming of a known result is invoked. Thus there is no circular step that makes the reported gains equivalent to the paper's inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method rests on standard SVD mathematics plus strong empirical assumptions about latent capabilities in pre-trained weights, the sufficiency of singular-value scaling, and cross-architecture alignment. The main free parameters are the KL coefficient and the CEM coefficients α, the latter being fitted directly to the target task data. No new physical or ontological entities are introduced.

free parameters (5)
  • KL coefficient λ in Eq. (1) = swept over {0.0, 0.1, 0.2, 0.3}, selected per task/model by validation
    Controls the KL penalty that stabilizes REINFORCE training; central to the RL objective.
  • Initial mean and variance of z = mean 0.1, variance 1e-3
    Hand-chosen initialization for the singular-value scaling vectors; affects training dynamics.
  • CEM interpolation coefficients α_k = learned via CEM on 10 held-out target-task examples; best of per-layer/per-vector and normalized/unconstrained…
    These weights directly determine the adapted model in few-shot adaptation; they are fitted to the target task evaluation set, so they are free parameters in the sense of transductive fitting.
  • Layer subset for SVF on Llama-3-70B and VLM = half of the layers
    A memory-saving design choice that trades off performance and could change results.
  • LoRA rank and alpha = rank 16, alpha 32
    Baseline hyperparameters that affect the comparison, though the paper states LoRA was tuned.
assumptions (5)
  • standard math Every weight matrix W admits an SVD W = U Σ V^T with orthonormal U and V and nonnegative singular values.
    Invoked in Section 3.1 as the foundation of SVF.
  • domain assumption Pre-trained LLM weights already contain the directions needed for downstream tasks, so scaling singular values alone is sufficient to elicit them.
    States in Section 3.2: 'the requisite capabilities for solving many downstream tasks appear to already exist within these pre-trained models'. If false, SVF cannot express the needed weight changes.
  • domain assumption Task identity can be inferred from the prompt or from the model's own hidden states during a first inference pass.
    Underpins the dispatch mechanisms (A, B, C) in Section 3.2; the paper validates this empirically through confusion matrices, but the principle is assumed.
  • domain assumption The ordering of singular vectors is consistent across different LLM architectures, so a z trained on Llama-3-8B can be applied to Mistral-7B.
    Required for the cross-model transfer results in Table 5; the paper itself notes the compatibility is 'surprising' and potentially tied to architectural similarity.
  • ad hoc to paper REINFORCE with a unitary reward ±1 and a KL penalty is a suitable proxy for task performance in fine-tuning.
    Adopted in Section 3.2 without theoretical justification; empirically motivated by the ablation study comparing against next-token prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformer-Squared: Self-adaptive LLMs." pith.science (2026). https://pith.science/paper/QWOSZJWJ

@misc{pith2026250106252,
  author       = {Pith},
  title        = {Pith review of: Transformer-Squared: Self-adaptive LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QWOSZJWJ}},
  note         = {Machine review of arXiv:2501.06252}
}
read the original abstract

Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse tasks. We introduce Transformer-Squared, a novel self-adaptation framework that adapts LLMs for unseen tasks in real-time by selectively adjusting only the singular components of their weight matrices. During inference, Transformer-Squared employs a two-pass mechanism: first, a dispatch system identifies the task properties, and then task-specific 'expert' vectors, trained using reinforcement learning, are dynamically mixed to obtain targeted behavior for the incoming prompt. Our method consistently outperforms ubiquitous approaches such as LoRA, with fewer parameters and greater efficiency. Furthermore, Transformer-Squared demonstrates versatility across different LLM architectures and modalities, including vision-language tasks. Transformer-Squared represents a significant leap forward, offering a scalable, efficient solution for enhancing the adaptability and task-specific performance of LLMs, paving the way for truly dynamic, self-organizing AI systems.

Figures

Figures reproduced from arXiv: 2501.06252 by the authors.

Figure 1
Figure 1. Overview of Transformer2 . In the training phase, we tune the scales of the singular values of the weight matrices to generate a set of “expert” vectors, each of which specializes in one type of tasks. In the inference phase, a two-pass process is adopted where the first applies the task￾specific expert and the second generates the answer. Self-adaptive large language models (LLMs) would represent a significant adva… view at source ↗
Figure 2
Figure 2. Method overview. Left) At training time, we employ SVF and RL to learn the “expert” vectors z’s that scale the singular values of the weight matrices. Right) At inference time, we propose three distinct methods to adaptively select/combine the learned expert vectors. Negligible parameters: Learning only a vector z for each weight matrix allows for very efficient fine-tuning with orders of magnitudes fewer optimized … view at source ↗
Figure 3
Figure 3. Prompt based adaptation. Self￾adaptation prompt used by Transformer2 to classify the task prompt into pre-defined cat￾egories. B) Classification expert: A direct extension of the prompt engineering approach comes from using a specialized system to handle task identification. Fol￾lowing the principles of self-adaptation, we ap￾ply SVF to fine-tune the base LLM itself to han￾dle this task. In particular, we collect a … view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: SVF learning curves. The dashed lines indicate the performance of LLAMA3-8B￾INSTRUCT on the test split of each task. SVF effectively fine-tunes to surpass the base performance. While we use the best validation score to select our checkpoint for evaluation (marked by re…
Figure 5
Figure 5. Figure 5: Results for the VLM domain. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrices. These matrices display the classification percentages, where rows represent the task classes (ground truth) and columns indicate the predicted categories. Some sam￾ples are misclassified as “Others,” which is reflected in rows where the totals do no…
Figure 7
Figure 7. Figure 7: αk learned weights. This operation leads to notable performance degradation across each task. Finally, by performing few-shot adaptation using the SVF vectors collected from both models, the performance of MISTRAL-7B-INSTRUCT-V0.3 further improves across the board. We …
Figure 8
Figure 8. Figure 8: Sample problem and answer. Math data sample used for LoRA instruction fine-tuning, text in blue is the unmasked solution. We follow community best practices for LoRA fine-tuning, applying it to query and value projection layers with learning rates around 5 × 10−5 . We …
Figure 9
Figure 9. Figure 9: gives the learning curves for LoRA training on the GSM8K task. 0 50 100 150 200 250 300 Iterations 0.55 0.60 0.65 0.70 0.75 0.80 Score Learning Curve on GSM8K with Lora and Policy gradient Train Accuracy Test Accuracy Base Model Performance [PITH_FULL_IMAGE:figures/fu…
Figure 10
Figure 10. Figure 10: PCA of LLAMA3-8B-INSTRUCT. We show the ratio of the variance captured by the top r singular components on the y-axis, and the layer indices on the x-axis. Except for the Query, Key and Value projection matrices, small r values only capture a tiny fraction of variance …
Figure 11
Figure 11. Figure 11: PCA of MISTRAL-7B-INSTRUCT-V0.3. We show the ratio of the variance captured by the top r singular components on the y-axis, and the layer indices on the x-axis. Except for the Query, Key and Value projection matrices, small r values only capture a tiny fraction of var…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Performant Unified GPU Kernels for Portable Singular Value Computation Across Hardware and Precision

    cs.DC 2025-08 conditional novelty 6.0 of 10

    A unified Julia implementation of two-stage QR SVD achieves near-cuSOLVER performance across four GPU vendors and three precisions, including firsts for Apple Metal and half precision.

  2. A quantum semantic framework for natural language processing

    cs.CL 2025-06 reject novelty 4.0 of 10

    The paper reports CHSH inequality violations from LLM interpretations of ambiguous sentences and uses them to claim that linguistic meaning is non-classical and observer-dependent.

  3. ChameleonLLM: Batch-Aware Dynamic Low-Rank Adaptation via Inference-Time Clusters

    cs.CL 2025-02 reject novelty 4.0 of 10

    ChameleonLLM generates low-rank LoRA updates from clustered batch statistics via a hypernetwork, claiming better perplexity than static LoRA, but the evidence is undercut by implausible baselines and confounded comparisons.

Reference graph

Works this paper leans on

51 extracted references · 16 canonical work pages · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187, 2024

    arXiv 2024

  3. [3]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  4. [5]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  5. [6]

    Fine-tuning llms with singular value decomposition

    Alberto Cetoli. Fine-tuning llms with singular value decomposition. Hugging Face Blog, June 2024. URL https://huggingface.co/blog/fractalego/svd-training. Accessed: 2024-07-01

    work page 2024

  6. [7]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  7. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  8. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

Show all 51 references
  1. [10]

    Brain network adaptability across task states

    Elizabeth N Davison, Kimberly J Schlesinger, Danielle S Bassett, Mary-Ellen Lynall, Michael B Miller, Scott T Grafton, and Jean M Carlson. Brain network adaptability across task states. PLoS computational biology, 11 0 (1): 0 e1004029, 2015

    2015

  2. [11]

    Improving factuality and reasoning in language models through multiagent debate

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023

    2023 arXiv

  3. [12]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 0 (120): 0 1--39, 2022

    2022

  4. [13]

    Arcee's mergekit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024

    2024 arXiv

  5. [14]

    Evolving modular fast-weight networks for control

    Faustino Gomez and J \"u rgen Schmidhuber. Evolving modular fast-weight networks for control. In International Conference on Artificial Neural Networks, pp.\ 383--389. Springer, 2005

    2005

  6. [15]

    Dai, and Quoc V

    David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkpACe1lx

    2017

  7. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  8. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  9. [18]

    A modern self-referential weight matrix that learns to modify itself

    Kazuki Irie, Imanol Schlag, R \'o bert Csord \'a s, and J \"u rgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In International Conference on Machine Learning, pp.\ 9660--9677. PMLR, 2022

    2022

  10. [19]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  11. [20]

    Self-moe: Towards compositional large language models with self-specialized experts

    Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406.12034, 2024

    2024 arXiv

  12. [21]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  13. [22]

    Eigenlo RA : Recycle trained adapters for resource efficient adaptation and inference, 2025

    Prakhar Kaushik, Ankit Vaidya, Alan Yuille, et al. Eigenlo RA : Recycle trained adapters for resource efficient adaptation and inference, 2025. URL https://openreview.net/forum?id=KxGGZag9gW

    2025

  14. [23]

    Adaptive knowledge bases in self-adaptive system design

    Verena Klös, Thomas Göthel, and Sabine Glesner. Adaptive knowledge bases in self-adaptive system design. In 2015 41st Euromicro Conference on Software Engineering and Advanced Applications, pp.\ 472--478, 2015. doi:10.1109/SEAA.2015.48

    2015 doi

  15. [24]

    Vera: Vector-based random matrix adaptation

    Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  16. [25]

    Evolving neural networks in compressed weight space

    Jan Koutnik, Faustino Gomez, and J \"u rgen Schmidhuber. Evolving neural networks in compressed weight space. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp.\ 619--626, 2010

    2010

  17. [26]

    Svft: Parameter-efficient fine-tuning with singular vectors

    Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors. arXiv preprint arXiv:2405.19597, 2024

    2024 arXiv

  18. [27]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 0 1950--1965, 2022

    1950

  19. [28]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  20. [29]

    Switch-independent task representations in frontal and parietal cortex

    Lasse S Loose, David Wisniewski, Marco Rusconi, Thomas Goschke, and John-Dylan Haynes. Switch-independent task representations in frontal and parietal cortex. Journal of Neuroscience, 37 0 (33): 0 8033--8042, 2017

    2017

  21. [30]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp.\ 3195--3204, 2019

    2019

  22. [31]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  23. [32]

    Trainable transformer in transformer

    Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, and Sanjeev Arora. Trainable transformer in transformer. arXiv preprint arXiv:2307.01189, 2023

    2023 arXiv

  24. [33]

    Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters, March 2024

    Qwen Team . Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters, March 2024. URL https://qwenlm.github.io/blog/qwen-moe/. Blog post

    2024

  25. [34]

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale

    Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale. In International conference on machine learni...

    2022

  26. [35]

    The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning, volume 133

    Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004

    2004

  27. [36]

    Learning to control fast-weight memories: An alternative to dynamic recurrent networks

    J \"u rgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4 0 (1): 0 131--139, 1992

    1992

  28. [37]

    A ‘self-referential’weight matrix

    J \"u rgen Schmidhuber. A ‘self-referential’weight matrix. In ICANN’93: Proceedings of the International Conference on Artificial Neural Networks Amsterdam, The Netherlands 13--16 September 1993 3, pp.\ 446--450. Springer, 1993

    1993

  29. [38]

    On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models

    J \"u rgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint arXiv:1511.09249, 2015

    2015 arXiv

  30. [39]

    The truth is in there: Improving reasoning in language models with layer-selective rank reduction

    Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558, 2023

    2023 arXiv

  31. [40]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 8317--8326, 2019

    2019

  32. [41]

    A hypercube-based encoding for evolving large-scale neural networks

    Kenneth O Stanley, David B D'Ambrosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial life, 15 0 (2): 0 185--212, 2009

    2009

  33. [42]

    Mixture-of-experts in the era of llms: A new odyssey

    Chen Tianlong, Cheng Yu, Chen Beidi, Zhang Minjia, and Bansal Mohit. Mixture-of-experts in the era of llms: A new odyssey. ICML 2024 presentation slides, 2024. International Conference on Machine Learning (ICML)

    2024

  34. [43]

    Milora: Harnessing minor singular components for parameter-efficient llm finetuning

    Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient llm finetuning. arXiv preprint arXiv:2406.09044, 2024

    2024 arXiv

  35. [44]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8: 0 229--256, 1992

    1992

  36. [45]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024

    2024

  37. [46]

    Proagent: building proactive cooperative agents with large language models

    Ceyao Zhang, Kaijie Yang, Siyi Hu, Zihao Wang, Guanghe Li, Yihang Sun, Cheng Zhang, Zhaowei Zhang, Anji Liu, Song-Chun Zhu, et al. Proagent: building proactive cooperative agents with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volu...

    2024

  38. [47]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  39. [48]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training

    Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from llama with continual pre-training. arXiv preprint arXiv:2406.16554, 2024

    2024 arXiv

  40. [49]

    Mindstorms in natural language-based societies of mind

    Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, R \'o bert Csord \'a s, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17066, 2023

    2023

  41. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  42. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  43. [52]

    Koyejo and S

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.