Pith. sign in

REVIEW 3 major objections 4 minor 26 references

Transfer in continual learning is a data-selection problem, not a parameter-reuse problem.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:52 UTC pith:EEHSEZIT

load-bearing objection A genuinely thought-provoking reframing of replay as a transfer mechanism, but the routing signature is validated on the same benchmarks it is tested on, and the headline stream margins sit within one standard deviation. the 3 major comments →

arxiv 2607.15587 v1 pith:EEHSEZIT submitted 2026-07-17 cs.LG

Rethinking Transfer in Continual Learning: A Replay-Based Realisation

Yang Meng , Zhenya Liu , Zhuokai Zhao , Yuxin Chen This is my paper
classification cs.LG
keywords continual learningforward transferreplayknowledge distillationcatastrophic forgettingtask signaturelow-budget learninglanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper sets out to explain when, and by what means, past tasks can help a continually learning language model learn a new task. It argues that three measurable conditions decide whether forward transfer exists at all: the new task must have headroom beyond its own small supervision, the carrier of transfer must persist through optimisation, and the replayed past data must come from a compatible source. The paper instantiates these conditions as Transfer-Selective Replay (TSR), which routes replay to the past task whose zero-training gradient signature aligns best with the incoming task, and assigns stability to distillation. If the central claim is right, replay's conventional role is inverted — replay is a transfer mechanism, not mainly a forgetting remedy — and the field's focus should shift from merging parameters to selecting data. The reported result is that under the standard continual-learning protocol with 50 target examples per task, TSR improves forward transfer while maintaining stability, beating uniform replay on overall accuracy in all four main settings.

Core claim

The paper's central claim has two parts. First, transfer in continual learning should be understood through three checkable conditions: headroom (the target task has room to improve beyond its own limited data), persistence (the transferred signal must be re-injected at every optimisation step rather than applied once), and compatibility (the source task's data must align with what the new task is trying to learn). Second, when these conditions hold, the natural carrier is replayed past data, and the practical question is which past task's data to replay. The proposed method, Transfer-Selective Replay, answers that question with a training-free task signature — the mean gradient of probe bat

What carries the argument

The central object is the task memory record (g_j, M_j, θ_j), one per past task, containing the task signature, a small data subsample, and the adapter snapshot from that task's end. The task signature g_t is the mean gradient over m=10 probe batches computed at the fixed shared initialisation, flattened and normalised; sources are scored by cosine similarity between the new task's signature and each stored signature, with softmax routing at temperature τ selecting which past task's data to mix into every training batch. Replay data is the persistent carrier: because it is re-presented at every step, its signal survives continued optimisation, whereas one-shot parameter reuse is washed out.

Load-bearing premise

The load-bearing assumption is that the gradient task signature taken at the shared initialisation reliably predicts which past task's data will help the current task; the paper's own evidence for this is limited to in-sample validation with a top-1 hit rate of four out of eight targets.

What would settle it

Run TSR on a stream of task types not used when the signature was designed; if the stored signature's best-source choices match a freshly measured pairwise transfer matrix no better than chance, or if signature-routed replay fails to beat uniform replay on a held-out heterogeneous stream, the central claim that transfer is a selection problem loses its support.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Replay buffers should be selected for what they can teach the incoming task, not only for what they preserve; selection replaces curriculum design as a way to control transfer.
  • Distillation in low-budget continual LLM fine-tuning should be treated as a stability regulariser; attempts to make it carry transfer reduce both transfer and stability.
  • Continual-learning benchmarks should report headroom and a measured pairwise transfer matrix as part of their specification, because without headroom no mechanism can show transfer.
  • Within task families that are highly homogeneous, pooling replay from all past tasks can beat single-source selection; heterogeneous streams favour sharp routing.
  • At larger model scales the transfer opportunity shrinks, so gains concentrate in low-budget, small-model regimes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: If transfer is a data-selection problem, then privacy-constrained deployments that cannot store raw examples should be able to replace stored data with synthetic or distilled replay judged by the same persistence criterion; the paper names this as a question, and it is a directly testable extension.
  • Inference: The signature's predictive power was validated on the same benchmark family used to choose it; a stronger test would hold out whole task types and check whether stored gradient signatures still pick the measured best source, and whether signature-based routing beats uniform replay on a stream of genuinely unseen tasks.
  • Inference: The three-condition decomposition suggests a design rule for transfer-aware continual learning systems: audit headroom before choosing a mechanism, choose persistent carriers, and spend the complexity budget on source selection rather than parameter merging or curriculum ordering.
  • Inference: Because signatures are scale-stable in the paper's measurements, a small proxy model could compute signatures cheaply for a larger deployed model; that cost-saving implication is the author's, but the cross-scale stability makes it plausible enough to test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies forward transfer in low-budget continual learning of LLMs. It proposes three conditions for realized transfer: headroom of the target task, persistence of the carrier, and compatibility of the source task. It instantiates these as Transfer-Selective Replay (TSR): a zero-training gradient signature of each task at a shared initialization is used to route replayed data from past tasks to the current task, while distillation from era snapshots is used for stability. The method is evaluated on TRACE-8 and NumGLUE-8 streams at 0.5B/7B with several backbones and compared against rehearsal-free and rehearsal-based baselines under the standard sequential protocol. The paper reports that TSR improves Overall accuracy and plasticity while maintaining positive backward transfer, and concludes that transfer in continual learning is a data-selection problem rather than a parameter-reuse problem.

Significance. If the framework holds, it provides a measurable decomposition of when transfer should occur and repositions replay as a forward-transfer mechanism; the pairwise transfer matrix and headroom audits are useful diagnostic tools. The manuscript has real strengths: a hindsight-oracle control, local interventions at the shared initialization, re-measurement that exposed an earlier merge artifact, fully specified hyperparameters, released code, and evaluation on additional backbones and benchmarks. However, the central empirical claim is currently supported only by in-sample signature selection and by stream-level margins that are within sampling noise. The conceptual contribution is significant, but the evidence for the distinctive 'whom' mechanism needs strengthening before the headline conclusion can be accepted.

major comments (3)
  1. [§4.3.2 / Table 2; §6.2 / Table 3] The 'whom' condition rests entirely on the gradient signature (Eq. 4), yet its validation is in-sample. The signature family was selected by top-1 hit rate against the measured TRACE-8 transfer matrix (Table 2), and the same eight tasks form the stream in Table 3. For NumGLUE-8, the matrix in Appendix A.4.2 is measured on the same task types before the stream. The 4/8 top-1 rate is therefore an in-sample estimate, and §7.2's limitation list does not acknowledge this selection-on-the-evaluation-set risk. A leave-one-task-out or held-out transfer-matrix prediction is needed before 'routing by signature' can be claimed as the mechanism behind TSR's gains.
  2. [§6.2 / Table 3] The headline margins over ER are not statistically supported. In all four main settings the Overall difference is within one standard deviation of the 5-seed means, e.g. TRACE-0.5B .475±.030 vs .451±.031; NumGLUE-7B .588±.017 vs .578±.012. No significance test or confidence interval is reported. The same applies to the BWT comparison, where TSR's positive values are often within noise of ER's (e.g. +.035±.015 vs +.025±.020). Paired tests across seeds, bootstrap intervals, or more seeds are needed to support 'consistently outperforms' and 'best everywhere'.
  3. [§4.3.1 / Table 1; §5.2 / Algorithm 2] The transfer matrix that defines source quality is measured by 50/50 joint training (source and target data mixed in the same batch), whereas TSR during the stream alternates between target and replay batches (α=0.5) and adds a KL distillation term. The paper does not show that the matrix's best source is the best replay source under this different optimization. The local intervention (Table 2, right) covers one step at the shared initialization, not the long-horizon stream. Because the claim 'selection matches the hindsight-oracle ceiling' is only demonstrated in the probe (Appendix A.2), the link between the measured matrix and TSR's routing success is incomplete.
minor comments (4)
  1. [§4.3.2] The mechanism paragraph says signatures capture 'optimisation bias'; however, the signature uses only the gradient at θ0, while the actual descent during the stream is from θ_t. A sentence clarifying why θ0 gradients remain informative after adaptation (beyond the local intervention) would help.
  2. [§6.1 / Appendix C.1] TSR hyperparameters α and λ are fixed at 0.5/0.5; no sensitivity analysis is reported in the main text. Appendix B.3 varies routing, but not α or λ.
  3. [Appendix B.4.1 / Table 6] The 'random teacher per batch' row reports 'ties cosine' but the preceding sentence says 'Even the per-batch oracle adds nothing over a random teacher'; clarify whether 'ties' means statistically indistinguishable and with what uncertainty.
  4. [Abstract / §6.3] The abstract promises 'consistently improves forward transfer while maintaining stability'; in the 7B TRACE setting the paper itself reports no transfer opportunity (§6.3, Figure 5), so the wording should be qualified to settings with headroom.

Circularity Check

0 steps flagged

No circular derivation: TSR's signature is not fitted to stream outcomes and the framework's conditions are empirically measured rather than defined by the conclusion.

full rationale

The paper's claimed derivation chain does not reduce to its inputs. TSR's routing signal (Eq. 4) is a training-free mean gradient cosine computed at a fixed shared initialization; no parameter of the signature or of the routing rule is fitted to the final stream metrics in Table 3. The transfer matrix (Table 1) is measured by independent pairwise joint-training runs and is used only to score candidate signatures (Table 2); the sequential stream evaluation accumulates one adapter and never feeds the matrix into Algorithm 2. The headroom audit, carrier persistence dissociation, and distillation teacher audit are empirical dissociations, not definitions in disguise: e.g., the persistent-parameter L2 anchor is measured to capture about two-thirds of the data-mixing gain, which is a substantive result rather than a tautology. The main methodological weakness is that the gradient-signature family was selected by top-1 accuracy on the same TRACE-8 transfer matrix whose tasks later appear in the TRACE-8 stream; that is an in-sample validation/benchmark-leakage concern, not a circularity by construction, since the stream metric is not equal to the selection criterion and the method is also evaluated on NumGLUE-8, SuperNI, PP15, and additional backbones. Self-citations (e.g., Meng et al., 2026; Li et al., 2026) are illustrative and non-load-bearing. No equation in the paper is equivalent to its own prediction target, and no fitted parameter is renamed as a prediction.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The central claim rests on an analytic decomposition (Eq. 2), an empirical persistence premise, a predictive signature whose validation is in-sample, the low-budget scope, and the standard single-adapter task-incremental setup. No new physical/conceptual entities with independent external handles are introduced; the task signature is a methodological construct validated within the paper. Hyperparameters (tau, alpha, lambda, m, N, buffer size) are chosen by hand, with partial sensitivity analysis.

free parameters (6)
  • routing temperature tau = 0.1
    Softmax temperature over signature cosines (Algorithm 2); tau->0 is argmax, tau->inf is ER. Value stated in C.1 without a tuning protocol; the method's behavior interpolates between selection and pooling.
  • replay ratio alpha = 0.5
    Probability of sampling a replay batch vs a target batch (Algorithm 2). Chosen by hand; no sensitivity scan reported for alpha in the stream.
  • distillation weight lambda = 0.5 (KL temperature 2)
    Stability term weight in Eq. (5). B.4.3 shows a flat curve over lambda in [0.1,2] on PP15, but the stream value is fixed without a tuning protocol.
  • probe batch count m = 10
    Number of forward-backward passes for the task signature (Eq. 4); robustness is mentioned in A.2 but no sweep is reported.
  • target-data budget N / reference Nmax = N=50, Nmax=500
    Low-budget regime definition; headroom Eq. (3) depends on Nmax=10N chosen by hand.
  • replay buffer size per task = 100 examples
    Subsample stored per task in the task memory (§6.1); chosen by hand.
axioms (6)
  • domain assumption Transfer decomposition: transfer(j->t)>0 requires headroom(t) AND a persistent carrier AND source j (Eq. 2)
    Presented as 'a simple decomposition' organizing the paper; it is an analytic/empirical claim, not proved. All downstream conditions and the method rely on this tripartition.
  • domain assumption Replay data is the cheapest persistent carrier; continual optimisation preserves persistent signals and suppresses transient ones
    §4.2–§5: 'the premise from which §5 derives the method.' Empirically supported by small-effect carrier studies (Appendix B.2) but treated as a law (§7.3).
  • domain assumption The mean gradient over m probe batches at the shared initialization theta0 captures task compatibility
    Eq. (4); the signature is the only tested predictor reaching 4/8 top-1 on TRACE-8 (Table 2). This is the load-bearing premise of the 'whom' condition, validated only on the same benchmark used for stream evaluation.
  • domain assumption The low-budget regime (N=50) is the setting where transfer is operationally valuable and measurable
    §4.1 shows headroom shrinks as N grows; all conclusions are scoped to this regime, including the 7B-scale decay in Figure 5.
  • domain assumption Task-incremental continual fine-tuning with a single LoRA adapter and no task identity at inference
    §3.1; standard CL formulation the paper evaluates. Excludes class-incremental and privacy-constrained settings (flagged in §7.2).
  • domain assumption Distillation (KL to era snapshot) can own the stability objective; teacher identity is second-order
    §5.3 and Appendix B.4; drawn from the PP15 audit. Used as a design premise in the stream comparison.
invented entities (1)
  • Task signature g_t (mean gradient over probe batches at theta0) no independent evidence
    purpose: Zero-training descriptor of a task's first-order learning direction; routes replay sources and picks teachers (Algorithm 2).
    No validation outside this paper's benchmarks; its predictive power is measured against the same TRACE-8/NumGLUE matrices used in the stream experiments. The local-intervention study provides internal mechanistic support but not an external falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 24854 in / 23893 out tokens · 273565 ms · 2026-08-01T22:52:44.652275+00:00 · methodology

0 comments
read the original abstract

Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch. Existing methods, whether rehearsal-based (replaying stored past data) or rehearsal-free (regularising or isolating parameters), overwhelmingly target one objective: preventing catastrophic forgetting. Forward transfer, the past helping the future, has meanwhile been pursued almost exclusively through parameter reuse, with no explicit account of when transfer should be expected at all. We begin one step earlier: before designing a transfer mechanism, we ask when transfer should exist at all. We answer with a framework of three measurable conditions: the target task must leave room for improvement beyond its own limited supervision, transferable information must survive continued optimisation, and replay must come from compatible previous tasks. We instantiate this view as Transfer-Selective Replay (TSR), which selects replay data predicted to benefit the incoming task rather than replaying past examples indiscriminately. Selection is guided by a zero-training task signature, while distillation preserves stability on previous tasks. Under the standard continual learning protocol in the low-budget regime, TSR consistently improves forward transfer while maintaining stability, outperforming existing replay baselines across heterogeneous and homogeneous task streams. More broadly, the results argue for treating transfer as a first-class objective of continual learning, to be understood before it is engineered.

Figures

Figures reproduced from arXiv: 2607.15587 by Yang Meng, Yuxin Chen, Zhenya Liu, Zhuokai Zhao.

Figure 1
Figure 1. Figure 1: A framework for continual transfer, and one instantiation. Continual learning serves two objectives. On the transfer side we organise the design space around three questions — where transfer is possible (headroom), what carries it (parameters vs. data), and whom to draw on (source selection); prior transfer mechanisms occupy the parameter branch (§2), and this work takes the data branch, turning transfer i… view at source ↗
Figure 2
Figure 2. Figure 2: Transfer opportunity shrinks with the target budget, and the gain lives where op￾portunity is large (flagship target numglue-cm, the matrix’s strongest pair; n=5 seeds per point; shading ±1 s.e.m.). (a) headroom vs. budget; (b) paired transfer gain over target-data (TSR denotes selected replay, ER uniform replay); (c) at N=10 the gain holds from the first evaluation to the last step. Per-target numbers: Ap… view at source ↗
Figure 3
Figure 3. Figure 3: Candidate carriers, ordered by the persistence of their optimization signal. The transfer gain follows persistence rather than whether the carrier is parameters or data; bars show the measured share of the data-mixing accuracy gain each carrier captures (Appendix B.2). The literature examples shown here are persistent by construction. source\target cstn fomc meet py15 sciq ngcm ngds 20mn cstance — −0.18 −0… view at source ↗
Figure 4
Figure 4. Figure 4: The Transfer-Selective Replay pipeline. 5.3 KNOWLEDGE DISTILLATION FOR STABILITY Once replay is reassigned to transfer, the stability term of the continual objective is unowned; TSR assigns it to distillation. Each training batch b is anchored by LKD(θ; b) = KL fθ ref (b) [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Transfer follows headroom across model scale (TRACE headroom audit per Qwen2.5 scale; error bands omitted for clarity). Transfer follows headroom [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 20 linked inside Pith

  1. [3]

    Efficient lifelong learning with A-GEM

    Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. InInternational Conference on Learning Representations, 2019a. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continua...

    Pith/arXiv arXiv 1902

  2. [6]

    Scaling laws for transfer

    Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293,

    Pith/arXiv arXiv

  3. [7]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    16 Preprint. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685,

    Pith/arXiv arXiv

  4. [8]

    Lorahub: Effi- cient cross-task generalization via dynamic LoRA composition.arXiv preprint arXiv:2307.13269,

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Effi- cient cross-task generalization via dynamic LoRA composition.arXiv preprint arXiv:2307.13269,

    Pith/arXiv arXiv

  5. [9]

    Continual learning of natural language processing tasks: A survey.arXiv preprint arXiv:2211.12701,

    Zixuan Ke and Bing Liu. Continual learning of natural language processing tasks: A survey.arXiv preprint arXiv:2211.12701,

    Pith/arXiv arXiv

  6. [12]

    Saleh Momeni and Bing Liu

    17 Preprint. Saleh Momeni and Bing Liu. Achieving upper bound accuracy of joint training in continual learning. arXiv preprint arXiv:2502.12388,

    Pith/arXiv arXiv

  7. [13]

    Lightspeed geometric dataset distance via sliced optimal transport.arXiv preprint arXiv:2501.18901, 2025a

    Khai Nguyen, Hai Nguyen, Tuan Pham, and Nhat Ho. Lightspeed geometric dataset distance via sliced optimal transport.arXiv preprint arXiv:2501.18901, 2025a. Thinh Nguyen, Cuong N Nguyen, Quang Pham, Binh T Nguyen, Savitha Ramasamy, Xiaoli Li, and Cuong V Nguyen. Sequence transferability and task order selection in continual learning.arXiv preprint arXiv:25...

    Pith/arXiv arXiv

  8. [14]

    Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks.arXiv preprint arXiv:1811.01088,

    Jason Phang, Thibault F´evry, and Samuel R Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks.arXiv preprint arXiv:1811.01088,

    Pith/arXiv arXiv

  9. [15]

    Merge before forget: A single LoRA continual learning via continual merging.arXiv preprint arXiv:2512.23017,

    Fuli Qiao and Mehrdad Mahdavi. Merge before forget: A single LoRA continual learning via continual merging.arXiv preprint arXiv:2512.23017,

    arXiv

  10. [17]

    Progressive neural networks.arXiv preprint arXiv:1606.04671,

    Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks.arXiv preprint arXiv:1606.04671,

    Pith/arXiv arXiv

  11. [19]

    Huashan Sun, Yizhe Yang, Yinghao Li, Jiawei Li, and Yang Gao

    arXiv:2404.12526. Huashan Sun, Yizhe Yang, Yinghao Li, Jiawei Li, and Yang Gao. Unveiling and addressing pseudo forgetting in large language models. InFindings of ACL,

    Pith/arXiv arXiv

  12. [20]

    A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a. Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model contin...

    Pith/arXiv arXiv

  13. [22]

    Direct acquisition optimization for low-budget active learning.arXiv preprint arXiv:2402.06045,

    Zhuokai Zhao, Yibo Jiang, and Yuxin Chen. Direct acquisition optimization for low-budget active learning.arXiv preprint arXiv:2402.06045,

    Pith/arXiv arXiv

  14. [23]

    19 Preprint

    arXiv:2501.13453. 19 Preprint. APPENDIX Appendix guide.Appendix A collects the benchmark-level evidence for the framework of §4: the four regimes — PP15 (no headroom), TRACE (controlled transfer measurement), SuperNI (generic transfer), and NumGLUE (homogeneous transfer). Appendix B collects the implementation evidence behind §5: carrier, replay, routing,...

    Pith/arXiv arXiv 2025

  15. [24]

    is the closest relative: it stores each buffered sample’s logits at insertion time and anchors to them on replay — a frozen, per-sample form of era anchoring. Our era snapshots generalise this in three ways: the anchor is a queryable model rather than stored values (it can answer on any batch, including current-task batches, and is taken at task end rathe...

    2024

  16. [25]

    Table 12 reports the stream under the exact configuration of Appendix C.1

    and Gemma-3-1B-it (Gemma Team, 2025), the TRACE-8 audit finds open headroom on every task (target-full minus target-data, n=2: +.045 to +.135 accuracy on Llama and +.045 to +.269 on Gemma across the five classification targets; the three generation targets improve by 1.8–4.6 in loss), so transfer is measurable on both. Table 12 reports the stream under th...

    2025

  17. [26]

    Training configuration as in Appendix C.1 (PP15 runs use max length 256)

    and Gemma-3- 1B-it (Gemma Team, 2025). Training configuration as in Appendix C.1 (PP15 runs use max length 256). Probe protocol, arm definitions, and per-cell seed counts as stated inline; full-seed statistics are reported throughout to avoid favourable-subset selection bias (Li et al., 2026). A minimal implementation reproducing the core TRACE-8 experime...

    2025

  18. [2016]

    Dynamic proxy-mixing: Transferring re- play controllers from small to large models for continual instruction tuning.arXiv preprint arXiv:2606.00400,

    Ibne Farabi Shihab, Fariya Afrin, and Anuj Sharma. Dynamic proxy-mixing: Transferring re- play controllers from small to large models for continual instruction tuning.arXiv preprint arXiv:2606.00400,

    Pith/arXiv arXiv

  19. [2017]

    MOMA: Masked orthogonal matrix alignment for zero-additional-parameter model merging.arXiv preprint arXiv:2412.13526,

    Fanshuang Kong, Richong Zhang, Zhijie Nie, et al. MOMA: Masked orthogonal matrix alignment for zero-additional-parameter model merging.arXiv preprint arXiv:2412.13526,

    arXiv

  20. [2018]

    C-LoRA: Continual low-rank adaptation for pre-trained models.arXiv preprint arXiv:2502.17920,

    Xin Zhang, Liang Bai, Xian Yang, and Jiye Liang. C-LoRA: Continual low-rank adaptation for pre-trained models.arXiv preprint arXiv:2502.17920,

    Pith/arXiv arXiv

  21. [2019]

    When AI benchmarks plateau: A systematic study of benchmark saturation.arXiv preprint arXiv:2602.16763,

    Mubashara Akhtar, Anka Reuel, Prajna Soni, et al. When AI benchmarks plateau: A systematic study of benchmark saturation.arXiv preprint arXiv:2602.16763,

    Pith/arXiv arXiv

  22. [2020]

    Glaze, Gabriel Orlanski, et al

    Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, et al. Continual learning bench: Evaluating frontier AI systems in real-world stateful environments.arXiv preprint arXiv:2606.05661,

    Pith/arXiv arXiv

  23. [2021]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  24. [2022]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen Team. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  25. [2024]

    Latent traits and cross-task transfer: Deconstructing dataset interactions in LLM fine-tuning.arXiv preprint arXiv:2509.13624,

    Shambhavi Krishna, Atharva Naik, Chaitali Agarwal, Sudharshan Govindan, Taesung Lee, and Haw-Shiuan Chang. Latent traits and cross-task transfer: Deconstructing dataset interactions in LLM fine-tuning.arXiv preprint arXiv:2509.13624,

    arXiv

  26. [2025]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv