Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Continual learning for small language models can avoid catastrophic forgetting by storing past experiences as compact correction directions in hidden space and applying them at inference time through a gated temporary hidden-state update, w

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:31 UTC pith:2ANX2XWH

load-bearing objection MIITA is a coherent, budget-conscious SLM continual-learning method whose core trick—storing correction gradients as prototypes and applying them as gated hidden-state nudges—is genuinely new and backed by honest local theory; the main weakness is that the empirical claims need error bars, code, and a broader hyperparameter audit before I'd trust the headline numbers. the 4 major comments →

arxiv 2607.22556 v1 pith:2ANX2XWH submitted 2026-05-20 cs.AI

MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models

Dong Li , Yanchi Liu , Xujiang Zhao , Wei Cheng , Zhengzhang Chen , Xintao Wu , Zhong Chen , Chen Zhao
show 1 more author
Haifeng Chen
This is my paper
classification cs.AI
keywords continual learningsmall language modelsmemory-based continual learninginference-time adaptationhidden-state perturbationuncertainty-guided retrievalcorrection directionscatastrophic forgetting
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that small language models, which lack the storage and in-context reasoning of large ones, can still do continual learning under a tight memory budget if they store past supervision as functional correction directions rather than as text, examples, or semantic embeddings. Each experience is converted into the gradient of its token-level cross-entropy loss with respect to the pre-LM-head hidden state; similar directions are merged into prototypes with lightweight semantic anchors, and at inference time the model retrieves relevant prototypes via semantic and an uncertainty-based directional cue, aggregates them, and applies the result as a gated, temporary hidden-state perturbation. The paper reports that this consistently improves final performance and reduces forgetting across classification, QA, and generation benchmarks, and it supports the design with local first-order analyses of loss decrease, entropy decrease, and directional coverage. A sympathetic reader would care because it suggests a non-destructive, storage-efficient route to continual adaptation for small deployed models.

Core claim

MIITA's central claim is that supervised experiences can be retained as compact correction directions defined in the pre-LM-head hidden space: for each target token, the gradient of the cross-entropy loss with respect to the hidden state, W_LM^T(e_y - p), averaged and normalized. These directions are organized into prototypes (direction, anchors, count) under a fixed storage budget using direction-merge thresholds and value-based repair. At inference, the query key is pooled from the hidden states, and an uncertainty direction is computed from the restricted-entropy gradient; prototypes are retrieved by semantic recall plus uncertainty reranking, aggregated into a normalized weighted directi

What carries the argument

The central object is the correction direction d_m = Norm( (1/|Y_m|) * sum_{t in Y_m} W_LM^T(e_{y_t} - p_t) ), the normalized average gradient of the target-token cross-entropy losses with respect to the pre-LM-head hidden states (Eq. 2). This is stored in prototypes u_j = (d_j, A_j, n_j) with a normalized direction, a set of semantic anchors, and a merge count. The carrying mechanism at inference is the gated temporary hidden-state adaptation p'_t = softmax(W_LM(h_out_t + eta * alpha_t * d_bar_q)) with alpha_t = max(0, cos(h_q, h_out_t)), which injects the aggregated retrieved direction only when the decoding state is close to the query state. Local theory (Propositions A.1-A.3, Theorem A.4

Load-bearing premise

The load-bearing premise is that the gradient of the supervised loss with respect to the pre-LM-head hidden state, computed at training-time targets, is still a useful correction when applied to the hidden states of new queries; if that local gradient does not transfer to the decoding distribution, or if memory repair removes directions needed for old tasks, the claimed gains collapse.

What would settle it

A reader could measure, on a held-out set, the cosine alignment between the aggregated retrieved direction d_bar_q and the oracle direction W_LM^T(e_y - p). If average alignment is near the random baseline (about 0) rather than the paper's reported OA around 0.5, the transfer claim fails. Alternatively, replace the stored correction directions with random unit vectors of the same norm while keeping retrieval and gating identical: if OP and BWT remain unchanged, the direction content is not doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Under a fixed 1% memory budget, MIITA reports the best final performance and least backward transfer across four benchmarks spanning classification, QA, and generation, for base models from 0.6B to 8B.
  • Because the backbone stays frozen, memory reuse is non-destructive: no parameter update, prompt extension, or test-time backpropagation is needed, so the method is compatible with low-storage, low-latency SLM deployment.
  • Uncertainty-guided retrieval is shown to be a useful label-free proxy: combining semantic recall with uncertainty-based reranking outperforms either signal alone, and its oracle alignment is substantially higher than semantic-only retrieval.
  • The theoretical directional-coverage bound implies that the benefit of memory is governed by how well stored directions cover historical correction directions, not by how many raw examples are kept, which justifies budget-aware repair that prioritizes diverse, well-supported directions.
  • The activation gate is necessary: uniformly applying the retrieved direction to all decoding steps lowers performance and worsens forgetting, confirming that the query-state-conditional gate is an active component.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension not claimed by the paper: the same direction-memory idea could plausibly be adapted to unsupervised or self-supervised continual settings by replacing the label-derived correction with a proxy such as consistency gradients or contrastive directions; whether that transfer works is an open question.
  • The weakly correlated semantic and functional similarity (Pearson ~ 0.18) suggests that any memory-compression scheme that clusters purely by semantics risks discarding useful corrections; a direct test would be to run semantic-prototype memory on a different backbone or task family and see if the gap persists.
  • Since the temporary hidden update is a rank-one perturbation, MIITA might be combined with other inference-time mechanisms (e.g., retrieval of multiple directional prototypes) or with prompt-based methods, but the paper does not explore such combinations; that is our extrapolation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MIITA, a memory-based continual learning framework for small language models under a fixed persistent-storage budget. Instead of storing raw examples or textual summaries, MIITA stores normalized correction directions—negative gradients of the token-level cross-entropy loss with respect to the pre-LM-head hidden state—together with lightweight semantic anchors used only for retrieval. At inference, prototypes are retrieved by semantic similarity and an uncertainty-based directional proxy, aggregated into a single correction direction, and applied as a gated temporary perturbation to decoding hidden states. The authors provide local theoretical justifications (Propositions A.1–A.3, Theorem A.4) and report experiments on Banking77, DSC, PAXQA, and SuperNI with Qwen3-0.6B/4B and LLaMA-3.x models, showing consistent OP and BWT improvements over a range of baselines.

Significance. If the empirical results are reliable, the paper makes a worthwhile contribution by shifting memory design from textual content to functional correction signals, a novel angle for small language models. The theoretical statements in Appendix A are self-contained and correctly derived (first-order optimality, entropy gradient, least-squares aggregation). The experimental work is extensive, covering four benchmarks, two model families, multiple scales, and memory-budget sensitivity, plus ablations of memory format and retrieval cues. However, the central claim that stored corrections transfer to held-out queries is not established by the supplied theory: Proposition A.1 is per-example, and Theorem A.4 is conditional on positive alignment between the retrieved direction and the query-specific correction. The empirical evidence also lacks error bars or significance tests, which is a serious omission given the moderate effect sizes. The design is plausible and interesting, but the current evidence does not yet support acceptance.

major comments (4)
  1. [§3.2.4, Appendix A (Prop. A.1, Thm. A.4)] The central transfer claim is not supported by the theoretical results. Proposition A.1 establishes only that the stored direction decreases the loss of the exact example from which it was computed. Theorem A.4 bounds old-stage risk under the assumption that ⟨b_z, ¯d_z⟩ is positive, i.e., that the retrieved direction aligns with the query-specific correction. The retrieval mechanism (Eqs. 8–9) is based on semantic similarity and an entropy-reduction proxy; neither guarantees alignment with the loss gradient. In particular, Eq. (9) rewards directions that reduce restricted entropy, but a confidently wrong prediction has low entropy, so the proxy can select directions that do not increase the probability of the correct token. The OA results in Table 2 are suggestive but diagnostic, not a proof. Please either add a transfer theorem with explicit assumptions under which retrieval yields posi
  2. [§4.1, Table 1] The paper states 'Results average three independent runs' but reports no measure of dispersion anywhere. The OP differences between MIITA and the next-best baseline are about 2–5 points (e.g., Banking77: 77.67 vs 73.12; SuperNI: 40.58 vs 37.89). Without standard deviations, confidence intervals, or per-run values, the claim that MIITA is 'consistently' superior cannot be quantitatively assessed. Because the main conclusion rests on these comparisons, please report standard deviations (or CIs) for the key results and, if possible, a paired test or the range over the three runs.
  3. [§4.1, Appendix C.1] The memory-budget accounting needs to be explicit. Cmem is defined as the byte size of 1% of the full training stream, but it is unclear what MIITA stores and how the budget is measured: are the direction vectors (H floats), anchors (H floats), and counts all counted? For baselines, are replay buffers, LoRA adapter parameters, and summaries counted with the same byte-based accounting? Please provide a per-method byte-count formula or a table listing the stored components and their sizes, so the 'fixed memory budget' comparisons are verifiable.
  4. [§4.6, Table 2] The Oracle Alignment (OA) values are the main quantitative evidence for the uncertainty-guided retrieval design, but no variability or statistical significance is reported. The gap between SemanticRet (OA=0.35) and SemUncRet (OA=0.52) is large, but with only three runs the reader cannot assess stability. Also, the statement that 'an OA around 0.5 already indicates a strong positive projection' depends on the effective dimension of the hidden space and should be justified or removed. Please report OA standard deviations and, if possible, confidence intervals.
minor comments (5)
  1. [§4.1] 'Qwen3 (0.6B, 1,7B, 4B)' should read '1.7B' (comma instead of period).
  2. [§4.1, Implementation Details] 'using T = 0.2' should be 'Tagg = 0.2' to avoid confusion with the number of tasks T.
  3. [§4.4, Figure 3] The caption mentions '∞ denotes the unlimited-memory setting,' but the x-axis in the figure appears to stop at 10%; please clarify how the unlimited setting is represented.
  4. [Appendix C.3] In the definition of the oracle direction d⋆_q, the predicted distribution p is used; please state explicitly whether p is computed under the frozen backbone before any adaptation.
  5. [§5] The phrase 'Extensive experiments...' is an overstatement given the absence of statistical rigor; consider tempering the wording, e.g., 'across multiple benchmarks and settings.'

Circularity Check

0 steps flagged

No significant circularity: MIITA's empirical claim rests on external benchmarks; its theoretical propositions are explicit local identities, not disguised predictions.

full rationale

The central mechanism of MIITA—storing negative cross-entropy gradients in the LM-head input space and applying a gated additive perturbation at inference—is evaluated against external benchmarks (Banking77, DSC, PAXQA, SuperNI) and non-circular baseline methods. The local propositions (A.1–A.3, Theorem A.4) are derivations from the definitions of the loss, entropy, and the stored direction; they do not assume the conclusions they are used to illustrate. Proposition A.1 states that the stored normalized gradient is the steepest first-order descent direction for the same example, which is a mathematical identity rather than an empirical prediction, and the paper does not claim this proves transfer to new queries. Theorem A.4 is explicitly conditional on directional coverage, namely positive alignment ⟨b_z, ¯d_z⟩, and the paper's Limitations section states that the analysis is local and does not guarantee global improvements. The uncertainty proxy is explicitly labeled as a proxy rather than equated with correctness. No fitted parameter is renamed as a prediction: the key hyperparameters are fixed, ablated, and subjected to sensitivity analysis. The only potentially self-authored citation (CMT [13]) is peripheral and not load-bearing. No exhibited circular reduction exists, so the paper's empirical claim is self-contained and externally evaluated.

Axiom & Free-Parameter Ledger

9 free parameters · 5 axioms · 0 invented entities

The central claim rests on a set of hyperparameters committed by the authors and on local-smoothness/transferability assumptions about hidden-space gradients. No new physical entity or external constant is introduced. The theoretical result does not establish that the greedy memory construction maintains directional coverage, and the finite adaptation scale is outside the strict first-order regime analyzed.

free parameters (9)
  • tau_d (direction-merge threshold) = 0.85
    Controls when a new direction becomes a new prototype versus being merged; chosen once and used across benchmarks.
  • tau_k (semantic-anchor novelty threshold) = 0.80
    Controls when a semantic key is added as a new anchor; chosen once.
  • Rk, Rd (candidate retrieval counts) = 8, 8
    Number of semantic and directional candidates retrieved at inference.
  • M (number of aggregated prototypes) = 4
    Top-M prototypes after reranking are aggregated into the query correction direction.
  • lambda_k (semantic weight) = 1.0
    Weight for semantic relevance in the retrieval score.
  • lambda_u (uncertainty weight) = 0.5
    Weight for the uncertainty utility term in the retrieval score.
  • Tagg (aggregation temperature) = 0.2
    Softmax temperature for computing prototype weights in the aggregated direction.
  • eta (adaptation scale) = 0.5
    Scale of the temporary hidden-state update during generation.
  • epsilon_r / probe scale epsilon = not reported
    Stability threshold and probe scale in Eqs. (7) and (9) are not specified, an additional unstated choice.
axioms (5)
  • domain assumption Token loss is beta-smooth in the pre-LM-head hidden space
    Invoked in Theorem A.4 to bound old-stage risk; gives a local guarantee only.
  • ad hoc to paper First-order Taylor expansions remain valid at eta=0.5 across decoding steps
    Propositions A.1-A.3 are o(eta) local statements, but the deployed update is finite and applied at every decoding step; no global error bound is supplied.
  • domain assumption The LM-head gradient W_LM^T(e_y - p_t) fully captures the functional correction of an experience
    Eq. (2) assumes the supervised correction signal is completely represented by this gradient.
  • domain assumption Semantic anchors and restricted-entropy gradients are reliable query-side retrieval cues
    Eqs. (1), (6)-(7); if semantic representations are weak or predictive uncertainty is poorly calibrated, retrieval quality degrades.
  • ad hoc to paper Greedy prototype merging and eviction preserve directional coverage
    Eqs. (3)-(5) are heuristic; Theorem A.4 is conditioned on the retrieved direction and does not prove the greedy memory manager maintains the needed coverage.

pith-pipeline@v1.3.0-alltime-deepseek · 17589 in / 14157 out tokens · 163394 ms · 2026-08-02T13:31:40.498326+00:00 · methodology

0 comments
read the original abstract

Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments. However, directly updating their limited parameter space causes catastrophic forgetting. While memory-based methods naturally address this by decoupling knowledge retention from parameters, existing approaches designed for large language models (LLMs) rely on abundant storage and strong in-context reasoning that SLMs lack. To address these challenges, we propose MIITA, a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage. MIITA stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues. The retrieved directions are applied through gated temporary hidden-state adaptation, enabling non-destructive reuse of past supervision without backbone updates, prompt extensions, or test-time backpropagation. A local theoretical analysis links this design to first-order loss reduction, uncertainty-guided retrieval, and directional coverage for retaining old-stage knowledge. Extensive experiments across diverse supervised CL settings show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.

Figures

Figures reproduced from arXiv: 2607.22556 by Chen Zhao, Dong Li, Haifeng Chen, Wei Cheng, Xintao Wu, Xujiang Zhao, Yanchi Liu, Zhengzhang Chen, Zhong Chen.

Figure 1
Figure 1. Figure 1: Model-size scaling of CIS [18] and FOR￾EVER [6], two LLM-oriented memory-based CL baselines, on SuperNI [27] benchmark. The “-B” suffix indicates variants restricted to a fixed memory budget, contrasting with their unconstrained counterparts. We evaluate Overall Perfor￾mance (OP) and Backward Transfer (BWT); higher values indicate better accuracy and less forgetting, respectively. Overcoming this dual bott… view at source ↗
Figure 2
Figure 2. Figure 2: Model-size scaling on SuperNI under 1% memory budget. To investigate whether ex￾isting methods remain ef￾fective as the base model becomes smaller under the same memory budget, we conduct model scaling ex￾periments, as shown in Fig￾ure 2. The results reveal a severe degradation in mem￾ory methods designed for LLMs. Specifically, replay￾based methods (CT0, FOR￾EVER) drop substantially, highlighting their re… view at source ↗
Figure 3
Figure 3. Figure 3: Budget sensitivity on SuperNI with Qwen3-0.6B for memory-based methods. Each column denotes a fixed auxiliary memory budget; ∞ denotes the unlimited-memory setting. A key question in MIITA is whether supervised experiences should be stored as textual or semantic content, or as functional correction directions that encode their effect on the model prediction. Memory variant comparison. To first iso￾late the… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of why MIITA stores functional correction [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 6 linked inside Pith

  1. [1]

    Casanueva, T

    I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić. Efficient intent detection with dual sentence encoders. In T.-H. Wen, A. Celikyilmaz, Z. Yu, A. Papangelis, M. Eric, A. Kumar, I. Casanueva, and R. Shah, editors,Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45, Online, July 2020. Association f...

    2020

  2. [2]

    E. Chen, W. Fang, S. Wang, and C. Brinton. Joint continual learning of local language models and cloud offloading decisions with budget constraints.arXiv preprint arXiv:2602.00166, 2026

    arXiv 2026

  3. [3]

    P. Das, S. Chaudhury, E. Nelson, I. Melnyk, S. Swaminathan, S. Dai, A. Lozano, G. Kollias, V. Chenthamarakshan, S. Dan, et al. Larimar: Large language models with episodic memory control.arXiv preprint arXiv:2403.11901, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    de Masson D’Autume, S

    C. de Masson D’Autume, S. Ruder, L. Kong, and D. Yogatama. Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019

    2019

  5. [5]

    Diera, L

    A. Diera, L. Galke, F. Karl, and A. Scherp. Efficient continual learning for small language models with a discrete key-value bottleneck. InProceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025), pages 162–176, 2025

    2025

  6. [6]

    Y. Feng, H. Wang, J. Li, X. Chu, Z. Kang, Y. Liu, Y. Wang, P. S. Yu, and X.-M. Wu. Forever: Forgetting curve-inspired memory replay for language model continual learning.arXiv preprint arXiv:2601.03938, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    B. J. Gutiérrez, Y. Shu, Y. Gu, M. Yasunaga, and Y. Su. Hipporag: Neurobiologically inspired long-term memory for large language models.Advances in neural information processing systems, 37:59532–59569, 2024

    2024

  8. [8]

    B. J. Gutiérrez, Y. Shu, W. Qi, S. Zhou, and Y. Su. From rag to memory: Non-parametric continual learning for large language models.arXiv preprint arXiv:2502.14802, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    2022

  10. [10]

    Katraouras and D

    T. Katraouras and D. Rafailidis. Memory bank compression for continual adaptation of large language models.arXiv preprint arXiv:2601.00756, 2026

    arXiv 2026

  11. [11]

    Z. Ke, B. Liu, N. Ma, H. Xu, and L. Shu. Achieving forgetting prevention and knowledge transfer in continual learning.Advances in Neural Information Processing Systems, 34:22443–22456, 2021

    2021

  12. [12]

    Li and C

    B. Li and C. Callison-Burch. Paxqa: Generating cross-lingual question answering examples at training scale. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 439–454, 2023

    2023

  13. [13]

    D. Li, Z. Sun, X. Hu, B. Hu, and M. Zhang. Cmt: A memory compression method for continual knowledge learning of large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24413–24421, 2025

    2025

  14. [14]

    Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu. Small language models: Survey, measurements, and insights.arXiv preprint arXiv:2409.15790, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Y. Luo, Z. Yang, F. Meng, Y. Li, J. Zhou, and Y. Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  16. [16]

    J. Mok, J. Do, S. Lee, T. Taghavi, S. Yu, and S. Yoon. Large-scale lifelong learning of in-context instructions and how to tackle it. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12573–12589, 2023. 12

    2023

  17. [17]

    Momeni, S

    S. Momeni, S. Mazumder, Z. Ke, and B. Liu. In-context continual learning assisted by an external continual learner. InProceedings of the 31st International Conference on Computational Linguistics, pages 7292–7306, 2025

    2025

  18. [18]

    J. Qiu, Z. Ke, and B. Liu. Continual learning using only large language model prompting. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6014–6023, 2025

    2025

  19. [19]

    Scialom, T

    T. Scialom, T. Chakrabarty, and S. Muresan. Fine-tuned language models are continual learners. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107–6122, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics

    2022

  20. [20]

    H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, Z. Wang, S. Ebrahimi, and H. Wang. Continual learning of large language models: A comprehensive survey.ACM Computing Surveys, 58(5):1–42, 2025

    2025

  21. [21]

    Somani, V

    M. Somani, V. R. Padala, P. Upadhyay, A. Das, et al. Continual-learning for modelling low- resource languages from large language models. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6258–6275, 2026

    2026

  22. [22]

    J. Tack, J. Kim, E. Mitchell, J. Shin, Y. W. Teh, and J. R. Schwarz. Online adaptation of language models with a memory of amortized contexts.Advances in Neural Information Processing Systems, 37:130109–130135, 2024

    2024

  23. [23]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Van Nguyen, X

    C. Van Nguyen, X. Shen, R. Aponte, Y. Xia, S. Basu, Z. Hu, J. Chen, M. Parmar, S. Kunapuli, J. Barrow, et al. A survey on small language models. InProceedings of the 15th International Conference on Recent Advances in Natural Language Processing-Natural Language Processing in the Generative AI Era, pages 807–821, 2025

    2025

  25. [25]

    Von Oswald, E

    J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. InInternational Conference on Machine Learning, pages 35151–35174. PMLR, 2023

    2023

  26. [26]

    F. Wang, Z. Zhang, X. Zhang, Z. Wu, T. Mo, Q. Lu, W. Wang, R. Li, J. Xu, X. Tang, et al. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness.ACM Transactions on Intelligent Systems and Technology, 16(6):1–87, 2025

    2025

  27. [27]

    Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. InProceedings of the 2022 conference on empirical methods in natural language processing, pages 5085–5109, 2022

    2022

  28. [28]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 13 A Theoretical Analysis We provide a local theoretical analysis of MIITA to justify the main components used in Section 3.2: correction-direction extraction, uncertainty-guided retrieval, temporary hid...

    Pith/arXiv arXiv 2025