Pith. sign in

REVIEW 5 major objections 7 minor 24 references

Return of the Encoder: Maximizing Parameter Efficiency for SLMs

T0 review · 5 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Encoder-decoder small models beat decoder-only on speed and accuracy

desk verdict A useful controlled comparison of encoder-decoder vs decoder-only SLMs, but the cross-architecture KD claims rest on an unstated tokenizer-sharing assumption that a referee must pin down. read the letter →

arxiv 2501.16273 v2 pith:UOZHUMCR submitted 2025-01-27 cs.CL cs.AIcs.CV

Mohamed Elfeki , Rui Liu , Chad Voegele This is my paper · ORCID
classification cs.CLcs.AIcs.CV
keywords encoder-decoderarchitecturesmalllanguagemodelsknowledgedistillationparameterefficiencyedgedeploymentfirst-tokenlatencythroughputcross-architecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that for language models at or below 1 billion parameters, the encoder-decoder architecture is the stronger choice, not a legacy design. Across GPU, CPU, and NPU hardware, it reports 29-47% lower first-token latency and 3.8-4.7x higher throughput than parameter-matched decoder-only models. With a new cross-architecture knowledge distillation method, the same encoder-decoder models also score up to 6 average points higher on downstream tasks in the paper's evaluations. The reason to care is that on-device and edge deployments are exactly where parameter budgets are tightest and per-token computation matters most.

What carries the argument

Two mechanisms carry the argument. The first is the architectural pairing: an encoder that processes the input once into fixed representations and a decoder that generates from those representations, with a 2/3-1/3 parameter allocation and RoPE/NTK sequence handling. The second is Algorithm 1, a knowledge-distillation scheme that aligns a decoder-only teacher's logits to the encoder-decoder student's decoder logits by padding the teacher input as [PAD]ne x y [PAD]nd and slicing teacher logits at the offset |x| + ne - 1. That alignment lets a small encoder-decoder model absorb knowledge from a large decoder-only teacher without changing its own architecture.

What would settle it

Run Algorithm 1 with a student that uses a different tokenizer from the teacher, such as a SentencePiece-based encoder-decoder with a BPE-based teacher, and check whether the reported KD gains survive; if the offset-based logit alignment cannot be implemented, the KD component is falsified.

Watch

Extended reading notes

Core claim

The central claim is that the encoder-decoder information bottleneck, long treated as a scaling handicap, acts as a useful inductive bias at small scale. The paper reports a 2/3-1/3 encoder-decoder split, with more encoder than decoder layers, built with RoPE, NTK-scaled positions, and grouped-query attention, matches or beats a parameter-matched 48-layer decoder-only model on reasoning, summarization, code translation, and creative writing tasks. It attributes the efficiency gain to one-time input processing and a fixed encoder representation that removes input KV-cache growth, and the quality gain to specialized encoder and decoder roles plus knowledge distillation from a larger decoder-only teacher. The same recipe is extended to an 800M vision-language model, where the encoder-decoder variant outperforms the decoder-only baseline by 7-11 points on VQA-style benchmarks.

Load-bearing premise

The knowledge-distillation gains assume the teacher and student share a tokenizer and vocabulary, because logits are aligned token-by-token using fixed padding offsets; a different tokenizer would break that alignment before the KL loss is computed.

Editorial extensions

If this is right

  • For sub-billion-parameter deployments, architecture should be treated as a first-class design variable alongside data quality and parameter count.
  • Encoder-decoder models become especially attractive for long-input tasks, since the reported efficiency gap grows with input length, reaching a 3.2x FLOP difference at 4096 tokens.
  • Cross-architecture knowledge distillation gives small encoder-decoder models a path to borrow decoder-only scaling benefits, weakening the case for restricting distillation to same-architecture pairs.
  • The vision-language extension suggests the same encoder-decoder core with a vision encoder is a practical template for multimodal small models.
  • If the pattern holds, reasoning models that produce long chains of thought from short prompts are a natural fit, because generation cost and fixed input memory dominate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported latency and throughput numbers come from ONNX-compiled 330M models at 512-token inputs and 128-token outputs, so the real-world gap should be re-measured at longer sequences and different batch sizes before making deployment decisions.
  • Editorial inference: if confirmed, the results suggest that sub-billion-parameter research reports should include an encoder-decoder baseline rather than treating decoder-only as the neutral default.
  • Editorial inference: the tokenizer-alignment requirement of the distillation scheme means the recipe is most directly portable to students that reuse the teacher's vocabulary; adapting it to SentencePiece-style students would require an added vocabulary-mapping step.
  • Editorial inference: the optimal encoder/decoder ratio likely tracks task asymmetry, so one could test whether a generation-heavy workload favors a smaller encoder than the 2/3-1/3 split used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper argues that for small language models (SLMs) at or below 1B parameters, encoder-decoder architectures are both more efficient and, after cross-architecture knowledge distillation from a decoder-only teacher, more accurate than parameter-matched decoder-only models. The authors report controlled comparisons of 330M, 500M, and 1B models across GPU, CPU, and NPU platforms, claiming 42-47% lower first-token latency and 3.8-4.7x higher throughput for the encoder-decoder variants (Table 2). They introduce a knowledge distillation procedure (Algorithm 1) that aligns teacher and student logits by slicing both sequences at token-level offsets, and they report downstream gains of up to 6 average points (Table 1). The paper also extends the architecture to vision-language tasks with a ViT-based encoder and reports consistent improvements over a decoder-only baseline (Figure 4).

Significance. If the empirical claims hold, the paper is a useful, timely counterpoint to the decoder-only scaling trend for sub-billion-parameter deployment. Its controlled comparison across GPU/CPU/NPU with ONNX models is a strength, and the efficiency measurements in Table 2 are concrete and reproducible in principle. The cross-architecture distillation recipe is also potentially valuable, since most current distillation pipelines assume matching architectures. However, the performance claim currently rests on small deltas without error bars, on pretraining results that are statistically indistinguishable and near chance, and on a distillation alignment assumption that is never stated. These are fixable but load-bearing issues, so the paper needs a major revision before the central claims can be accepted.

major comments (5)
  1. [Section 3.2, Algorithm 1] The token-level logit alignment in Algorithm 1 is only defined if the teacher and student share one tokenizer and vocabulary, so that the same input x and output y tokenize identically in both models. The manuscript never states this for the Phi-3.5Mini teacher and the encoder-decoder student built on the Raffel et al. (2020) design; if their tokenizers differ, the slice lt[|x|+ne-1 : ...] compared to ls[:|y|] does not correspond to aligned tokens, and the KL objective is not a well-defined token-level distillation. Since the KD gains in Table 1 are a central claim, please state the tokenizer relationship explicitly, provide the vocabulary sizes, or release code/checkpoints so the alignment can be verified.
  2. [Section 3.3, Pretraining Results and Table 1] The pretraining results are at chance level (MMLU 24.82±0.44, Arc-easy 24.92±0.51, Arc-challenge 22.95±0.36 against a 25% random baseline) and show no statistically significant difference across architectures. Combined with the absence of error bars or significance tests in Table 1, the small downstream deltas that carry the architectural claim (e.g., CodeX RL 0.93 for all variants, XSum RL 0.27 vs 0.25) could be within run-to-run noise. Please report confidence intervals or multiple-seed standard deviations for the main table and show that the KD-specific improvements are not noise.
  3. [Section 3.4, Table 2 and Abstract] The abstract's headline numbers, '47% lower first-token latency and 4.7x higher throughput... on edge devices,' match only the NPU row of Table 2. The GPU row shows 42% lower latency and 3.9x throughput, while the CPU row shows 29% and 3.8x. Please either report the full range of results in the abstract, or explicitly and consistently define which platform is being called an edge device and which comparison is being summarized.
  4. [Section 4.3, Figure 4] The vision-language comparison reports only relative percentage gains (VQAv2 +11.21%, TextVQA +8.17%, ChartQA +7.28%) with no absolute scores, no error bars, and no specification of how many examples or runs underlie Figure 4. The surrounding text also contains a verbatim duplicated paragraph. Please provide the underlying absolute metrics, evaluation-set sizes, and variance information so the multimodal claim can be assessed.
  5. [Section 3.4, Table 2] The efficiency comparison does not state the ONNX export and runtime settings used for both models: quantization, batch size, padding strategy, warmup iterations, thread counts, or whether the same graph optimizations were applied. Since first-token latency and throughput are highly sensitive to these choices, please document them in full for reproducibility.
minor comments (7)
  1. [Section 3.3] The word 'preplexity' should be 'perplexity'.
  2. [Section 4.3] The paragraph beginning 'Our evaluation framework for vision-language models maintains...' is duplicated verbatim; please remove the duplicate.
  3. [Table 3 caption] The metric is named 'RougeLSum' but the text refers to 'Rouge-L'; please unify the terminology and clarify why a Rouge-based summary metric is used for TextVQA, a visual question answering task.
  4. [Supplementary Details] The supplementary sections are labeled '5.1' and '5.2' even though Section 5 is 'Conclusion and discussion'; please renumber the appendix as A.1 and A.2.
  5. [Figure 2] The caption for Figure 2 does not define the metric on the y-axis or the tasks being scaled; please include this information so the scaling claim can be interpreted.
  6. [Algorithm 1] In line 18, it should be clarified that CE(y, ys) is the cross-entropy between the reference output y and the student's decoder logits, not the student's token ids.
  7. [Section 3.3, Implementation Details] The reported training-time comparison ('250 hours vs 350 hours', '42% of the FLOPs') is not accompanied by the hardware configuration, sequence-length distribution, or whether the timing includes evaluation; please state these details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's efficiency and KD claims are empirical and self-contained, with no fitted quantity renamed as a prediction.

full rationale

The paper makes two central claims: (1) encoder-decoder SLMs achieve lower first-token latency and higher throughput than parameter-matched decoder-only models, and (2) cross-architecture knowledge distillation from a decoder-only teacher improves encoder-decoder performance. Both claims are supported by direct measurements and controlled training runs rather than by definitional or self-referential reductions. The efficiency results in Table 2 are measured across GPU, CPU, and NPU using ONNX-compiled models of matched parameter count; the latency and throughput numbers are empirical observations, not consequences of a fitted parameter. The KD results in Table 1 compare Seq2Seq and KD training on the same architectures, with the mixing ratio α and temperature τ tuned per dataset, which is standard model selection. The choice of the 2/3-1/3 encoder-decoder split is made after comparing three explicit variants, and this selection does not make the reported downstream gains true by construction. Citations to prior work (Raffel et al., Wang et al., Agarwal et al.) motivate design choices such as pretraining objective and on-policy distillation, but the paper's reported outcomes are not derived from those citations. The tokenizer-alignment assumption in Algorithm 1 is a potential correctness gap, not a circular step: it does not render any prediction equivalent to its input. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is merely renamed. The paper is self-contained as an empirical study, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The architecture uses standard components (T5-style encoder-decoder, RoPE, GQA, ViT), and the only fitted quantities are hyperparameters and architecture splits chosen by model selection.

free parameters (3)
  • KD mixing ratio alpha = Not reported per dataset
    Section 3.2 says the mixing ratio is tuned per dataset; it controls the balance between KL and cross-entropy loss and affects all KD results.
  • Distillation temperature tau = Not reported
    Section 3.2 and Algorithm 1 use a temperature parameter to soften logits; its value is not given, and it is tuned as a hyperparameter.
  • Encoder/decoder layer split (2/3-1/3) = 32 encoder, 12 decoder layers at 330M
    The paper selects the 2/3-1/3 split as the best among three configurations (Section 3.1) rather than deriving it from theory; the performance claims rely on this choice.
assumptions (5)
  • standard math Transformer attention and KV cache behavior as described in Vaswani et al. and Wang et al.
    Used throughout for FLOPs and latency comparisons, especially the claim that one-time input processing reduces decoding cost.
  • domain assumption Teacher and student share tokenizer/vocabulary so logit alignment is valid.
    Required by Algorithm 1; the paper does not state that the encoder-decoder student uses Phi's tokenizer.
  • domain assumption Pretraining on 100B tokens of FineWeb-Edu with span corruption gives all variants a comparable starting point.
    Pretraining benchmarks are near chance, so the paper attributes downstream differences to architecture and post-training rather than pretraining quality.
  • domain assumption ONNX compilation and hardware measurements are fair to both architectures.
    Efficiency conclusions in Section 3.4 rest on this; no runtime details, batch sizes, or measurement variance are supplied.
  • domain assumption FineWeb-Edu decontamination following Tulu 3 is adequate to avoid test leakage.
    Section 3.3 cites Lambert et al. 2024 for decontamination; the paper provides no leakage checks of its own.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Return of the Encoder: Maximizing Parameter Efficiency for SLMs." pith.science (2026). https://pith.science/paper/UOZHUMCR

@misc{pith2026250116273,
  author       = {Pith},
  title        = {Pith review of: Return of the Encoder: Maximizing Parameter Efficiency for SLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UOZHUMCR}},
  note         = {Machine review of arXiv:2501.16273}
}
read the original abstract

The dominance of large decoder-only language models has overshadowed encoder-decoder architectures, despite their fundamental efficiency advantages in sequence processing. For small language models (SLMs) - those with 1 billion parameters or fewer - our systematic analysis across GPU, CPU, and NPU platforms reveals that encoder-decoder architectures achieve 47% lower first-token latency and 4.7x higher throughput compared to decoder-only models on edge devices. These gains may be attributed to encoder-decoder's one-time input processing and efficient separation of understanding and generation phases. We introduce a novel knowledge distillation framework that enables encoder-decoder models to leverage capabilities from large scalable decoder-only teachers while preserving their architectural advantages, achieving up to 6 average performance points improvement across diverse tasks, with significant gains in asymmetric sequence tasks where input and output distributions can benefit from different processing approaches. When combined with modern advances like Rotary Positional Embeddings (RoPE) and Vision encoders, our systematic investigation demonstrates that encoder-decoder architectures provide a more practical path toward deploying capable language models in resource-constrained environments. Our findings challenge the prevailing trend toward decoder-only scaling, showing that architectural choices become increasingly crucial as parameter budgets decrease, particularly for on-device and edge deployments where computational efficiency is paramount.

Figures

Figures reproduced from arXiv: 2501.16273 by the authors.

Figure 1
Figure 1. Architectural Efficiency in SLMs. Left: Comparison of architectures where encoder-decoder creates a fixed input representation with KV cache only for output, while decoder-only requires growing KV caches for both input and output. Top right: Inference time scaling with input length, showing encoder-decoder’s efficient fixed-representation approach versus decoder-only’s steeper computational growth. Bottom right: Per… view at source ↗
Figure 2
Figure 2. Performance across various model scales across top two architectures (2/3-1/3 enc-dec vs dec-only). hensive reasoning (SQuAD: 0.69/0.94 vs 0.57/0.90). Ad￾ditionally, on standard SQuAD 2.0 F1/EM metrics, the decoder-only KD model measures 70/66 while the encoder￾decoder (2/3-1/3) KD reaches 78/74. Another important remark is that while decoder-only (Seq2Seq) initially shows stronger performance in creative writing ta… view at source ↗
Figure 3
Figure 3. Vision Language Encoder-Decoder Architecture. tween encoding and generation, with the complete model comprising three stages: vision encoding, text encoding, and decoding. For comparison, we also implement a decoder￾only variant by removing the text encoder component, al￾lowing direct evaluation of architectural choices in the mul￾timodal setting. 4.2. Efficient Training Strategy Our process begins with feature alig… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Knowledge Distillation ablation study on SQuAD show￾ing Rouge-L scores across different KD configurations. First six columns compare KD methods (forward/reverse) and loss mix￾ing ratios (α) on 330M models, where α = 0.0 represents pure sequence-to-sequence learning. La…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages

  1. [1]

    A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [4]

    L., and Zou, J

    Chen, K., Thapa, R., Chalamala, R., Athiwaratkun, B., Song, S. L., and Zou, J. Dragonfly: Multi-resolution zoom supercharges large visual-language model. arXiv preprint arXiv:2406.00977,

    arXiv

  3. [5]

    K., Hong, P., Bing, L., and Poria, S

    Chia, Y . K., Hong, P., Bing, L., and Poria, S. Instructeval: Towards holistic evaluation of instruction-tuned large lan- guage models. arXiv preprint arXiv:2306.04757,

    arXiv

  4. [6]

    Ac- cessed: 2024-12-03

    URL https:// huggingface.co/datasets/chillies/ IELTS-writing-task-2-evaluation . Ac- cessed: 2024-12-03. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113,

    work page 2024

  5. [7]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  6. [8]

    M.-C., Hu, S., Liu, Z., and Collier, N

    Fu, Z., Lam, W., Yu, Q., So, A. M.-C., Hu, S., Liu, Z., and Collier, N. Decoder-only or encoder-decoder? inter- preting language model as a regularized encoder-decoder. arXiv preprint arXiv:2304.04052,

    arXiv

  7. [9]

    Accessed: 2024- 12-07

    URL https: //docs.ragas.io/en/stable/. Accessed: 2024- 12-07. Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531,

    arXiv 2024

  8. [11]

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T

    URL https: //kellerjordan.github.io/posts/muon/. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    arXiv 2001

Show all 24 references
  1. [14]

    Llava- onevision: Easy visual task transfer

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y ., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326,

    arXiv

  2. [15]

    Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306, 2024a. Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y ., Fedorov, I., Xiong, Y ...

    arXiv

  3. [17]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neu- ral networks for extreme summarization. arXiv preprint arXiv:1808.08745,

    arXiv

  4. [18]

    Radford, A

    URL https://arxiv.org/abs/ 2406.17557. Radford, A. Improving language understanding by gen- erative pre-training. Advances in Neural Information Processing Systems (2017),

    2017 arXiv

  5. [19]

    Squad: 100,000+ questions for machine com- prehension of text

    Rajpurkar, P. Squad: 100,000+ questions for machine com- prehension of text. arXiv preprint arXiv:1606.05250 ,

    arXiv

  6. [20]

    T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P

    Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408.11796,

    arXiv

  7. [21]

    Tay, Y ., Tran, V

    Accessed: 2023-11-07. Tay, Y ., Tran, V . Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., and Met- zler, D. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672,

    2023 arXiv

  8. [22]

    Q., Garcia, X., Wei, J., Wang, X., Chung, H

    Tay, Y ., Dehghani, M., Tran, V . Q., Garcia, X., Wei, J., Wang, X., Chung, H. W., Shakeri, S., Bahri, D., Schuster, T., et al. Ul2: Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022a. Tay, Y ., Wei, J., Chung, H. W., Tran, V . Q., So, D. R., Shakeri,...

    arXiv

  9. [24]

    Ureader: Uni- versal ocr-free visually-situated language understanding with multimodal large language model

    Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., et al. Ureader: Uni- versal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126,

    arXiv

  10. [2017]

    Lambert, N., Morrison, J., Pyatkin, V ., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V ., Liu, A., Dziri, N., Lyu, S., et al. T \” ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124,

    arXiv

  11. [2019]

    Leopard: A vision language model for text-rich multi-image tasks

    Jia, M., Yu, W., Ma, K., Fang, T., Zhang, Z., Ouyang, S., Zhang, H., Jiang, M., and Yu, D. Leopard: A vision language model for text-rich multi-image tasks. arXiv preprint arXiv:2410.01744,

    arXiv

  12. [2020]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp. 235–251. Springer,

    2016

  13. [2021]

    X., Tan, J

    Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

  14. [2023]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey, D. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929,

    2010 arXiv

  15. [2024]

    M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al

    Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403,

    arXiv

  16. [2025]

    Llama: Open and efficient foundation lan- guage models

    12 Return of the Encoder: Maximizing Parameter Efficiency for SLMs Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arX...

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.