Pith. sign in

REVIEW 4 major objections 5 minor 85 references

Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read One frozen LLM layer lifts small-model NLP scores on understanding and generation tasks.

desk verdict PiFi has a clean idea and unusually thorough ablations, but its central claim is currently hostage to baselines that are far below published scores. read the letter →

arxiv 2506.07424 v1 pith:4SDVHVH3 submitted 2025-06-09 cs.CL cs.AI

Kyeonghyun Kim , Jinhee Jang , Juhwan Choi , Yoonji Lee , Kyohoon Jin , YoungBin Kim This is my paper
classification cs.CLcs.AI
keywords plug-inandfine-tuningsmalllanguagemodelsfrozenLLMlayerknowledgetransferdomaingeneralizationmultilingualnaturalgenerationlow-resource
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes PiFi, a way to give a small language model some of the knowledge of a large one without paying for the large model's compute. The idea is to take a single transformer layer from a frozen LLM, insert it into a small encoder-based or encoder-decoder model, add cheap projection layers, and fine-tune only the small model plus projections on the target task. Across sentiment, NLI, QA, translation, and summarization benchmarks, PiFi reports consistent gains over vanilla fine-tuning of the same small models. The authors argue the gain comes from the LLM layer's pretrained knowledge, not from added parameters, because random layers or a similarly sized BERT-large do not reproduce it. If true, this offers a drop-in recipe for improving resource-constrained models by reusing one layer of an LLM.

What carries the argument

The central mechanism is the 'plug-in' residual insertion: $h_{LLM} = L_{LLM}(L_{in}(h_{enc}))$, with output $L_{out}$ mapping back to the small model's hidden size. $L_{LLM}$ is one frozen transformer layer extracted from an LLM such as Llama-3.1-8B or Mistral-7B; for classification, only the CLS token passes through it, which keeps the added FLOPs around 2.6%. The fine-tuning objective trains the original SLM, the two projections, and the head, but never updates $L_{LLM}$, preserving the layer's pretrained knowledge and avoiding catastrophic forgetting.

What would settle it

Re-run the Table 1 comparisons with per-model hyperparameter search that reproduces the standard published accuracies for BERT-base and RoBERTa; if the tuned vanilla models equal or exceed PiFi's numbers on SQuAD v1.1 and SST-2, the transfer claim would be falsified.

Watch

Extended reading notes

Core claim

PiFi's central claim is that a single frozen LLM layer, placed between a small model's encoder and its task head (or between encoder and decoder) and accompanied by learned input/output projections, transfers enough LLM-style generalization to raise the fine-tuned small model's performance on both understanding and generation tasks. The paper reports average accuracy gains of roughly 1-3 percentage points across five encoder models on NLU benchmarks, larger relative gains on out-of-domain transfer and on low-data settings, and gains on translation and summarization metrics for T5 and BART. The analysis isolates the mechanism: pretrained layer weights matter, freezing them matters, and matching the LLM's language to the task language matters.

Load-bearing premise

The claim rests on the premise that the single fixed training schedule (three epochs, learning rate 5e-5, batch size 32) gives a fair baseline for the vanilla small models; if those baselines are undertuned, PiFi could look better than it is.

Editorial extensions

If this is right

  • PiFi gives encoder-based classifiers like BERT a few points on SST-2, IMDB, offensive language, CoLA, MNLI, SNLI, and SQuAD v1.1 over its own vanilla baselines.
  • It improves out-of-domain transfer: for instance, a BERT-plus-PiFi model trained on IMDB gains about 13 accuracy points on Tweet sentiment over the vanilla model.
  • Language-specific adaptation works: plugging a Korean Llama-3 layer into mBERT adds further gains on Korean classification, and a German layer does likewise for German.
  • Under 10% of the training data, PiFi retains its advantage over vanilla fine-tuning, suggesting value in low-resource settings.
  • Because the inserted layer is frozen and only a single token is processed in classification, the added inference cost is small relative to running the full LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own Limitations section flags that picking the last LLM layer is heuristic and that the benchmark suite is limited to relatively straightforward tasks; automatic layer selection and harder multitask benchmarks are the clear next probes.
  • The efficiency measurement of about 2.6% added FLOPs is made for the CLS-token classification setting; in generation, the inserted layer sees a full encoded sequence, so the same cost advantage should not be assumed without measurement.
  • The gains over a same-size BERT-large suggest the benefit comes from the layer's pretrained content, but a comparison against a well-tuned larger SLM would test whether PiFi remains additive where the small model is already strong.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes PiFi, a training method that inserts one frozen transformer layer extracted from an LLM into a small encoder-based or encoder-decoder LM, with trainable linear projection layers, and then fine-tunes the combined model on downstream tasks. The authors report consistent improvements over vanilla SLM fine-tuning across text classification, NLI, SQuAD, machine translation, and summarization, and they include additional experiments on domain shift, multilingual transfer, limited training data, different LLM sources, and ablation controls.

Significance. PiFi is methodologically simple and, if the reported gains are robust, offers a low-cost way to transfer LLM knowledge to SLMs. The paper's strengths include broad task coverage, a random-initialization control (Table 5), a parameter-matched comparison with BERT-large (Table 6), several further ablations, and a statement that source code is released. However, the current evidence is weakened by vanilla baselines that are far below published results for the same models and by the absence of error bars or significance tests for the main comparisons, so the magnitude and consistency of the claimed gains require confirmation.

major comments (4)
  1. [Section 4.1, Appendix A, Tables 1-2] The fixed training schedule reported in Appendix A (3 epochs, LR 5e-5, batch 32) produces vanilla baselines that are far below established results for the same models. For example, BERT-base SQuAD EM is 63.81 in Table 1, whereas the original BERT paper reports roughly 80.8 EM, and SST-2 is 89.41 versus roughly 93.5. The ELECTRA SQuAD baseline of 44.44 EM is even more concerning. Because every PiFi gain in Tables 1, 2, 7, 8, 13, 17, and 19 is measured against these weak baselines, the claimed consistent improvement may be an artifact of added capacity compensating for an undertrained baseline. Please add properly tuned vanilla baselines (e.g., per-task learning-rate search, early stopping, or published fine-tuning schedules) and compare PiFi against those.
  2. [Section 4.1, Tables 1, 3, 13, 19] The main results are averages over five seeds, but no standard deviations, confidence intervals, or paired significance tests are reported; the only p-values (Table 12) concern a different comparison. Several reported gains are small (e.g., RoBERTa MNLI accuracy 84.00 to 84.25, DeBERTa SQuAD F1 0.8249 to 0.8283), so the claim that PiFi delivers 'consistent performance improvements' is not statistically grounded. Please report per-seed variance or confidence intervals and run paired tests across the five seeds for at least the main NLU and NLG comparisons.
  3. [Section 3.2, Section 4.3, Appendix B.8, Table 16] The efficiency claim in the abstract and Section 5 is supported only by measurements on a CLS-token classification setting, where the LLM layer processes a single token. In the generation experiments on Multi30K and CNN/DailyMail, the entire encoded sequence is passed through the LLM layer, so the FLOPs and memory overhead are not captured by Table 16. Please measure or bound the inference cost for the encoder-decoder/generation setting, or explicitly qualify the efficiency claim to classification tasks.
  4. [Section 3.1, Section 4.2, Table 1] The method description in Section 3.1 defines only a classification head, yet Table 1 reports SQuAD exact match and F1. No span-extraction head, QA loss, or decoding procedure is described. If a classification-style head was used for SQuAD, the numbers are not comparable to standard QA evaluations; if a span head was used, it must be specified. This is especially important given the very low ELECTRA SQuAD baseline (44.44 EM) and the large jump to 67.99 EM after applying PiFi, which suggests a possible mismatch in the evaluation setup.
minor comments (5)
  1. [Appendix B.6] The term 'layer knowledge density' is introduced to explain why larger models do not consistently improve PiFi, but it is never defined or measured; either provide a concrete definition with a supporting measurement or remove the term.
  2. [Appendix B.2, Table 9] The table header says 'SmolLM2-135B' while the text refers to 'SmolLM2-135M'; the parameter count and context indicate the intended model is 135M.
  3. [Section 2.1] There is a typo: 'DeBERTa DeBERTa (He et al., 2021)' duplicates the model name.
  4. [Tables 1-4, 13] Several captions and table entries use inconsistent model names (e.g., 'Llam-3-8B' in Table 3, 'Llama-3-8B' elsewhere) and some cells align accuracy and F1 values with unclear column separators; please standardize table formatting and model naming.
  5. [Appendix B.8, Table 16] The GPU memory measurement shows an increase from 1.33 GB to 2.27 GB (about 70%), and the text describes this as 'relatively larger' but 'reasonable'; this trade-off deserves a more quantitative discussion in terms of total memory budget, especially for resource-constrained deployment.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: PiFi's claims are empirical, externally benchmarked, and supported by random-initialization and parameter-matched controls; the only author-overlap citation appears in related work and is not load-bearing.

full rationale

The paper's central claim is empirical: inserting a frozen LLM layer into an SLM improves fine-tuned performance on standard benchmarks. There is no derivation chain in which an output is defined as an input; PiFi is an architecture proposal, not a fitted predictor, and no quantity labeled a prediction is constructed from the same data it is tested on. The controls that normally reveal circularity are present: pre-trained LLLM outperforms frozen randomly initialized LLLM and even fine-tuned randomly initialized LLLM (PiFi-Random and PiFi-Random-Full in Table 5); PiFi applied to BERT-base outperforms a similarly sized BERT-large (Table 6); and PiFi is compared against alternative knowledge-transfer and parameter-efficient baselines (ZEROGEN in Table 17, LoRA in Table 18). The only author-overlap reference, Choi et al. (2024), appears in a related-work survey sentence on non-parametric distillation (Section 2.2) and is neither used to justify PiFi's design nor invoked to rule out alternatives. Concerns about weak vanilla baselines under the uniform 3-epoch, 5e-5 schedule (e.g., BERT-base SQuAD EM of 63.81 versus roughly 80.8 reported in prior work) are correctness and comparison-fairness risks, not circular reductions: the paper's own equations do not guarantee any particular gain, and the results are externally falsifiable. Therefore, no significant circularity is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on domain assumptions about frozen layer transfer and on a training schedule that is fixed across all tasks. No free parameters are fit in the statistical sense, but the layer position, layer count, token representation, and schedule are hand-chosen and validated on the same benchmarks, which entangles them with the claimed gains.

free parameters (4)
  • LLM layer position = last layer (layer 32 of 32 in Llama-3.1-8B)
    Selected by heuristic; Appendix B.1 shows it wins on SST-2 and Multi30K but the paper admits the optimum may vary by task.
  • Number of plugged LLM layers = 1
    The core design choice. Appendix B.6 tries two layers from Llama-3.1-70B and finds only marginal gains, so one layer is retained.
  • Token representation (encoder LMs) = CLS token only
    Chosen after Appendix B.3 comparison with average pooling; the choice reduces the LLM layer's input to a single token, changing what knowledge can be used.
  • Fine-tuning schedule = 3 epochs, LR 5e-5, batch size 32, Adam
    A single schedule is applied to all tasks and models (Appendix A), producing baselines well below published scores; the schedule is therefore a hand-chosen factor that the size of the reported gains depends on.
assumptions (4)
  • domain assumption A single frozen pre-trained LLM layer, accessed through learned linear projections, retains task-relevant knowledge when transplanted into an SLM.
    This is the core hypothesis, stated in Section 3 and tested indirectly through ablations; no mechanistic proof is given.
  • domain assumption Fine-tuning the host SLM around the frozen layer does not destroy the SLM's own representations or the transferred knowledge.
    Stated in Section 3.3 with catastrophic forgetting rationale; tested only via the freeze/full-fine-tune ablation in B.4.
  • domain assumption The benchmark datasets and evaluation metrics in Section 4 are processed consistently with standard practice.
    Baseline numbers (e.g., BERT base SQuAD 63.81 EM) are far below published values, so either the processing or the training schedule diverges from standard practice; the paper does not compare with published results.
  • domain assumption Transformer layers are compatible across model families despite differences in normalization, attention, and hidden sizes.
    The method assumes any LLM layer can be inserted after learned projections; tested for several LLMs but not theoretically justified.
invented entities (1)
  • Layer knowledge density
    purpose: Post-hoc explanation for why a layer from Llama-3.1-70B does not outperform one from Llama-3.1-8B (fewer layers in 8B imply denser per-layer knowledge).
    Introduced in B.6; no independent measurement or falsifiable prediction is provided. It is a label for the observed size plateau, not an entity with empirical support.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models." pith.science (2026). https://pith.science/paper/4SDVHVH3

@misc{pith2026250607424,
  author       = {Pith},
  title        = {Pith review of: Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SDVHVH3}},
  note         = {Machine review of arXiv:2506.07424}
}
read the original abstract

Large language models (LLMs) are renowned for their extensive linguistic knowledge and strong generalization capabilities, but their high computational demands make them unsuitable for resource-constrained environments. In contrast, small language models (SLMs) are computationally efficient but often lack the broad generalization capacity of LLMs. To bridge this gap, we propose PiFi, a novel framework that combines the strengths of both LLMs and SLMs to achieve high performance while maintaining efficiency. PiFi integrates a single frozen layer from an LLM into a SLM and fine-tunes the combined model for specific tasks, boosting performance without a significant increase in computational cost. We show that PiFi delivers consistent performance improvements across a range of natural language processing tasks, including both natural language understanding and generation. Moreover, our findings demonstrate PiFi's ability to effectively leverage LLM knowledge, enhancing generalization to unseen domains and facilitating the transfer of linguistic abilities.

Figures

Figures reproduced from arXiv: 2506.07424 by the authors.

Figure 1
Figure 1. The comparison between vanilla fine-tuning of SLMs and our proposed PiFi architecture. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the performance of PiFi models with different position of [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the performance of PiFi models with different position of [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

85 extracted references · 62 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart \' n Bl \'a zquez, Guilherme Penedo, Lewis Tunstall, Andr \'e s Marafioti, Hynek Kydl \' c ek, Agust \' n Piqueres Lajar \' n, Vaibhav Srivastav, et al. 2025. https://arxiv.org/abs/2502.02737 Smollm2: When smol goes big--data-centric training of a small language model . arXiv preprint arXiv:2502.02737

    arXiv 2025

  3. [3]

    Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, M \'e rouane Debbah, \'E tienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. https://arxiv.org/abs/2311.16867 The falcon series of open language models . arXiv preprint arXiv:2311.16867

    arXiv 2023

  4. [4]

    Amit Artzy and Roy Schwartz. 2024. https://aclanthology.org/2024.blackboxnlp-1.10/ Attend first, consolidate later: On the importance of attention in different llm layers . In Proceedings of EMNLP 2024 BlackboxNLP Workshop, pages 177--184

    2024

  5. [5]

    Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65--72

    2005

  6. [6]

    Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. https://aclanthology.org/2020.findings-emnlp.148 T weet E val: Unified benchmark and comparative evaluation for tweet classification . In Findings of EMNLP, pages 1644--1650

    2020

  7. [7]

    Bowman, Gabor Angeli, Christopher Potts, and Christopher D

    Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://aclanthology.org/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of EMNLP, pages 632--642

    2015

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html Language models are few-shot learners . In Proceedings of NeurIPS, pages 1877--1901

    2020

Show all 85 references
  1. [9]

    Juhwan Choi, Yeonghwa Kim, Seunguk Yu, Jungmin Yun, and Youngbin Kim. 2024. https://aclanthology.org/2024.emnlp-main.1/ Unigen: Universal domain generalization for sentiment classification via zero-shot dataset generation . In Proceedings of EMNLP, pages 1--14

    2024

  2. [10]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://openreview.net/forum?id=r1xMH1BtvB Electra: Pre-training text encoders as discriminators rather than generators . In Proceedings of ICLR

    2020

  3. [11]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html Qlora: Efficient finetuning of quantized llms . In Proceedings of NeurIPS, pages 10088--10115

    2023

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://aclanthology.org/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of NAACL, pages 4171--4186

    2019

  5. [13]

    Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. https://dl.acm.org/doi/10.1145/1341531.1341561 A holistic lexicon-based approach to opinion mining . In Proceedings of WSDM, pages 231--240

    2008

  6. [14]

    DiscoResearch. 2024. https://huggingface.co/DiscoResearch/Llama3-German-8B Llama-3-german-8b . Hugging Face Repository

    2024

  7. [15]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://openreview.net/forum?id=YicbFdNTTy An image is worth 1...

    2021

  8. [16]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    2024 arXiv

  9. [17]

    Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. https://aclanthology.org/W16-3210 Multi30k: Multilingual english-german image descriptions . In Proceedings of ACL 2016 Workshop on Vision and Language, pages 70--74

    2016

  10. [18]

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. https://www.jmlr.org/papers/v20/18-598.html Neural architecture search: A survey . Journal of Machine Learning Research, 20(55):1--21

    2019

  11. [19]

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/6fa4d985e7c434002fb6289ab9b2d654-Abstract-Conference.html Improving clip training with language rewrites . In Proceedings of NeurIPS, page...

    2023

  12. [20]

    Jiahui Gao, Renjie Pi, Lin Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023 a . https://openreview.net/forum?id=h5OpjGd_lo6 Self-guided noise-free data generation for efficient zero-shot learning . In Proceedings of ICLR

    2023

  13. [21]

    Ze-Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao, and Ji-Rong Wen. 2023 b . https://aclanthology.org/2023.acl-long.212 Small pre-trained language models can be fine-tuned as large models via over-parameterization . In Proceedings of ACL, pages 3819--3834

    2023

  14. [22]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. https://openreview.net/forum?id=5h0qf7IBZZ Minillm: Knowledge distillation of large language models . In Proceedings of ICLR

    2024

  15. [23]

    Oliver Guhr, Anne-Kathrin Schumann, Frank Bahrmann, and Hans Joachim B \"o hme. 2020. https://aclanthology.org/2020.lrec-1.202 Training a broad-coverage G erman sentiment classification model for dialog systems . In Proceedings of LREC, pages 1627--1632

    2020

  16. [24]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://aclanthology.org/2020.acl-main.740 Don’t stop pretraining: Adapt language models to domains and tasks . In Proceedings of ACL, pages 8342--8360

    2020

  17. [25]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. https://openreview.net/forum?id=sE7-XhLxHA Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . In Proceedings of ICLR

    2023

  18. [26]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In Proceedings of ICLR

    2021

  19. [27]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In Proceedings of ICLR

    2021

  20. [28]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In Proceedings of ICLR

    2022

  21. [29]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. https://arxiv.org/abs/2310.06825 Mistral 7b . arXiv preprint arXiv:2310.06825

    2023 arXiv

  22. [30]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://aclanthology.org/2020.findings-emnlp.372 T iny BERT : Distilling BERT for natural language understanding . In Findings of EMNLP, pages 4163--4174

    2020

  23. [31]

    Tianjie Ju, Weiwei Sun, Wei Du, Xinwei Yuan, Zhaochun Ren, and Gongshen Liu. 2024. https://aclanthology.org/2024.lrec-main.722 How large language models encode context knowledge? a layer-wise probing study . In Proceedings of LREC-COLING, pages 8235--8246

    2024

  24. [32]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. https://arxiv.org/abs/1412.6980 Adam: A method for stochastic optimization . In Proceedings of ICLR

    2015 arXiv

  25. [33]

    Junbum Lee. 2024. https://huggingface.co/beomi/Llama-3-Open-Ko-8B Llama-3-open-ko . Hugging Face Repository

    2024

  26. [34]

    Pierre Lepagnol, Thomas Gerald, Sahar Ghannay, Christophe Servan, and Sophie Rosset. 2024. https://aclanthology.org/2024.lrec-main.1299 Small language models are good too: An empirical study of zero-shot classification . In Proceedings of LREC-COLING, pages 14923--14936

    2024

  27. [35]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://aclanthology.org/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translation, ...

    2020

  28. [36]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ Rouge: A package for automatic evaluation of summaries . In Proceedings of ACL 2004 Workshop Text Summarization Branches Out, pages 74--81

    2004

  29. [37]

    Zheng Lin, Guanqiao Qu, Qiyuan Chen, Xianhao Chen, Zhe Chen, and Kaibin Huang. 2023. https://openreview.net/forum?id=VF8OGR9skB Pushing large language models to the 6g edge: Vision, challenges, and opportunities . CoRR

    2023

  30. [38]

    Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022. https://www.sciencedirect.com/science/article/pii/S0004370221002058?dgcid=coauthor Quantifying and alleviating political bias in language models . Artificial Intelligence, 304:103654

    2022

  31. [39]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized BERT pretraining approach . arXiv preprint arXiv:1907.11692

    2019 arXiv

  32. [40]

    Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024. https://aclanthology.org/2024.findings-acl.866 Fantastic semantics and where to find them: Investigating which layers of generative LLM s reflect lexical semantics . In Findings of ACL, pages 14551--14558

    2024

  33. [41]

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. 2024. https://arxiv.org/abs/2409.15790 Small language models: Survey, measurements, and insights . arXiv preprint arXiv:2409.15790

    2024 arXiv

  34. [42]

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. https://arxiv.org/abs/2308.08747 An empirical study of catastrophic forgetting in large language models during continual fine-tuning . arXiv preprint arXiv:2308.08747

    2023 arXiv

  35. [43]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. https://aclanthology.org/P11-1015 Learning word vectors for sentiment analysis . In Proceedings of ACL, pages 142--150

    2011

  36. [44]

    Meta. 2019. https://github.com/facebookresearch/fvcore fvcore . GitHub Repository

    2019

  37. [45]

    Meta. 2024. https://huggingface.co/meta-llama/Llama-3.1-8B Llama-3.1-8b . Hugging Face Repository

    2024

  38. [46]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. https://arxiv.org/abs/2402.06196 Large language models: A survey . arXiv preprint arXiv:2402.06196

    2024 arXiv

  39. [47]

    u l c ehre, and Bing Xiang

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C a g lar G \. u l c ehre, and Bing Xiang. 2016. https://aclanthology.org/K16-1028 Abstractive text summarization using sequence-to-sequence RNN s and beyond . In Proceedings of SIGNLL, pages 280--290

    2016

  40. [48]

    Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Radityo Eko Prasojo, and Alham Fikri Aji. 2020. https://arxiv.org/abs/2012.08958 Costs to consider in adopting nlp for your business . arXiv preprint arXiv:2012.08958

    2020 arXiv

  41. [49]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html Trai...

    2022

  42. [50]

    Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. 2024. https://openreview.net/forum?id=t0FI3Q66K5 Frozen transformers in language models are effective visual encoder layers . In Proceedings of ICLR

    2024

  43. [51]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://aclanthology.org/P02-1040 Bleu: A method for automatic evaluation of machine translation . In Proceedings of ACL, pages 311--318

    2002

  44. [52]

    Lucy Park. 2016. https://github.com/e9t/nsmc/ Naver sentiment movie corpus . GitHub Repository

    2016

  45. [53]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  46. [54]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019 b . https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html Pytorch: An imper...

    2019

  47. [55]

    Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. https://aclanthology.org/P19-1493 How multilingual is multilingual BERT ? In Proceedings of ACL, pages 4996--5001

    2019

  48. [56]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. https://proceedings.mlr.press/v139/radford21a Learning transferable visual models from natural language supervision ...

    2021

  49. [57]

    Alec Radford and Karthik Narasimhan. 2018. https://openai.com/index/language-unsupervised/ Improving language understanding by generative pre-training . OpenAI Blog

    2018

  50. [58]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. https://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  51. [59]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://aclanthology.org/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of EMNLP, pages 2383--2392

    2016

  52. [60]

    Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. https://aclanthology.org/S17-2088/ Semeval-2017 task 4: Sentiment analysis in twitter . In Proceedings of ACL 2017 Workshop on Semantic Evaluation, pages 502--518

    2017

  53. [61]

    Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, and Heng Ji. 2023. https://aclanthology.org/2023.findings-emnlp.608 Democratizing llms: An exploration of cost-performance trade-offs in self-refined open-source models . In Findings of EMNLP, pages 9070--9084

    2023

  54. [62]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170 Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of EMNLP, pages 1631--1642

    2013

  55. [63]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical s...

    2024 arXiv

  56. [64]

    Inar Timiryasov and Jean-Loup Tastet. 2023. https://aclanthology.org/2023.conll-babylm.24 Baby llama: Knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty . In Proceedings of CoNLL 2024 BabyLM Challenge, pages 279--289

    2023

  57. [65]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Proceedings of Ne...

    2017

  58. [66]

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023. https://aclanthology.org/2023.findings-emnlp.715 Orthogonal subspace learning for language model continual learning . In Findings of EMNLP, pages 10658--10671

    2023

  59. [67]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. https://aclanthology.org/Q19-1040 Neural network acceptability judgments . Transactions of the Association for Computational Linguistics, 7:625--641

    2019

  60. [68]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. https://openreview.net/forum?id=yzkSU5zdwD Emergent abilities of large language models . Transactions on Machine Learning Research

    2022

  61. [69]

    Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. https://aclanthology.org/2022.naacl-main.341 Symbolic knowledge distillation: From general language models to commonsense models . In Proceeding...

    2022

  62. [70]

    Colin White, Mahmoud Safari, Rhea Sukthanker, Binxin Ru, Thomas Elsken, Arber Zela, Debadeepta Dey, and Frank Hutter. 2023. https://arxiv.org/abs/2301.08727 Neural architecture search: Insights from 1000 papers . arXiv preprint arXiv:2301.08727

    2023 arXiv

  63. [71]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://aclanthology.org/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of NAACL, pages 1112--1122

    2018

  64. [72]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2020. https://aclanthology.org/2020.emnlp-demos.6/ Transformers: State-of-the-art natural language processing . In Proce...

    2020

  65. [73]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . arXiv preprint arXiv:2407.10671

    2024 arXiv

  66. [74]

    Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. https://aclanthology.org/2022.emnlp-main.801/ Zerogen: Efficient zero-shot learning via dataset generation . In Proceedings of EMNLP, pages 11653--11669

    2022

  67. [75]

    Qinyuan Ye. 2024. https://aclanthology.org/2024.naacl-srw.27 Cross-task generalization abilities of large language models . In Proceedings of NAACL 2024 Student Research Workshop, pages 255--262

    2024

  68. [76]

    Hao Yu, Zachary Yang, Kellin Pelrine, Jean-François Godbout, and Reihaneh Rabbany. 2023. https://arxiv.org/abs/2308.10092 Open, closed, or small language models for text classification? arXiv preprint arXiv:2308.10092

    2023 arXiv

  69. [77]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Abstract.html Bartscore: Evaluating generated text as text generation . In Proceedings of NeurIPS, pages 27263--27277

    2021

  70. [78]

    Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. https://aclanthology.org/S19-2010/ Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval) . In Proceedings of NAACL 2019 Workshop on...

    2019

  71. [79]

    Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. 2024 a . https://openreview.net/forum?id=d4UiXAHN2W LL a MA -adapter: Efficient fine-tuning of large language models with zero-initialized attention . In Proceedings of ICLR

    2024

  72. [80]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In Proceedings of ICLR

    2020

  73. [81]

    Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. 2024 b . https://aclanthology.org/2024.blackboxnlp-1.29/ Investigating layer importance in large language models . In Proceedings of EMNLP 2024 BlackboxNLP Workshop, pages 469--479

    2024

  74. [82]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. https://arxiv.org/abs/2303.18223 A survey of large language models . arXiv preprint arXiv:2303.18223

    2023 arXiv

  75. [83]

    Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. 2024. https://openreview.net/forum?id=mIEHIcHGOo Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective . In Proceedings of ICLR

    2024

  76. [84]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  77. [85]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.