TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs

Hao Zhang; Lanxiang Hu; Tajana Rosing

REVIEW 4 major objections 5 minor 47 references

TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TrimLLM shows that LLM layers are domain-specialized and that dropping the least important ones during fine-tuning yields a model at 40–50% of original size with nearly no accuracy loss on the target domain.

desk verdict TrimLLM has a practical idea and promising results, but the 'no loss' claim is not yet established because calibration and evaluation may share the same examples. read the letter →

arxiv 2412.11242 v2 pith:STP76XEO submitted 2024-12-15 cs.LG cs.AIcs.CL

Lanxiang Hu , Tajana Rosing , Hao Zhang This is my paper · ORCID

classification cs.LGcs.AIcs.CL

keywords LLMcompressionlayerdroppingdomainspecializationfine-tuninginferencespeedupimportanceprogressivemodelefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the layers of a large language model are not equally valuable across knowledge domains: some layers, particularly the MLP layers, carry most of the domain-specific knowledge, while others can be removed without hurting performance. Building on this, the authors propose TrimLLM, a fine-tuning procedure that repeatedly identifies the least important layer using a small calibration set and drops it, continuing until the model reaches a target size. On LLaMA-7B and LLaMA-13B fine-tuned on medical, legal, financial, and commonsense QA benchmarks, they report that models can be cut to 40–50% of their original size with nearly no accuracy loss, and that the resulting shallower models run 2.1–5.7× faster on consumer GPUs without any special hardware or kernel support. The significance is a compression method that gives real wall-clock speedups on ordinary hardware, unlike quantization and pruning methods that often need specialized kernels to beat the baseline.

What carries the argument

The core mechanism is a two-part importance metric evaluated on a small calibration set drawn from the fine-tuning validation set. The primary score is sensitivity-based: temporarily delete each layer, measure the model's accuracy, and rank layers by the reciprocal of that accuracy via $s_{i,\mathrm{scan}} = (100 - a_i)/((1+\delta^2)+(1+\delta)a_i)$. The tie-breaker is an activation-norm score using the Frobenius norm of each layer's activations, $s_{i,\mathrm{norm}} = 100 \min\{\|x_j\|_F\}/\|x_i\|_F$, which approximates the nuclear norm while avoiding SVD cost. After each epoch the layer with the lowest score is removed, and a sparse-update rule freezes layers whose initial importance scores mark them as likely to be dropped, keeping only $r=1/4$ of layers trainable to offset the extra epochs needed.

What would settle it

Compute the Spearman rank correlation between the initial importance scores of all layers and the importance scores of the remaining layers after the model has been reduced to 75% of its original size via TrimLLM's greedy dropping; if the correlation is below, say, 0.5, the paper's assumption that fine-tuning and layer dropping do not disturb importance orderings is false, and the sparse-update freezing could be removing layers that would have been kept.

Watch

Extended reading notes

Core claim

TrimLLM establishes the layer-wise specialization phenomenon: for any downstream task distribution, there is a subset of the original transformer layers whose composition can be fine-tuned to approximate the full model's output distribution. The method operationalizes this by assigning each layer an importance score from a calibration scan (accuracy after deleting the layer) with an activation-norm tie-breaker, then greedily dropping the lowest-scoring layer after each epoch of fine-tuning, while freezing all but a quarter of the layers to control training cost. In experiments on LLaMA-7B and LLaMA-13B across SciQ, PIQA, MedMCQA, LexGLUE, and FinanceQA, TrimLLM maintains more than 90% of full-fine-tuning accuracy at 50% compression, and on MedMCQA the 40%-size model still beats the zero-shot baseline by a wide margin.

Load-bearing premise

The load-bearing premise is that a layer's importance score measured at the start (or after each epoch) stays stable enough as layers are dropped and weights are fine-tuned to guide the whole greedy removal sequence, so that the frozen layers never include one that later becomes essential.

Editorial extensions

If this is right

If layer-wise specialization holds broadly, model compression can become a standard fine-tuning step that yields measured speedups on any hardware, since reducing depth directly cuts sequential compute regardless of kernel support.
TrimLLM's compression is orthogonal to quantization and structured pruning; combining it with AWQ-int4 on LLaMA-7B reportedly reaches an 8× memory compression ratio with a 4.5× throughput gain on A100, suggesting stacked compression is viable.
The calibration scan and activation-norm tie-breaker produce a flexible trade-off frontier, letting practitioners pick any target model size (such as 50%, 40%, or 30%) rather than the discrete operating points quantization offers.
The method's effectiveness on OPT-1.3B and OPT-6.7B indicates the phenomenon is not specific to LLaMA, though the paper only tests a handful of architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If layer-wise specialization is real, then the common practice of fine-tuning all layers may be wasteful: optimal domain adaptation could instead edit only a small residual subnetwork, and the dropped layers could be repurposed as capacity for out-of-domain tasks or continual learning.
The reliance on a calibration set from the validation distribution means TrimLLM's reported numbers likely reflect in-domain tuning; a natural extension is to test whether importance scores transfer across domains, for example whether a ranking computed on medical data predicts which layers matter for legal data.
The stability assumption (initial importance scores predict later ones) could be tested directly by recomputing scores mid-run; if the ranking flips, the sparse-update freezing would need to be replaced by a periodic re-freeze.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

TrimLLM is a practical, well-scoped contribution to the layer-dropping line of work. The new bit is doing the dropping during fine-tuning, guided by a calibration-scan importance score with an activation-norm tie-breaker and sparse updates. That combination is not in the cited prior work, and it's a reasonable way to adapt a model to a domain while compressing it. The paper also backs the method with experiments across five benchmarks and three model families, and the speedups on V100 and RTX 3090 look real—depth reduction doesn't need special kernels.

The soft spots are about what the numbers can support. The stress-test note is on target: Section 3.3 says the calibration set is a subset of the validation set, and the paper never states that those examples are excluded from the final lm-evaluation-harness scores. If the same split is used for both importance scanning and final accuracy, the headline results are directly contaminated. The fix is straightforward—state the split, or better, re-evaluate on a held-out test set.

Second, there are no error bars. At 50% remaining size, TrimLLM is 0.6–1.5 accuracy points below full fine-tuning on every benchmark. That may be within noise, but without multiple seeds it's impossible to call it "no loss." The abstract overclaims.

Third, the A100 speedup claims need a second look. The abstract says "up to 3.1× speedup on A100." Table 1 shows AWQ at 115.3 tokens/s and TrimLLM at 103.1 on the same GPU—so on A100, TrimLLM is slower than AWQ. The numbers only beat AWQ on consumer GPUs. The comparison basis should be specified.

The importance-score stability assumption in Section 3.4 is stated as a correlation without evidence. It's plausible, but greedy layer selection based on initial scores could lock in bad choices if scores shift after other layers are removed. That's a moderate concern; an ablation that recomputes importance after each drop would settle it.

Who should read this: anyone working on efficient deployment of LLMs, especially pruning and quantization. The method is orthogonal to quantization, and the composition results are useful. The paper deserves a serious referee—the shortcomings are fixable in revision, not fundamental. I'd hold off citing it until the eval protocol is clarified.

Referee Report

4 major / 5 minor

Summary. This paper proposes TrimLLM, a method that specializes an LLM to a domain by fine-tuning while progressively dropping decoder layers deemed least important, using either a deletion-accuracy calibration score or an activation-norm score. The authors report that LLaMA-7B can be reduced to 40-50% of its original size with nearly no accuracy loss on SciQ and MedMCQA, that TrimLLM achieves 2.1-5.7x inference speedup on consumer GPUs relative to quantization/pruning baselines, and that the method is orthogonal to quantization and sparsity techniques, yielding up to 8x compression when combined with AWQ or SparseGPT. The empirical evaluation covers LLaMA-7B/13B and OPT-1.3B/6.7B on commonsense, medical, legal, and financial QA benchmarks.

Significance. If the central empirical claims hold, TrimLLM would be a practically valuable compression method: it reduces model depth, so inference speedup does not depend on specialized kernels or sparsity-aware hardware, and it integrates naturally with existing post-training compression. The paper has several strengths: it evaluates across multiple model families and domains, compares against quantization and pruning baselines, reports wall-clock throughput on three GPU types, and includes ablations of the scoring methods and sparse-update ratios. The layer-wise specialization hypothesis is clearly stated and falsifiable. However, the evaluation currently lacks explicit separation between calibration and test splits, contains no error bars, and includes hyperparameter choices made on the same benchmarks, so the headline 'no loss in accuracy' claim is not yet established.

major comments (4)

[§3.3 and §4.1, Algorithm 1] Section 3.3 states that the calibration dataset is 'a small subset of the fine-tuning dataset's validation set,' and Algorithm 1 uses it for every layer-importance scan, but Section 4.1 and Table 2 never state that the examples used for the reported final accuracies exclude this calibration subset. If the same split supplies both layer selection and final evaluation, the layer choices and stopping point are selected on the evaluation set and the headline 'no loss' claims are contaminated. Please state the exact split provenance, report results on a held-out test split disjoint from calibration, and clarify whether any hyperparameters (sparse-update ratio r, stopping thresholds) were selected on the same benchmarks.
[§4.1, Table 2] The paper reports a single accuracy value per condition with no variance or confidence intervals. At 50% remaining size, Table 2 shows differences relative to full fine-tuning of -0.6 (PIQA), -1.4 (SciQ), -1.5 (MedMCQA), -0.9 (LexGLUE), and -1.5 (FinanceQA) points for LLaMA-7B. Without repeated-seed or bootstrap estimates, and without an operational definition of 'no loss,' these differences cannot be distinguished from noise, and the central compression claim is not quantitatively supported. Please provide variance estimates and state the threshold used to define 'no loss.'
[§3.4] The assumption that 'the initial distribution is highly correlated with the latter ones' is load-bearing for the greedy one-at-a-time layer-dropping algorithm, but it is stated without evidence. If importance rankings shift after layers are removed or after sparse fine-tuning, early choices can lock in a poor sequence of removals. Please verify this correlation empirically by re-scanning importance scores at several stages of the dropping process, and ideally compare against a strategy that periodically re-ranks layers.
[Abstract and §4.2, Table 1] The abstract's claim of '2.1-5.7x inference speedup ... compared to state-of-the-art model compression algorithms' is not supported on all reported hardware: in Table 1, TrimLLM achieves 103.1 tokens/s on A100 versus 115.3 tokens/s for AWQ-int4, i.e., it is slower than that baseline on that GPU. Additionally, the 'no loss in accuracy at 50-60% compression' claim is not consistent with Table 8, where OPT-6.7B at 50% remaining size drops by 6.8 points on SciQ (88.5 vs 95.3) and 3.5 points on LexGLUE (37.5 vs 41.0). Please qualify the speedup and accuracy claims to the specific hardware and model families for which they hold.

minor comments (5)

[Algorithm 1, lines 13-14] Line 13 says 'append sm to UX', but the algorithm should append the layer index m (or a layer identifier), not the importance score; as written, the set UX is populated with scores rather than layer indices. The notation GUX0 in line 3 is also used before UX is well-defined.
[§3.3] The text repeatedly writes 'Forbenius norm'; this should be 'Frobenius norm.' Also, the paper never specifies the value of δ in Equation (3), leaving an important free parameter of the scoring function unstated.
[§4.1] The reference to 'lm-evaluation-hardness' should be 'lm-evaluation-harness,' and Table 7's caption contains the typo 'meausred'; please correct these.
[§4.1 and Table 3] The paper states that the two-step algorithm gives the best model at every sparse-update ratio, but Table 3 only shows r=1/4 in the main text; the claim across all ratios relies on Appendix Table 6 and should be stated as such. More importantly, the table does not report the number of calibration examples used, which is needed to assess the informativeness of the deletion-based scores.
[Reproducibility] The paper does not state whether code or model checkpoints will be released; for a compression method whose value depends on measured speedups and exact evaluation splits, providing code and split files would greatly aid verification.

Circularity Check

2 steps flagged · score 5.0 of 10

Layer selection and the sparse-update ratio r are fitted on the same QA benchmarks that produce the headline 'no loss in accuracy' claim; the paper never states that calibration and evaluation splits are disjoint.

fitted input called prediction [Section 3.3, Eq. (3); Algorithm 1, lines 9-15; Section 4.1 (benchmark evaluation)]
"Before each time a layer is to be dropped, a small subset of the fine-tuning dataset's validation set is sampled as the calibration dataset. For each layer, its importance score is the reciprocal of the model's performance after dropping the layer. ... Choose which layer to drop with index m s.t. s_m = min(U)."

The greedy layer-removal sequence is selected by exactly the quantity the paper later headlines: accuracy on the target-domain benchmark. The calibration samples are drawn from the validation set of the fine-tuning data, and Section 4.1 reports final accuracy via lm-evaluation-harness without ever stating that the calibration samples are excluded from those reported numbers. Hence the 'nearly no loss in accuracy at 50% compression' result is partly an artifact of optimizing the reported metric on the reported evaluation distribution; the layer-removal path is fitted, not independently predicted.
fitted input called prediction [Section 4.1, Tables 3 and 6; Abstract and Table 2 use the selected r=1/4 strategy]
"For each of the three methods, we evaluate specialized models performance when they are trained with different sparse update ratio r = {1, 1/2, 1/4, 1/8}. As we can see in Table 3, in comparison with other target selection techniques, we find at a sparse update ratio of r = 1/4, the model performs the best."

The sparse-update ratio r=1/4 is chosen by comparing accuracy on the same SciQ, MedMCQA, LexGLUE, and FinanceQA columns that later produce the abstract's 'no loss in accuracy at 50-60% compression' claim. Table 3 is run directly on those benchmarks, and Table 2 then presents r=1/4 as TRIMLLM's best strategy. With no held-out split or error bars, the headline accuracy is a model-selection number rather than an independent evaluation of the method. The speedup and memory measurements are not affected by this circularity, but the accuracy component of the central claim is partially fitted to the evaluation set.

full rationale

The paper does not rely on a self-citation chain; citations to prior work are external, and the speedup, memory-usage, and hardware-independence results are self-contained measured quantities. The circularity burden is confined to the accuracy dimension. Layer-importance scores are computed by deleting a layer and measuring accuracy on a calibration subset of the fine-tuning validation set (Eq. 3), and Algorithm 1 minimizes exactly this score at every removal step. Additionally, the sparse-update ratio r=1/4 is selected by running the method on the same benchmarks used for final reporting. The paper never states that the calibration subset is disjoint from the evaluation examples used in Table 2 and Figure 1, nor does it report error bars or a held-out split for hyperparameter choice. Consequently, the 'no loss in accuracy' claim is partly fitted to the evaluation distribution and is not yet established as an independent prediction. With an explicit disjoint calibration/evaluation protocol, the remaining claims would be substantially independent; as written, the evaluation protocol supports a partial-circularity finding of 5.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim relies on the assumed existence of a compact layer subset (Hypothesis 1), the stability of importance scores under fine-tuning, and several heuristic approximations. There are no new physical entities. The main tunable parameter is the sparse-update ratio r=1/4, selected on the evaluation benchmarks.

free parameters (4)

sparse update ratio r = 1/4
Selected as the best from {1, 1/2, 1/4, 1/8} after evaluating on the same QA benchmarks (Tables 3 and 6).
delta in Eq. 3 = unspecified
A small positive constant in the calibration importance score; its value is never given, though the score is a monotonic transform of accuracy so ranking is insensitive.
thresholding function C (Ca or Ce) = not specified
The stopping rule for layer dropping is left abstract; no concrete accuracy or efficiency thresholds are defined.
calibration dataset size = not specified
Defined only as 'a small subset of the fine-tuning dataset's validation set'; size affects importance estimates.

assumptions (5)

domain assumption There exists a subset of attention and MLP layers that, after fine-tuning, approximates the full model's output distribution for a domain (Hypothesis 1).
Invoked in Section 3.1; the paper does not prove existence, it assumes the subset exists and then searches for it.
domain assumption Fine-tuning with layer dropping will not significantly disturb each layer's importance score.
Section 3.4; this justifies computing importance once and freezing most layers, and is essential to the sparse-update scheme.
domain assumption Activation tensors have low intrinsic dimension and are dense with small elements plus a few outliers; the Frobenius norm approximates the nuclear norm for identifying high-rank activations.
Section 3.3; used to justify activation-norm scoring.
domain assumption Gradual (one layer at a time) dropping preserves performance better than dropping many layers at once (Proposition 1).
Section 3.2; motivates progressive dropping, based on prior work and empirical observation.
domain assumption MLP layers are more task-specific and attention layers more general (knowledge localization from cited work).
Sections 2 and 3.1; used to interpret dropping patterns, though the method itself does not require this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs." pith.science (2026). https://pith.science/paper/STP76XEO

@misc{pith2026241211242,
  author       = {Pith},
  title        = {Pith review of: TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/STP76XEO}},
  note         = {Machine review of arXiv:2412.11242}
}

abstract

Specializing large language models (LLMs) for local deployment in domain-specific use cases is necessary for strong performance while meeting latency and privacy constraints. However, conventional task-specific adaptation approaches do not show simultaneous memory saving and inference speedup at deployment time. Practical compression techniques like quantization and pruning require dedicated hardware or kernel support to achieve measured inference speedup. We develop TrimLLM based on the layer-wise specialization phenomenon we empirically observed and verified on contemporary LLMs. TrimLLM reduces the depth of LLMs via progressive layer dropping. We show it retains LLMs' capacity in specific domains and achieves inference speedup irrespective of hardware and deep learning frameworks. We evaluated TrimLLM on LLMs of various sizes for inference; models adapted on medical, legal, and financial datasets all demonstrate $2.1-5.7\times$ inference speedup on consumer GPUs and up to $3.1\times$ speedup on A100 when compared to state-of-the-art model compression algorithms, with no loss in accuracy at 50$\sim$60\% model compression ratio.

Figures

Figures reproduced from arXiv: 2412.11242 by the authors.

**Figure 2.** The Pareto Frontier of LLaMA-7B-TRIMLLM on SciQ and MedMCQA. TRIMLLM has a much wider spectrum of operating points to fit the model into different hardware with competitive performance. The layer dropping strategy employed is with calibration scanning and activation-norm tie breaker and + sparse udpate at r = 1 4 . on the premise that some layers carry less information for a task and can be discarded. However, fine… view at source ↗

**Figure 3.** Layer dropping patterns when TRIMLLM (calibration + activation-norm tie breaker) is applied to LLaMA7B on QA benchmarks. Results for the first 32 iterations are shown. At this point, the model has been reduced to one half of its original size with nearly no performance loss, evidenced in [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: An illustration of fine-tuning time complexity for different combinations of sparse fine-tuning schemes [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 9 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255

arXiv 2020
[4]

Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://arxiv.org/abs/2401.15024 Slicegpt: Compress large language models by deleting rows and columns . Preprint, arXiv:2401.15024

arXiv 2024
[5]

Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734

arXiv 2023
[6]

Gaurang Bharti. 2023. https://huggingface.co/datasets/gbharti/finance-alpaca gbharti/finance-alpaca . Accessed: 2023-09-20

work page 2023
[7]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

2020
[8]

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2021. Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint arXiv:2110.00976

arXiv 2021

Show all 47 references

[9]

Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092

2023 arXiv
[10]

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339

2022 arXiv
[11]

Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078

2023 arXiv
[12]

Elias Frantar and Dan Alistarh. 2023 a . Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774

2023 arXiv
[13]

Elias Frantar and Dan Alistarh. 2023 b . Sparsegpt: Massive language models can be accurately pruned in one-shot

2023
[14]

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323

2022 arXiv
[15]

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913

2020 arXiv
[16]

Neel Guha, Julian Nyarko, Daniel E Ho, Christopher R \'e , Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. arX...

2023 arXiv
[17]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[18]

Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023 a . Lawyer llama. https://github.com/AndrewZhe/lawyer-llama

2023
[19]

Quzhe Huang, Mingxu Tao, Zhenwei An, Chen Zhang, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023 b . Lawyer llama technical report. ArXiv, abs/2305.15062

2023 arXiv
[20]

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146

2019 arXiv
[21]

Matt Gardner Johannes Welbl, Nelson F. Liu. 2017. Crowdsourcing multiple choice science questions

2017
[22]

Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101--24116

2022
[23]

Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. 2022. Surgical fine-tuning improves adaptation to distribution shifts. arXiv preprint arXiv:2210.11466

2022 arXiv
[24]

Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2023. Pmet: Precise model editing in a transformer. arXiv preprint arXiv:2308.08742

2023 arXiv
[25]

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978

2023 arXiv
[26]

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. 2023. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, pages 22137--2...

2023
[27]

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627

2023 arXiv
[28]

Macedo Maia, André Freitas, and Alexandra et al. Balahur. 2023. https://sites.google.com/view/fiqa/home fiqa . Accessed: 2023-09-20

2023
[29]

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022 a . Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359--17372

2022
[30]

Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022 b . Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229

2022 arXiv
[31]

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on Health, Inference, and Learning, pages 248--260. PMLR

2022
[32]

Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 77:101429

2023
[33]

Nigam H Shah, David Entwistle, and Michael A Pfeffer. 2023. Creation and adoption of large language models in medicine. JAMA, 330(9):866--869

2023
[34]

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155

2018 arXiv
[35]

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695

2023 arXiv
[36]

Aaquib Syed, Phillip Huang Guo, and Vijaykaarti Sundarapandiyan. 2023. Prune and tune: Improving efficient pruning techniques for massive language models

2023
[37]

Hashimoto

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

2023
[38]

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, pages 1--11

2023
[39]

Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023 a . Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454

2023 arXiv
[40]

Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023 b . Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564

2023 arXiv
[41]

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR

2023
[42]

Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031

2023
[43]

Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Wei Lin, et al. 2023. Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:2309.11325

2023 arXiv
[44]

Minjia Zhang and Yuxiong He. 2020. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33:14011--14023

2020
[45]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[46]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[47]

economics

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255

arXiv 2020

[4] [4]

Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. https://arxiv.org/abs/2401.15024 Slicegpt: Compress large language models by deleting rows and columns . Preprint, arXiv:2401.15024

arXiv 2024

[5] [5]

Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734

arXiv 2023

[6] [6]

Gaurang Bharti. 2023. https://huggingface.co/datasets/gbharti/finance-alpaca gbharti/finance-alpaca . Accessed: 2023-09-20

work page 2023

[7] [7]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

2020

[8] [8]

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2021. Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint arXiv:2110.00976

arXiv 2021