Pith. sign in

REVIEW 4 major objections 6 minor 47 references

DaMoC claims a twentyfold-cheaper way to choose which open LLM to fine-tune for a domain task.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DaMoC combines data filtering, token compression, and layer pruning to select the best LLM for domain fine-tuning, claiming ~20x faster training while preserving model rankings.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful cheap-rank heuristic, but the paper's universal claim is contradicted by its own Table 3 and Table 7; deserves review with substantial revision, not rejection. the 4 major comments →

arxiv 2509.01221 v2 pith:VFUXEGCS submitted 2025-09-01 cs.CL cs.AIcs.LG

DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression

Wei Huang , Huang Wei , Yinggui Wang This is my paper
classification cs.CL cs.AIcs.LG
keywords LLM fine-tuningmodel selectiondata filteringtoken compressionlayer pruningsparse mergingtask vectorranking preservation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fine-tuning a domain-specific model usually starts with a hard choice: among many open-weights LLMs, which one will train up best on your data? The paper proposes DaMoC, a pipeline that compresses both the training data and the model so that a full grid of fine-tuning runs is replaced by cheap proxy runs that preserve the relative ranking of candidate models. It reports that data filtering by diversity (GraphCut-style), token compression by perplexity with iterative rewriting, and pruning roughly a quarter of the layers via sparse merging together identify the same top model as full fine-tuning across medical, financial, general Q&A, and reading-comprehension benchmarks, at about 1/20 of the training time. The same pipeline also picks the better of full fine-tuning and LoRA fine-tuning. If the ranking-preservation claim holds, model selection becomes a routine pre-step rather than a full training campaign.

Core claim

DaMoC's central claim is that the optimal pre-trained LLM for a given fine-tuning task can be selected without fine-tuning all candidates fully. On the data side, twelve filtering methods are organized into distribution-aware, quality-aware, and hybrid paradigms, and the experiments conclude that distribution-aware methods (especially GraphCut) preserve inter-model ranking best, while quality-aware methods do worst and sampling below 10% destabilizes the ranking. On the model side, layer importance is measured by cosine similarity of input/output activations; layers whose input and output are most similar are pruned, and the pruned layer's parameters are subtracted from released base weights

What carries the argument

The load-bearing mechanism is sparse task-vector merging during layer pruning. For a layer judged unimportant by high input-output cosine similarity, the algorithm forms the difference between that layer's fine-tuned weights and the corresponding released base-model weights (a task vector), sparsifies it by keeping only entries above a magnitude threshold, and merges it into the preceding layer with coefficients set by the layer-importance scores. This merging is what preserves both accuracy and the relative ordering of models after pruning; without Wpre, the subtraction has nothing to work on, and the pruning step is unavailable.

Load-bearing premise

Every candidate model must have its original untuned base weights publicly released and must be fine-tuned from a chat/instruct version, because the pruning step subtracts those base weights from the fine-tuned weights; two of the eight tested models lack base releases and cannot be ranked at all, and base-model fine-tuning cannot use the pruning component.

What would settle it

Run the full DaMoC pipeline (GraphCut data filtering, token compression and rewriting, layer pruning with sparse merging) on a new domain dataset with a candidate pool that includes a model with no publicly released base weights, and compare the pipeline's top pick with the top model from full fine-tuning of every candidate. If the missing-base model would have been the best fine-tuned model, or if the compressed proxy's ranking disagrees with the full-fine-tuning ranking on any candidate, the central claim fails. A cheaper check: on the paper's own BillSum setup, compute whether ranking agree

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Model selection for domain fine-tuning can be done at a fraction of training cost, making it practical to evaluate more candidates or iterate on datasets.
  • The three-paradigm taxonomy gives practitioners a guideline: for ranking purposes, diversity-preserving data filters are safer than quality-only filters, and high filter ratios (>90%) should be avoided.
  • Token compression of roughly 50% with ~8% iterative rewriting retains enough signal to keep model rankings intact, suggesting compressed training data can be used for proxy runs.
  • The layer-pruning recipe transfers across architectures of different sizes (7B-9B, 28-48 layers), pruning about a quarter of layers without losing the ordering.
  • The full-vs-LoRA selection result extends the pipeline from 'which model' to 'which fine-tuning method,' at least for these two options.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ranking-preservation effect is likely driven by the stable relative ordering of model capacities under compression; a testable extension is whether the same proxy pipeline can rank hyperparameter configurations (learning rate, LoRA rank) beyond model identity.
  • Because quality-aware filters hurt ranking most, the authors' result implies that for model selection, data diversity is a better proxy signal than estimated sample quality; this reverses a common instinct to use quality scores for data curation when the goal is comparison.
  • A practical way to remove the Wpre bottleneck would be to construct a reference model by continued pretraining on generic text or to use a peer model's base weights; whether ranking survives such a substitution is an open, testable question.
  • The reported 20-fold saving covers the training phase only; the full cost including data selection, rewriting, and pruning calibration should be measured end-to-end before adopting the pipeline as a general substitute for grid search.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DaMoC, a data- and model-compression framework intended to select the best large language model for fine-tuning on domain tasks without running full fine-tuning for every candidate. At the data level it categorizes filtering methods into distribution-aware, quality-aware, and hybrid paradigms, and adds token compression via perplexity-based pruning with iterative rewriting; at the model level it prunes layers based on activation cosine similarity and merges pruned layers using a sparse task-vector formulation. Experiments on PubMedQA, BillSum, Alpaca, and SQuAD with eight open LLMs report roughly 20x training-time savings and claim that DaMoC selects the optimal LLM and, in a second analysis, the optimal fine-tuning method between full fine-tuning and LoRA. The authors also provide ablations and a limitation discussion centered on the need for released base-model weights.

Significance. If the claims were fully supported, the work would offer a practical and inexpensive way to choose a fine-tuning base model, a genuinely useful service for practitioners. The paper's strengths include a broad experimental matrix (multiple models, datasets, filtering methods, ablations), a transparent runtime measurement, and an explicit limitation statement. However, the headline claims are not consistent with the paper's own tables: DaMoC mis-selects in at least one main-table configuration, and the fine-tuning-method section contains a direct counterexample to the word 'consistently.' The contribution is therefore better described as a heuristic with measured successes and failures rather than a method that 'selects the optimal LLM.' The authors should narrow the claims, quantify the success rate, and explain the failure cases.

major comments (4)
  1. [§5.2, Table 3 (SQuAD, LoRA)] The abstract's unqualified claim 'can select the optimal LLM' is contradicted by Table 3. For SQuAD with LoRA, the GS column ranks GLM4 first (0.6685) and Gemma2 second (0.6669), while DaMoC-GraphCut ranks Gemma2 first (0.6574) and GLM4 second (0.6569). DaMoC therefore selects a suboptimal model in a reported configuration. Please either narrow the claim to 'usually selects' with an explicit success rate, or analyze this failure mode (e.g., score margins, sensitivity to compression/pruning).
  2. [§5.4, Table 7] The text claims DaMoC 'consistently identifies the optimal fine-tuning method across eight models and three datasets,' but Table 7 lists only six models (Phi-3-small and Ministral-8B are absent) and contains a counterexample: for Internlm2.5 on SQuAD, GS prefers Full fine-tuning (0.6586) over LoRA (0.6565), while DaMoC prefers LoRA (0.6488) over Full (0.6287). The 'consistently' claim is false as written, and the model count is inaccurate. Correct the count and qualify the conclusion.
  3. [§3.3, Eq. (4); Limitations] The sparse merging operation in Eq. (4) requires the pre-trained/base weights Wpre. The note under Table 3 and the Limitations state that Phi-3-small and Ministral-8B have not released base models, so DaMoC cannot rank them. Thus the method does not select the optimal LLM from an arbitrary pool of open models; it only applies when every candidate has publicly available Wpre. The abstract and introduction should state this scope restriction explicitly.
  4. [Appendix D; §C.2] The layer-pruning similarity threshold is set to 0.85 because Table 9 shows that 0.80 fails to recover the PubMedQA ranking; the same threshold is then applied to all datasets. The sparsity rate (20%) and BERTScore threshold (0.9) are fixed without sensitivity analysis. Since the target metric is rank preservation, choosing the threshold on PubMedQA and then reporting all-dataset success is a form of selection on the evaluation metric. Please provide a threshold sweep across datasets or a held-out justification for the chosen constants.
minor comments (6)
  1. [Table 13] The header column reads 'Duality-aware methods'; this should be 'Quality-aware methods.'
  2. [§5.1] The text says 'Distribution-aware methods consistently select the optimal pre-trained model. In contrast, the Distribution-aware method exhibits a higher number of incorrect selections.' The second occurrence should presumably be 'Quality-aware method'; as written, the two sentences are contradictory.
  3. [Table 12] For Internlm2.5 at 5% sampling with Full fine-tuning, the GS value 0.5261(7) is inconsistent with adjacent rows (0.5914 at 10% and typical values around 0.59). Please verify this entry.
  4. [Table 14] For GLM4 at 5% sampling with Full fine-tuning, the AG column shows 0.0493(7), which is implausibly low compared to the baseline 0.5927 and other methods. Please check whether this should be ~0.6049.
  5. [Appendix E] The text says 'Tables 12, 13, and 14 present ... on the PubMedQA, BillSum, and SQuAD datasets,' but Table 13 is Alpaca and BillSum is Table 2. The cross-references should be corrected.
  6. [§5.3] The 'approximately 20-fold' speedup is based on a single 3000-sample, one-epoch run. Please clarify whether the 0.72 minutes includes data compression, rewriting, and pruning overhead, and state the hardware/software configuration, so readers can assess the generality of the time claim.

Circularity Check

0 steps flagged

No significant circularity; DaMoC's compressed-training ranking is an independent surrogate, not a restatement of its inputs.

full rationale

The derivation chain is self-contained: DaMoC produces model rankings by actually fine-tuning data-compressed and layer-pruned models and comparing their validation accuracies. The ground-truth Grid Search rankings are used only as an evaluation benchmark, not as inputs to the loss, the pruning criterion, the sparse merging, or the compression objective. No load-bearing self-citation appears: the only 'Huang et al.' reference is to Prodigy (a different group), and the framework's components are drawn from external prior work (LLMLingua, GraphCut, etc.). The pruning threshold of 0.85 is selected via an ablation on PubMedQA (Appendix D), which is a mild hyperparameter-selection concern, but it does not reduce the prediction to the fit: the threshold alone does not determine the ranking, and the same threshold is then applied to other datasets. The Wpre dependency, the exclusion of Phi-3-small and Ministral-8B from pruning experiments, and the SQuAD/LoRA cases where DaMoC misranks models are correctness/robustness limitations explicitly acknowledged or visible in the tables, but they are not circularity. The method's central claim is an empirical surrogate-effectiveness claim, not a definitional equivalence.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on several empirical assumptions about rank preservation and on hyperparameters (pruning threshold, BERTScore threshold, sparsity rate) that are set with limited sensitivity analysis. No new physical entities are introduced.

free parameters (3)
  • Layer pruning similarity threshold = 0.85
    Tuned via Appendix D on PubMedQA: threshold 0.8 fails to select the optimal model, 0.85 and 0.90 work. This is a constant fitted to the evaluation outcome.
  • Sparsity rate for sparse merging = 20%
    Set in Section 3.3; no sensitivity analysis is provided.
  • BERTScore threshold for rewriting acceptance = 0.9
    Set in Section 3.2; no sensitivity analysis is provided.
axioms (5)
  • domain assumption Ranking of fine-tuned models is preserved under data filtering, token compression, and layer pruning.
    Core empirical premise; no theoretical argument is given, and it is contradicted by the SQuAD LoRA result in Table 3.
  • domain assumption Cosine similarity between layer input and output activations measures layer importance.
    Used in Eq. (3); similar to prior layer-pruning work, but no supporting citation is provided here.
  • domain assumption W_chat - W_base is a sparse 'task vector' that can be merged into neighboring layers to preserve capability.
    Eq. (4) relies on task-vector arithmetic; no citation or verification is given.
  • domain assumption BERTScore above 0.9 between original and compressed text is sufficient for fine-tuning quality.
    Section 3.2, threshold chosen without sensitivity analysis.
  • domain assumption Two random seeds and a small calibration set give stable model rankings.
    Section C.2; no variance or rank-stability analysis is reported.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression." pith.science (2026). https://pith.science/paper/VFUXEGCS

@misc{pith2026250901221,
  author       = {Pith},
  title        = {Pith review of: DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFUXEGCS}},
  note         = {Machine review of arXiv:2509.01221}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) excel in general tasks but struggle with domain-specific ones, requiring fine-tuning with specific data. With many open-source LLMs available, selecting the best model for fine-tuning downstream tasks is challenging, primarily focusing on how to quickly identify the optimal LLM. We introduce a Data and Model Compression Framework (DaMoC) that addresses this challenge by: 1) Data Level: A systematic categorization of data filtering methodologies for LLMs is first established, classifying them into three distinct paradigms: (1) distribution-aware methods, (2) quality-aware methods, and (3) hybrid approaches considering both dimensions. Further, we enhance the density of key tokens in the text achieving token compression. Subsequently, we use an LLM to iterative rewrite the text to optimize its expression. 2) Model Level: We use layer similarity scores to assess each layer's importance and remove those with lower importance. Then, we introduce a sparse merging paradigm to preserve as much of the original model's capability as possible. Extensive experiments on four datasets, medical Q&A, financial Q&A, general Q&A, and reading comprehension, show that we can select the optimal LLM while saving approximately 20-fold in training time.

Figures

Figures reproduced from arXiv: 2509.01221 by Huang Wei, Wei Huang, Yinggui Wang.

Figure 1
Figure 1. Figure 1: Illustration optimal model selection problem, the desired objectives, and methods for accelerating training. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Flowchart of the layer pruning algorithm. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    Pith/arXiv arXiv 2024

  2. [2]

    Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. https://doi.org/10.1609/aaai.v38i10.28960 Fluctuation-based adaptive structured pruning for large language models . Proceedings of the AAAI Conference on Artificial Intelligence, 38(10):10865--10873

    doi:10.1609/aaai.v38i10.28960 2024

  3. [3]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. arXiv preprint arXiv:2403.17297

    Pith/arXiv arXiv 2024

  4. [4]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701

    Pith/arXiv arXiv 2023

  5. [5]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2024. Alpagasus: Training a better alpaca with fewer data

    work page 2024

  6. [6]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36

    work page 2024

  7. [7]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    Pith/arXiv arXiv 2018

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, et al. 2024. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1107--1128

    work page 2024

  9. [9]

    Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653

    Pith/arXiv arXiv 2023

  10. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  11. [11]

    Yuan Ge, Yilun Liu, and Chi Hu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.28 Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 464--478, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.emnlp-main.28 2024

  12. [12]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    Pith/arXiv arXiv 2024

  13. [13]

    Qian Huang, Hongyu Ren, Peng Chen, Gregor Kr z manc, Daniel Zeng, Percy S Liang, and Jure Leskovec. 2024. Prodigy: Enabling in-context learning over graphs. Advances in Neural Information Processing Systems, 36

    work page 2024

  14. [14]

    Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. 2021. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722--754. PMLR

    work page 2021

  15. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    Pith/arXiv arXiv 2023

  16. [16]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023 b . Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736

    Pith/arXiv arXiv 2023

  17. [17]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023 c . Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839

    Pith/arXiv arXiv 2023

  18. [18]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567--2577

    2019

  19. [19]

    Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. 2024. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. Advances in Neural Information Processing Systems, 36

    work page 2024

  20. [20]

    Anastassia Kornilova and Vladimir Eidelman. 2019. Billsum: A corpus for automatic summarization of us legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 48--56

    work page 2019

  21. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611--626

    2023

  22. [22]

    Ming Li, Yong Zhang, and Shwai He. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.769 Superfiltering: Weak-to-strong data filtering for fast instruction-tuning . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14255--14273, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.acl-long.769 2024

  23. [23]

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024 b . From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    work page 2024

  24. [24]

    Siyi Liu, Chen Gao, and Yong Li. 2024 a . Large language model agent for hyper-parameter optimization. arXiv preprint arXiv:2402.01881

    Pith/arXiv arXiv 2024

  25. [25]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024 b . What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations

    work page 2024

  26. [26]

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  27. [27]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  28. [28]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/d16-1264 2016

  29. [29]

    Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations

    2018

  30. [30]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  31. [31]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    Pith/arXiv arXiv 2024

  32. [32]

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. 2023. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863

    Pith/arXiv arXiv 2023

  33. [33]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: Selecting influential data for targeted instruction tuning. In Forty-first International Conference on Machine Learning

    work page 2024

  34. [34]

    Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116

    Pith/arXiv arXiv 2024

  35. [35]

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

    Pith/arXiv arXiv 2023

  36. [36]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024 a . Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  37. [37]

    Songhua Yang, Hanjie Zhao, Senbin Zhu, Guangyu Zhou, Hongfei Xu, Yuxiang Jia, and Hongying Zan. 2024 b . Zhongjing: Enhancing the chinese medical capabilities of large language model through expert feedback and real-world multi-turn dialogue. In Proceedings of the AAAI Conference on Artificial Intelligence

    work page 2024

  38. [38]

    Yifei Yang, Zouying Cao, and Hai Zhao. 2024 c . Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

    Pith/arXiv arXiv 2024

  39. [39]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    Pith/arXiv arXiv 2024

  40. [40]

    Hengyuan Zhang and Yanru Wu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.445 Balancing speciality and versatility: a coarse to fine framework for supervised fine-tuning large language model . In Findings of the Association for Computational Linguistics: ACL 2024, pages 7467--7509, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-acl.445 2024

  41. [41]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    Pith/arXiv arXiv 2019

  42. [42]

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024 a . Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. In Forty-first International Conference on Machine Learning

    work page 2024

  43. [43]

    Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2024 b . https://arxiv.org/abs/2408.05517 Swift:a scalable lightweight infrastructure for fine-tuning . Preprint, arXiv:2408.05517

    Pith/arXiv arXiv 2024

  44. [44]

    Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. 2023. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17205--17216

    work page 2023

  45. [45]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633

    Pith/arXiv arXiv 2023

  46. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  47. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.