Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 10,000-sample subset chosen by distribution matching and diversity beats fine-tuning on the full 92,000-sample pool for code generation.

desk verdict A clear, honest port of ActiveFT to code instruction data, with a practically appealing but statistically fragile headline result; deserves review, but the central claim needs seeds and a non-post-hoc budget. read the letter →

arxiv 2507.02378 v1 pith:S7CCKD6S submitted 2025-07-03 cs.CL

Weijie Lyu , Sheng-Jun Huang , Xuan Xia This is my paper
classification cs.CL
keywords dataselectioninstructiontuningcodegenerationparametricmodeldistributionconsistencydiversityfeaturespaceDeepSeek-Coder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data quantity is not what is driving code-model quality, the paper argues; selection is. The authors propose a parametric data-selection method that embeds every instruction-code pair into a feature space, then optimizes a small set of continuous parameter points so that they cover the full dataset's feature distribution while staying far apart from one another. The data samples closest to those optimized points form the training subset. On a mixed 92K-sample Python instruction pool, fine-tuning DeepSeek-Coder-Base-6.7B on just 10K selected samples reaches 69.5% on HumanEval and 77.2% on MBPP, beating the full-data run by 2.4 and 2.3 points respectively, and the selection itself takes about 13.5 minutes. If the result holds, cheap feature-space curation can replace both brute-force data scaling and expensive LLM-based scoring.

What carries the argument

The machinery is a parametric model $\theta_S = \{\theta_j^S\}_{j=1}^{m}$ of the selected subset: a set of $m$ trainable points confined to the unit sphere in the feature space of the sentence-embedding encoder all-mpnet-base-v2, where features are L2-normalized embeddings of the instruction text only. Optimizing the joint loss $L = M(p_{F_D}, p_{\theta_S}) - \lambda R(\theta_S)$ attracts each parameter toward the nearest cluster of data features via a max-cosine-similarity term and repels pairs of parameters via a log-sum-exp penalty on their similarity, so the fitted parameters act as diverse prototypes that cover the dataset's distribution. After optimization, each parameter votes for its single closest real sample (Equation 8), producing the final subset. The temperature $\tau = 0.07$ and 300 Adam iterations are carried over from the active-fine-tuning work this adapts.

What would settle it

Check whether the selection signal is functional: take the 10K subset, run each sample's code in a sandbox, and count how many compile or pass their own tests. If a large share fails yet the subset still beats full-data training, the gain comes from instruction-text geometry rather than code correctness, and swapping the encoder for a code-specific model or a random projection would reveal whether that geometry is semantic or merely distributional.

Watch

Extended reading notes

Core claim

The central discovery is that a 10K subset chosen purely by geometry in a text-embedding space can outperform the entire 92K instruction pool. The paper frames the selection as an optimization: find $m$ points $\theta_S$ on the unit hypersphere whose distribution best matches the dataset's feature distribution while the points repel each other, then pull out the one real sample nearest to each point (Equations 4-8). The loss is a distribution-matching term (each data feature should be close to some parameter) plus a diversity regularizer (parameters should be dissimilar to each other), balanced by $\lambda = 1$. Trained this way, the selected subset yields 69.5% Pass@1 on HumanEval and 77.2% on MBPP versus 67.1% and 74.9% for the full-data baseline, and the same selection outperforms random, complexity-based (PPL, IFD), quality-scored (DEITA, DQ), and coreset (K-Center) baselines at matching budgets while taking the least sampling time.

Load-bearing premise

The entire gain rests on the assumption that, for instruction-code pairs, closeness in the embedding space of a general-purpose sentence encoder is a reliable signal for how useful a sample is for teaching code generation; the paper does not verify that the chosen code is correct or that embedding similarity tracks downstream training value.

Editorial extensions

If this is right

  • A 10K-sample subset suffices to beat full-data fine-tuning on the same pool, so the 92K instruction set contains a small high-value core plus samples that add noise.
  • The method beats every compared sampler (random, DQ, DEITA, PPL, IFD, K-Center) at the 10K budget, and beats models trained on 20K-78K samples (WaveCoder, WizardCoder, Magicoder), so distribution-plus-diversity selection transfers across training pools.
  • Selection time is 13.5 minutes versus hours for scoring-based and binning-based methods, so the approach scales to larger pools without per-sample forward passes of a large model.
  • Because the encoder processes instruction text only, which the paper argues fully defines task semantics, the pipeline does not depend on code-specific structure or a code-trained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If embedding proximity tracks training value, the same selection should transfer to other programming languages and to larger base models; the paper lists both as untested, so a Java or C++ run or a 34B-parameter run would be the direct confirmation.
  • The pool itself contains flawed code (the paper concedes no functional-correctness check), so the gains are achieved despite noise; filtering the selected 10K for executability before training could raise the ceiling further, or if it does not, would show the selection signal is already capturing something subtler than correctness.
  • A natural stress test is swapping the generic all-mpnet-base-v2 encoder for a code-aware model or a random projection: if the gains survive a random projection, the mechanism is distribution geometry rather than semantics; if they vanish, the specific semantic space is load-bearing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes a parametric-model-based data selection method for code instruction tuning. It embeds instruction texts with all-mpnet-base-v2, optimizes m continuous parameters so that the selected subset matches the pool's feature distribution while maximizing pairwise diversity (Equation 7), and then selects the real sample nearest to each optimized parameter. Experiments on DeepSeek-Coder-Base-6.7B over a merged 92K Python instruction set report that training on 10K selected samples improves HumanEval by 2.4 points and MBPP by 2.3 points over full-data training, and that the selection step is faster than the compared baselines. The paper also compares against published code LLMs and candidly lists limitations, including the absence of code-correctness verification and the restriction to Python and one model size.

Significance. If the headline result is reproducible, the paper makes a useful practical contribution: a cheap, scalable selection procedure that turns a 10K subset into a better training set than the full 92K pool on two standard code benchmarks, with code released. The derivation is not circular in the problematic sense: Equation 7 uses only instruction embeddings and the geometry of the pool, never HumanEval or MBPP labels, and the citation to Lv et al. (2025) is for Dynamic Pack, not for the selection mechanism. The contribution is, however, incremental relative to ActiveFT, and the empirical support is currently too thin to establish the central claim: all comparisons in Tables 1-2 and Figure 2 come from single unseeded runs, the 10K budget is selected after observing the peak in Figure 2, and no ablation isolates the distribution term from the diversity term. These issues are fixable with additional experiments and would materially strengthen the paper.

major comments (4)
  1. [Section 5.3, Table 1] All reported numbers come from single unseeded runs, so the headline differences of 2.4 points on HumanEval (4 of 164 problems) and 2.3 points on MBPP are within the seed-to-seed variation typically observed for instruction tuning of a 6.7B code model. The paper should report means and variances over at least three seeds for the key comparisons (Ours versus full-data training, and Ours versus Random and DEITA), or provide a paired significance test; without this, the central claim that 10K selected samples beat 92K full data is not statistically established.
  2. [Figure 2, Section 5.3] The 10K budget is not prespecified; the paper selects the maximum of a noisy performance curve over 5K, 10K, 15K, 20K, 25K, and 92K. This post hoc selection of the best-performing point inflates the apparent effect even if the true differences are zero. The authors should either fix the budget before running the comparison or report independent replications at every sampled quantity and show that the benefit persists away from the chosen peak.
  3. [Section 3.2, Equation 7] No ablation separates the distribution-matching term M from the diversity regularizer R; lambda is fixed at 1 with no sensitivity analysis. Without a distribution-only variant, a diversity-only variant, and a lambda sweep, the paper cannot attribute the observed gain to the combination of distribution consistency and diversity that is claimed in the title and abstract. Adding such ablations is necessary to support the mechanism.
  4. [Section 3.3, Limitations] The selection uses only instruction-text embeddings from a general sentence encoder and does not verify code correctness, as the Limitations section itself acknowledges. Because the selected samples may contain flawed code, the abstract's phrase 'guaranteeing high-quality data' is not supported. The paper should quantify how many selected samples execute successfully or report a small-scale manual inspection of the selected subset.
minor comments (6)
  1. [Section 4.4] The word 'eigth' should be 'eight'.
  2. [Section 4.3, Equation 9] In the sentence defining Pass@k, 'denoted as k' is inconsistent with the formula, where c is the number of correct samples; the text should refer to c.
  3. [Table 2] The comparison against published code LLMs mixes training setups: Magicoder-DS numbers are taken from the original paper, WizardCoder uses a different base model, and token budgets and training schedules are not controlled. The claim that the method outperforms these models with 10K data should be caveated accordingly.
  4. [Abstract] The code link is given as 'here' without an actual URL or repository identifier, which prevents readers from accessing the released code.
  5. [Section 4.1] The paper does not report how many OSS-Instruct entries remained after filtering for Python, nor the final class balance across Evol-Instruct-Python-26K, CodeExercise-Python-27K, and OSS-Instruct; this information is needed for reproducibility.
  6. [References] Several reference entries contain 'V olume' instead of 'Volume' and similar spacing artifacts; these should be cleaned up.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the selection objective is label-free, so the headline gain is an independent empirical result; only a minor non-load-bearing self-citation keeps the score just above zero.

full rationale

The derivation chain is not circular. The selection method (Section 3.2, Eqs. 3-8) optimizes a parametric model against a loss that depends only on instruction-text embeddings of the pool: M(p_FD, p_thetaS) matches the selected features to the full dataset's feature distribution, and R(thetaS) enforces diversity. Neither term, nor the sample retrieval in Eq. 8, uses HumanEval or MBPP labels, so the reported 69.5%/77.2% versus 67.1%/74.9% results (Section 5.3) are not forced by construction. The self-citation to Lv et al. (2025) appears only in Section 4.4 to justify Dynamic Pack for training efficiency; it does not support the data-selection claim. The 10K budget is indeed the empirical peak of the Figure 2 sweep rather than a prespecified quantity, which is a statistical risk (selecting the best of several unseeded runs), but it is not a circularity because the budget was not fitted inside the selection objective and no fitted parameter is relabeled as a prediction. The Limitations section's admission that functional correctness is not validated weakens external validity but does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on empirical assumptions about the feature space and the transfer of ActiveFT's objective to code data. No new entities are postulated. The main free parameters are inherited from ActiveFT or chosen post hoc from the performance curve.

free parameters (4)
  • Selection budget m = 10,000
    The choice of 10K is made after inspecting the performance-versus-size curve in Figure 2, where it is the observed peak. The headline claim of beating the 92K baseline depends on this budget.
  • Temperature tau = 0.07
    Adopted from ActiveFT (Xie et al., 2023), Section 4.4. It scales the similarity in Equations 4-7 and therefore changes which samples are selected.
  • Scaling factor lambda = 1
    Set to 1 by default (Section 3.2) to weight diversity against distribution matching; not tuned for code data.
  • Optimization steps and learning rate = T=300, lr=0.001
    Same as ActiveFT (Section 4.4). The selected prototypes depend on this optimization budget.
assumptions (4)
  • domain assumption The all-mpnet-base-v2 embedding of instruction text captures semantic content relevant to code generation training utility.
    Section 3.2 maps each instruction I_i to f_i = E(I_i) and performs all selection in this space.
  • domain assumption Minimizing distribution distance between selected prototypes and the full pool, while maximizing prototype diversity, improves downstream fine-tuning performance.
    The loss in Equation 7 encodes this objective; no proof links it to final benchmark scores, and the link is assumed from ActiveFT's prior success.
  • domain assumption The post-filtered Mix-Python-92K pool is a sound training mixture whose full-data run is a fair baseline.
    Section 4.1 merges three public datasets; the exact remaining count after Python filtering is not stated, and potential benchmark contamination is not analyzed.
  • standard math Gradient descent on Equation 7 converges to a useful prototype set in the stated 300 iterations.
    The optimization is standard Adam with normalized parameters; convergence is assumed from the ActiveFT recipe.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection." pith.science (2026). https://pith.science/paper/S7CCKD6S

@misc{pith2026250702378,
  author       = {Pith},
  title        = {Pith review of: Efficient Code LLM Training via Distribution-Consistent and Diversity-Aware Data Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S7CCKD6S}},
  note         = {Machine review of arXiv:2507.02378}
}
read the original abstract

Recent advancements in large language models (LLMs) have significantly improved code generation and program comprehension, accelerating the evolution of software engineering. Current methods primarily enhance model performance by leveraging vast amounts of data, focusing on data quantity while often overlooking data quality, thereby reducing training efficiency. To address this, we introduce an approach that utilizes a parametric model for code data selection, aimed at improving both training efficiency and model performance. Our method optimizes the parametric model to ensure distribution consistency and diversity within the selected subset, guaranteeing high-quality data. Experimental results demonstrate that using only 10K samples, our method achieves gains of 2.4% (HumanEval) and 2.3% (MBPP) over 92K full-sampled baseline, outperforming other sampling approaches in both performance and efficiency. This underscores that our method effectively boosts model performance while significantly reducing computational costs.

Figures

Figures reproduced from arXiv: 2507.02378 by the authors.

Figure 1
Figure 1. Parametric Model Optimization Process: By optimizing the loss in Equation 7, each parameter θ j S is attracted to nearby sample features(gold in the figure, Equation 5) and repelled by other parameters θ k S , k ̸= j (green in the figure, Equation 6). two aspects, we aim to select a representative sub￾set that covers edge cases in the original dataset, thereby enhancing model performance. Compared to selecting in th… view at source ↗
Figure 2
Figure 2. Performance comparison of different sampling methods with varying sampling quantities. Our method [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L Leavitt, and Mansheej Paul. 2024. Perplexed by perplexity: Perplexity-based data pruning with small reference models. arXiv preprint arXiv:2405.20541

    arXiv 2024

  5. [5]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  6. [6]

    Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca

    2023

  7. [7]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and 1 others. 2024. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations

    2024

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

Show all 43 references
  1. [9]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  2. [10]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, and 1 others. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  3. [11]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  4. [12]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196

    2024 arXiv

  5. [13]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186

    2024 arXiv

  6. [14]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  7. [15]

    Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. 2021. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722--754. PMLR

    2021

  8. [16]

    Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023 a . Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991--52008

    2023

  9. [17]

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024 a . Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024

  10. [18]

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2024 b . From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. In Proceedings of the 2024 Conference of the No...

    2024

  11. [19]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, and 1 others. 2023 b . Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161

    2023 arXiv

  12. [20]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  13. [21]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558--21572

    2023

  14. [22]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024 b . What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  16. [24]

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2024. \# instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations

    2024

  17. [25]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations

    2023

  18. [26]

    Weijie Lv, Xuan Xia, and Sheng-Jun Huang. 2025. Data-efficient llm fine-tuning for code generation. arXiv preprint arXiv:2504.12687

    2025 arXiv

  19. [27]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  20. [28]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  21. [29]

    Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations

    2018

  22. [30]

    CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette-Choo, Jingyue Shen, Joe Kelley, and 1 others. 2024. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409

    2024 arXiv

  23. [31]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  24. [32]

    Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. 2024. Inscl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the As...

    2024

  25. [33]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lingui...

    2023

  26. [34]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652

    2021 arXiv

  27. [35]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering code generation with oss-instruct. In International Conference on Machine Learning, pages 52632--52657. PMLR

    2024

  28. [36]

    Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, pages 54104--54132

    2024

  29. [37]

    Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, and Wei Zhan. 2023. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23715--23724

    2023

  30. [38]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. https://openreview.net/forum?id=CfXh93NDgH Wizardlm: Empowering large pre-trained language models to follow complex instructions . In The Twelfth Internationa...

    2024

  31. [39]

    Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. 2023. Rethinking the instruction quality: Lift is what you need. arXiv preprint arXiv:2312.11508

    2023 arXiv

  32. [40]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  33. [41]

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2024. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Proceedings of the 62nd Annual Meeting of the Association for C...

    2024

  34. [42]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, and 1 others. 2023 a . Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006--55021

    2023

  35. [43]

    Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. 2023 b . Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17205--17216

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.