Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Improving Large Language Models with Concept-Aware Fine-Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Concept-aware fine-tuning: predict several tokens at once and beat next-token baselines across coding, math, clinical summaries, molecules, and proteins.

desk verdict Clean evidence on coding and math, but a real confound on three of five tasks; worth refereeing after controlled baselines and a corrected loss formula. read the letter →

arxiv 2506.07833 v2 pith:Q6S3N2LF submitted 2025-06-09 cs.LG cs.AIcs.CL

Michael K. Chen , Xikun Zhang , Jiaxing Huang , Dacheng Tao This is my paper
classification cs.LGcs.AIcs.CL
keywords concept-awarefine-tuningmulti-tokenpredictionpost-trainingauxiliaryheadsLoRAnext-tokenLLMdenovoproteindesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the conventional next-token fine-tuning objective leaves large language models with a fragmented, token-by-token grasp of multi-token concepts, and that this bottleneck can be removed during post-training. It introduces Concept-Aware Fine-Tuning (CAFT), which adds auxiliary heads that predict the next several tokens in parallel, a weighted auxiliary loss, and pretrains those heads on a broad, self-distilled instruction mixture. Across coding, mathematics, clinical summarization, molecular generation, and de novo protein design, CAFT consistently outperforms next-token full and LoRA fine-tuning, with gains such as HumanEval pass@1 rising from 40.5% to 49.3%. The auxiliary heads are discarded after fine-tuning, so inference is unchanged, meaning the method can be dropped into existing fine-tuning pipelines with only a few extra lines of code.

What carries the argument

The central object is the set of auxiliary heads $F_{h_k}$ for $k=2,\ldots,n$, each an independent hidden layer initialized from the last hidden layer of the base model and sharing the original unembedding layer. These heads predict the $k$-th future token given the same hidden state, and their losses are summed with a geometric decay $\alpha^{k-2}$ during head pretraining and with an additional weighting $\beta\gamma$ during CAFT, where $\gamma$ follows a reflected-sine schedule. The auxiliary heads are trained once on a 100,000-sample instruction mixture, then either frozen or briefly fine-tuned on the task, and finally discarded so that only the original model remains at inference.

What would settle it

Run CAFT on the summarization, molecular generation, and protein design tasks without the extra one-epoch task-specific head pretraining, or give the next-token baseline the same extra training; if the reported improvements shrink to within noise, the claimed benefit of the multi-token objective itself is not established.

Watch

Extended reading notes

Core claim

CAFT establishes that multi-token prediction, previously limited to the expensive pretraining phase and reported to fail when naively applied to fine-tuning, can be made effective in post-training. The key is to prepare the auxiliary heads before fine-tuning: train them on a general instruction-tuning dataset whose responses are self-distilled from the base model, so the heads produce reliable gradient signal on arbitrary downstream tasks, and then scale down the auxiliary losses with a small constant and a decaying schedule so the primary next-token loss remains the optimization priority. The authors report that CAFT improves performance over next-token fine-tuning in all five studied tasks, that LoRA-based CAFT often beats full next-token fine-tuning, and that the gains are especially large when multi-token concepts are prevalent, as in code and molecular structures.

Load-bearing premise

CAFT's gains rely on the auxiliary heads, pretrained on a general self-distilled instruction mixture, providing useful gradient signal for whichever downstream task the model is later fine-tuned on; for three of the five tasks the heads also receive one extra epoch of task-specific training that the next-token baseline does not get.

Editorial extensions

If this is right

  • CAFT can be added to standard full or LoRA fine-tuning pipelines with a few lines of code, yielding consistent gains over next-token fine-tuning across diverse tasks.
  • Because the auxiliary heads are removed after training, CAFT improves downstream performance without any additional inference-time compute or code changes.
  • The method extends to non-natural-language outputs such as SMILES strings and protein sequences, suggesting multi-token objectives are useful beyond conventional text generation.
  • LoRA-based CAFT often outperforms full next-token fine-tuning, implying that the multi-token objective itself, not parameter count, drives much of the improvement.
  • CAFT reduces the model's reliance on tokenizer quality, since it can recover multi-token concepts even when the tokenizer splits them into arbitrary fragments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the CAFT gains replicate across more base models and task families, multi-token objectives could become a default component of supervised fine-tuning, making next-token-only post-training look like a missed opportunity.
  • A direct test of the causal story would be to probe hidden states for multi-token concept encodings before and after CAFT, going beyond the paper's proxy-based ablation.
  • The one-epoch task-specific head pretraining used in three of the five tasks is a plausible confound; isolating the contribution of the multi-token loss versus this extra training would sharpen the claim.
  • CAFT's auxiliary heads are pretrained on one model family, so an open question is whether such task-agnostic heads transfer across architectures and tokenizers without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Concept-Aware Fine-Tuning (CAFT), a method for fine-tuning next-token LLMs with auxiliary heads that predict up to n future tokens. The auxiliary heads are first trained on a self-distilled instruction mixture, then during task-specific fine-tuning the base model is trained with a weighted combination of the next-token loss and auxiliary losses. The authors report large gains over conventional next-token full and LoRA fine-tuning on five tasks: HumanEval code generation, MATH-500 mathematics, MIMIC-IV-BHC clinical summarization, L+M-24 molecular generation, and Mol-Instructions de novo protein design. They also provide an analysis linking CAFT's gains to better generation of multi-token concepts.

Significance. If the reported gains are attributable to the multi-token objective, CAFT would be a practically valuable addition to standard SFT pipelines: it requires only a few lines of code, does not change inference, and is far cheaper than multi-token pretraining. The authors provide a clean, fully specified method and promising evidence in the coding and mathematics tasks, where the auxiliary heads are task-agnostic and the next-token baseline receives the same training budget. The open-source release and the attempt to analyze concept-level effects are also strengths. However, the evidence for three of the five downstream tasks is currently confounded by an extra epoch of task-data training given only to the CAFT auxiliary heads, and the loss equation in Section 2.3 is internally inconsistent. These issues must be resolved before the central claim can be accepted.

major comments (3)
  1. [Section 3.2 / C.2] For Tasks 3-5 (BHC summarization, molecular generation, protein design), the CAFT auxiliary heads are additionally trained for 1 epoch on the task's own training set before the main fine-tuning, while the next-token baselines receive no analogous warm-start. This is stated in Section 3.2 ('From Task 3 onwards, the auxiliary heads are pretrained on the training set for 1 epoch') and in Section C.2. As a result, the reported CAFT gains in Tables 3-5 could stem from the extra task-data exposure, from the additional trainable parameters in the heads, or from interactions between the two, rather than from the multi-token objective per se. Please provide a control baseline that receives the same extra training (e.g., train auxiliary heads for 1 epoch on the task data, then discard them and fine-tune the base model with the standard next-token loss), or restrict the claim of multi-token benefit to Tasks 1-2 where the comparison is clean.
  2. [Section 2.3, Eq. (4)] Equation (4) writes L_n = sum_{k=1}^n -alpha^{k-1} beta gamma log p_{t+k}(y_{t+k}), which applies beta and gamma to the k=1 next-token loss and uses alpha^{k-1} rather than alpha^{k-2}. The text states that beta adjusts 'the weightage of all auxiliary head losses' and that only auxiliary losses should be scaled, which contradicts the formula. Please specify the exact loss used in the experiments: if k=1 is not scaled, correct Equation (4); if it is scaled, revise the text. This is essential for reproducibility and for interpreting the relative contribution of the auxiliary heads.
  3. [Section C.3] The evaluation protocol reports that models are evaluated on earlier checkpoints and the top-performing checkpoint is selected for the final metrics. Because this selection uses the evaluation set itself, it can inflate reported differences, especially for small test sets such as HumanEval (164 examples). Please report results from the final checkpoint or from a checkpoint selected on a held-out validation split, in addition to the current top-checkpoint numbers, so that the magnitude of the reported gains can be assessed.
minor comments (5)
  1. [Section 3.2.3 / Table 3] The caption of Table 3 says 'CNN-DailyMail and WritingPrompts', but the text describes the task as MIMIC-IV-BHC clinical summarization. Please correct the caption to match the dataset actually used.
  2. [Section 2.2] The text refers to 'a full breakdown can be found in Table X'; the placeholder 'Table X' should be replaced with the actual table number (Table 6 in the appendix).
  3. [Section 3.3 and Figure 4] The text and Figure 4 caption use 'L+P-24' in one place while the dataset is introduced as 'L+M-24'. Please standardize the notation.
  4. [Section 2.4] The sentence 'monitor both L1 and Ln as defined in Equation 3 (without beta or gamma)' is ambiguous: Equation (3) is the auxiliary-head training loss, not the CAFT loss. Presumably the intended reference is Equation (4) with beta and gamma removed; please clarify.
  5. [Section 3.3] The phrase 'conclusive prove' should be 'conclusive proof'; small typographical fixes throughout would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CAFT's reported gains are empirical, and the task-specific auxiliary-head pretraining is a confound rather than a circular reduction.

full rationale

The paper's derivation is not circular by construction. CAFT adds auxiliary heads pretrained on a self-distilled instruction mixture, then fine-tunes the base model with a weighted multi-token cross-entropy loss (Eq. 4) using ground-truth task tokens; the auxiliary heads are discarded at inference. The reported improvements on HumanEval, MATH-500, MIMIC-IV-BHC, L+M-24, and protein design are measured against external benchmarks, not derived from the fitted values. The main caveat is methodological, not circular: from Task 3 onward (Sec. 3.2; Sec. C.2), the auxiliary heads are additionally trained for 1 epoch on each task's own training set ('for BHC summarization, molecular generation, and protein design, the auxiliary heads are specifically trained for 1 epoch before the main fine-tuning'), while next-token baselines receive no equivalent extra training or extra parameters. For Tables 3-5, the reported gain is therefore not a controlled test of the multi-token objective alone; it may reflect the extra task-data exposure, extra head capacity, or both. This is a validity threat, but it does not reduce the result to its inputs by construction, so it is not circularity. There is also a self-citation to the authors' JustLogic benchmark (Chen et al., 2025), but it appears only as a related-work example and is not load-bearing. Equation 4's notation appears inconsistent (k=1 term would be scaled by beta and gamma despite the text saying those apply only to auxiliary losses), but this obscures the method rather than making the derivation circular. The central empirical claim remains independent of these caveats.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the auxiliary heads transferring to new tasks, self-distillation being sufficient to avoid distribution shift, and the weighted auxiliary losses improving rather than degrading the primary head. These are not proven; they are assumptions validated only by the reported experiments, some of which are confounded by extra task-data head training. The hyperparameters alpha, beta, gamma, and n are chosen by search or heuristics.

free parameters (5)
  • n (number of future tokens) = 5
    Chosen as a balance between multi-token representation and compute; authors note other studies found four additional heads optimal (Section 3.1).
  • alpha (geometric decay) = 0.8
    Hyperparameter search result: alpha=0.8 preferred over 0.7 and 0.9 (Section C.2); scales auxiliary losses by distance.
  • beta (auxiliary loss weight) = 0.01
    Hyperparameter search: beta=0.01 preferred over 0.05 and 0.10; intended to keep auxiliary losses subordinate to L1 (Section 2.3, C.2).
  • gamma schedule = RSine: sin((1 - t/T) * pi / 2), decaying from 1 to 0
    Chosen as better than constant or sine schedules; controls auxiliary loss weight over iterations. Eq. 4 places gamma on the primary loss too, an inconsistency.
  • L2 reliability threshold = 4.0
    Heuristic in Section 2.4: if auxiliary head perplexity exceeds 4.0, heads are considered unreliable and task-specific head fine-tuning is recommended; no quantitative justification is given.
assumptions (4)
  • domain assumption Auxiliary heads trained on a general instruction-distilled mixture transfer to arbitrary downstream tasks.
    Section 2.2 claims task-agnostic heads; Section 2.3 later concedes tasks with unusual vocabularies need task-specific head training.
  • ad hoc to paper Self-distilling ground-truth responses from the original head prevents distribution shift in the auxiliary heads.
    Section 2.2: 'to match the output distribution of the 1st head, the dataset's ground truth responses are self-distilled from the original head'; no independent verification is provided.
  • domain assumption Future-token auxiliary losses provide gradients that improve the first-head loss L1 when weighted by alpha, beta, and gamma.
    Section 2.3: this is the mechanism behind CAFT; supported only by the reported experiments and hyperparameter tuning.
  • ad hoc to paper The proxy fragments chosen for concept analysis measure genuine conceptual understanding.
    Section 3.3: proxies for code and SMILES concepts were selected by qualitative analysis after results; not a validated concept measure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Large Language Models with Concept-Aware Fine-Tuning." pith.science (2026). https://pith.science/paper/Q6S3N2LF

@misc{pith2026250607833,
  author       = {Pith},
  title        = {Pith review of: Improving Large Language Models with Concept-Aware Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6S3N2LF}},
  note         = {Machine review of arXiv:2506.07833}
}
read the original abstract

Large language models (LLMs) have become the cornerstone of modern AI. However, the existing paradigm of next-token prediction fundamentally limits their ability to form coherent, high-level concepts, making it a critical barrier to human-like understanding and reasoning. Take the phrase "ribonucleic acid" as an example: an LLM will first decompose it into tokens, i.e., artificial text fragments ("rib", "on", ...), then learn each token sequentially, rather than grasping the phrase as a unified, coherent semantic entity. This fragmented representation hinders deeper conceptual understanding and, ultimately, the development of truly intelligent systems. In response, we introduce Concept-Aware Fine-Tuning (CAFT), a novel multi-token training method that redefines how LLMs are fine-tuned. By enabling the learning of sequences that span multiple tokens, this method fosters stronger concept-aware learning. Our experiments demonstrate significant improvements compared to conventional next-token finetuning methods across diverse tasks, including traditional applications like text summarization and domain-specific ones like de novo protein design. Multi-token prediction was previously only possible in the prohibitively expensive pretraining phase; CAFT, to our knowledge, is the first to bring the multi-token setting to the post-training phase, thus effectively democratizing its benefits for the broader community of practitioners and researchers. Finally, the unexpected effectiveness of our proposed method suggests wider implications for the machine learning research community. All code and data are available at https://github.com/michaelchen-lab/caft-llm

Figures

Figures reproduced from arXiv: 2506.07833 by the authors.

Figure 1
Figure 1. (a, b) Next-token vs. multi-token training. Language models are typically trained using the next-token objective (left), where each token is an artificial text fragment. At every forward pass, models are tasked to predict the next immediate token. However, in the multi-token setting (right), models are tasked to predict the next n tokens in parallel in each forward pass, thus facilitating conceptual understanding ac… view at source ↗
Figure 2
Figure 2. (a) Sample code implementation. Building on top of the industry-standard Transformers library (Wolf et al., 2020), researchers and practitioners can incorporate CAFT into their existing Transformers training pipelines with just a few lines of code using our open-source library caft. (b) Downstream Tasks. These tasks empirically underscore the effectiveness and broad applicability of CAFT. The examples are adapted fr… view at source ↗
Figure 3
Figure 3. Perplexities of auxiliary heads over four epochs. Note that head [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) Examples of CAFT’s concept-informed generation. A comparison of the ground truth, next-token fine-tuned model generation, and CAFT model generation for two questions from the HumanEval (top) and L+M-24 (bottom) datasets. The red boxes show the relevant proxy concep…
Figure 5
Figure 5. Figure 5: Scaling of training losses of auxiliary heads for iterations [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Found to Designed: Concepts as a Design Axis for Large Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Concepts should be an explicit design axis for LLMs, organized by pipeline stage and internal-vs-external origin, rather than recovered post-hoc.

Reference graph

Works this paper leans on

63 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    A dataset and benchmark for hospital course summarization with adapted large language models // Journal of the American Medical Informatics Association

    Aali Asad, Van Veen Dave, Arefeen Yamin Ishraq, Hom Jason, Bluethgen Christian, Reis Eduardo Pontes, Gatidis Sergios, Clifford Namuun, Daws Joseph, Tehrani Arash S, others . A dataset and benchmark for hospital course summarization with adapted large language models // Journal of the American Medical Informatics Association. 2025. 32, 3. 470--479

    work page 2025

  2. [2]

    Program synthesis with large language models // arXiv preprint arXiv:2108.07732

    Austin Jacob, Odena Augustus, Nye Maxwell, Bosma Maarten, Michalewski Henryk, Dohan David, Jiang Ellen, Cai Carrie, Terry Michael, Le Quoc, others . Program synthesis with large language models // arXiv preprint arXiv:2108.07732. 2021

    arXiv 2021

  3. [3]

    The pitfalls of next-token prediction // arXiv preprint arXiv:2403.06963

    Bachmann Gregor, Nagarajan Vaishnavh . The pitfalls of next-token prediction // arXiv preprint arXiv:2403.06963. 2024

    arXiv 2024

  4. [4]

    Large Concept Models: Language Modeling in a Sentence Representation Space // arXiv preprint arXiv:2412.08821

    Barrault Lo \" c, Duquenne Paul-Ambroise, Elbayad Maha, Kozhevnikov Artyom, Alastruey Belen, Andrews Pierre, Coria Mariano, Couairon Guillaume, Costa-juss \`a Marta R, Dale David, others . Large Concept Models: Language Modeling in a Sentence Representation Space // arXiv preprint arXiv:2412.08821. 2024

    arXiv 2024

  5. [5]

    Language models are few-shot learners // Advances in neural information processing systems

    Brown Tom, Mann Benjamin, Ryder Nick, Subbiah Melanie, Kaplan Jared D, Dhariwal Prafulla, Neelakantan Arvind, Shyam Pranav, Sastry Girish, Askell Amanda, others . Language models are few-shot learners // Advances in neural information processing systems. 2020. 33. 1877--1901

    work page 2020

  6. [6]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads // arXiv preprint arXiv:2401.10774

    Cai Tianle, Li Yuhong, Geng Zhengyang, Peng Hongwu, Lee Jason D, Chen Deming, Dao Tri . Medusa: Simple llm inference acceleration framework with multiple decoding heads // arXiv preprint arXiv:2401.10774. 2024

    arXiv 2024

  7. [7]

    Code Alpaca: An Instruction-following LLaMA model for code generation

    Chaudhary Sahil . Code Alpaca: An Instruction-following LLaMA model for code generation. 2023

    work page 2023

  8. [8]

    Evaluating large language models trained on code // arXiv preprint arXiv:2107.03374

    Chen Mark, Tworek Jerry, Jun Heewoo, Yuan Qiming, Pinto Henrique Ponde De Oliveira, Kaplan Jared, Edwards Harri, Burda Yuri, Joseph Nicholas, Brockman Greg, others . Evaluating large language models trained on code // arXiv preprint arXiv:2107.03374. 2021

    arXiv 2021

Show all 63 references
  1. [9]

    JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models // arXiv preprint arXiv:2501.14851

    Chen Michael K, Zhang Xikun, Tao Dacheng . JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models // arXiv preprint arXiv:2501.14851. 2025

    2025 arXiv

  2. [10]

    Training verifiers to solve math word problems // arXiv preprint arXiv:2110.14168

    Cobbe Karl, Kosaraju Vineet, Bavarian Mohammad, Chen Mark, Jun Heewoo, Kaiser Lukasz, Plappert Matthias, Tworek Jerry, Hilton Jacob, Nakano Reiichiro, others . Training verifiers to solve math word problems // arXiv preprint arXiv:2110.14168. 2021

    arXiv

  3. [11]

    Qlora: Efficient finetuning of quantized llms // Advances in neural information processing systems

    Dettmers Tim, Pagnoni Artidoro, Holtzman Ari, Zettlemoyer Luke . Qlora: Efficient finetuning of quantized llms // Advances in neural information processing systems. 2023. 36. 10088--10115

    2023

  4. [12]

    Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina . Bert: Pre-training of deep bidirectional transformers for language understanding // Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language ...

    2019

  5. [13]

    Faith and fate: Limits of transformers on compositionality // Advances in Neural Information Processing Systems

    Dziri Nouha, Lu Ximing, Sclar Melanie, Li Xiang Lorraine, Jiang Liwei, Lin Bill Yuchen, Welleck Sean, West Peter, Bhagavatula Chandra, Le Bras Ronan, others . Faith and fate: Limits of transformers on compositionality // Advances in Neural Information Processing Systems. 2023....

    2023

  6. [14]

    L+ m-24: Building a dataset for language+ molecules@ acl 2024 // arXiv preprint arXiv:2403.00791

    Edwards Carl, Wang Qingyun, Zhao Lawrence, Ji Heng . L+ m-24: Building a dataset for language+ molecules@ acl 2024 // arXiv preprint arXiv:2403.00791. 2024

    2024 arXiv

  7. [15]

    Mol-instructions: A large-scale biomolecular instruction dataset for large language models // arXiv preprint arXiv:2306.08018

    Fang Yin, Liang Xiaozhuan, Zhang Ningyu, Liu Kangwei, Huang Rui, Chen Zhuo, Fan Xiaohui, Chen Huajun . Mol-instructions: A large-scale biomolecular instruction dataset for large language models // arXiv preprint arXiv:2306.08018. 2023

    2023 arXiv

  8. [16]

    Bridging the data gap between children and large language models // Trends in Cognitive Sciences

    Frank Michael C . Bridging the data gap between children and large language models // Trends in Cognitive Sciences. 2023. 27, 11. 990--992

    2023

  9. [17]

    Better & faster large language models via multi-token prediction // arXiv preprint arXiv:2404.19737

    Gloeckle Fabian, Idrissi Badr Youbi, Rozi \`e re Baptiste, Lopez-Paz David, Synnaeve Gabriel . Better & faster large language models via multi-token prediction // arXiv preprint arXiv:2404.19737. 2024

    2024 arXiv

  10. [18]

    Unpacking tokenization: Evaluating text compression and its correlation with model performance // arXiv preprint arXiv:2403.06265

    Goldman Omer, Caciularu Avi, Eyal Matan, Cao Kris, Szpektor Idan, Tsarfaty Reut . Unpacking tokenization: Evaluating text compression and its correlation with model performance // arXiv preprint arXiv:2403.06265. 2024

    2024 arXiv

  11. [19]

    The llama 3 herd of models // arXiv preprint arXiv:2407.21783

    Grattafiori Aaron, Dubey Abhimanyu, Jauhri Abhinav, Pandey Abhinav, Kadian Abhishek, Al-Dahle Ahmad, Letman Aiesha, Mathur Akhil, Schelten Alan, Vaughan Alex, others . The llama 3 herd of models // arXiv preprint arXiv:2407.21783. 2024

    2024 arXiv

  12. [20]

    Training large language models to reason in a continuous latent space // arXiv preprint arXiv:2412.06769

    Hao Shibo, Sukhbaatar Sainbayar, Su DiJia, Li Xian, Hu Zhiting, Weston Jason, Tian Yuandong . Training large language models to reason in a continuous latent space // arXiv preprint arXiv:2412.06769. 2024

    2024 arXiv

  13. [21]

    Amino acid substitution matrices from protein blocks

    Henikoff Steven, Henikoff Jorja G . Amino acid substitution matrices from protein blocks. // Proceedings of the National Academy of Sciences. 1992. 89, 22. 10915--10919

    1992

  14. [22]

    Lora: Low-rank adaptation of large language models

    Hu Edward J, Shen Yelong, Wallis Phillip, Allen-Zhu Zeyuan, Li Yuanzhi, Wang Shean, Wang Lu, Chen Weizhu, others . Lora: Low-rank adaptation of large language models. // ICLR. 2022. 1, 2. 3

    2022

  15. [23]

    MIMIC-IV, a freely accessible electronic health record dataset // Scientific data

    Johnson Alistair EW, Bulgarelli Lucas, Shen Lu, Gayles Alvin, Shammout Ayad, Horng Steven, Pollard Tom J, Hao Sicheng, Moody Benjamin, Gow Brian, others . MIMIC-IV, a freely accessible electronic health record dataset // Scientific data. 2023. 10, 1. 1

    2023

  16. [24]

    Highly accurate protein structure prediction with AlphaFold // nature

    Jumper John, Evans Richard, Pritzel Alexander, Green Tim, Figurnov Michael, Ronneberger Olaf, Tunyasuvunakool Kathryn, Bates Russ, Z \' dek Augustin, Potapenko Anna, others . Highly accurate protein structure prediction with AlphaFold // nature. 2021. 596, 7873. 583--589

    2021

  17. [25]

    Concept bottleneck models // International conference on machine learning

    Koh Pang Wei, Nguyen Thao, Tang Yew Siang, Mussmann Stephen, Pierson Emma, Kim Been, Liang Percy . Concept bottleneck models // International conference on machine learning. 2020. 5338--5348

    2020

  18. [26]

    De novo protein design—From new structures to programmable functions // Cell

    Kortemme Tanja . De novo protein design—From new structures to programmable functions // Cell. 2024. 187, 3. 526--544

    2024

  19. [27]

    Attribute and simile classifiers for face verification // 2009 IEEE 12th international conference on computer vision

    Kumar Neeraj, Berg Alexander C, Belhumeur Peter N, Nayar Shree K . Attribute and simile classifiers for face verification // 2009 IEEE 12th international conference on computer vision. 2009. 365--372

    2009

  20. [28]

    Lattice-BERT: leveraging multi-granularity representations in Chinese pre-trained language models // arXiv preprint arXiv:2104.07204

    Lai Yuxuan, Liu Yijia, Feng Yansong, Huang Songfang, Zhao Dongyan . Lattice-BERT: leveraging multi-granularity representations in Chinese pre-trained language models // arXiv preprint arXiv:2104.07204. 2021

    2021 arXiv

  21. [29]

    T " ulu 3: Pushing frontiers in open language model post-training // arXiv preprint arXiv:2411.15124

    Lambert Nathan, Morrison Jacob, Pyatkin Valentina, Huang Shengyi, Ivison Hamish, Brahman Faeze, Miranda Lester James V, Liu Alisa, Dziri Nouha, Lyu Shane, others . T " ulu 3: Pushing frontiers in open language model post-training // arXiv preprint arXiv:2411.15124. 2024

    2024 arXiv

  22. [30]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions // Hugging Face repository

    Li Jia, Beeching Edward, Tunstall Lewis, Lipkin Ben, Soletskyi Roman, Huang Shengyi, Rasul Kashif, Yu Longhui, Jiang Albert Q, Shen Ziju, others . Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions // Hugging Face repo...

    2024

  23. [31]

    Pre-trained language models for text generation: A survey // ACM Computing Surveys

    Li Junyi, Tang Tianyi, Zhao Wayne Xin, Nie Jian-Yun, Wen Ji-Rong . Pre-trained language models for text generation: A survey // ACM Computing Surveys. 2024 b . 56, 9. 1--39

    2024

  24. [32]

    Table-gpt: Table-tuned gpt for diverse table tasks // arXiv preprint arXiv:2310.09263

    Li Peng, He Yeye, Yashar Dror, Cui Weiwei, Ge Song, Zhang Haidong, Fainman Danielle Rifinski, Zhang Dongmei, Chaudhuri Surajit . Table-gpt: Table-tuned gpt for diverse table tasks // arXiv preprint arXiv:2310.09263. 2023

    2023 arXiv

  25. [33]

    Let's verify step by step // The Twelfth International Conference on Learning Representations

    Lightman Hunter, Kosaraju Vineet, Burda Yuri, Edwards Harrison, Baker Bowen, Lee Teddy, Leike Jan, Schulman John, Sutskever Ilya, Cobbe Karl . Let's verify step by step // The Twelfth International Conference on Learning Representations. 2023

    2023

  26. [34]

    Lin Chin-Yew, Hovy Eduard . Automatic evaluation of summaries using n-gram co-occurrence statistics // Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics. 2003. 150--157

    2003

  27. [35]

    Ben, Zimmerman Sam, Rivoire Kelley, Conerly Thomas, Olah Chris, Batson Joshua

    Lindsey Jack, Gurnee Wes, Ameisen Emmanuel, Chen Brian, Pearce Adam, Turner Nicholas L., Citro Craig, Abrahams David, Carter Shan, Hosmer Basil, Marcus Jonathan, Sklar Michael, Templeton Adly, Bricken Trenton, McDougall Callum, Cunningham Hoagy, Henighan Thomas, Jermyn Adam, J...

    2025

  28. [36]

    Deepseek-v3 technical report // arXiv preprint arXiv:2412.19437

    Liu Aixin, Feng Bei, Xue Bing, Wang Bingxuan, Wu Bochao, Lu Chengda, Zhao Chenggang, Deng Chengqi, Zhang Chenyu, Ruan Chong, others . Deepseek-v3 technical report // arXiv preprint arXiv:2412.19437. 2024 a

    2024 arXiv

  29. [37]

    Superbpe: Space travel for language models // arXiv preprint arXiv:2503.13423

    Liu Alisa, Hayase Jonathan, Hofmann Valentin, Oh Sewoong, Smith Noah A, Choi Yejin . Superbpe: Space travel for language models // arXiv preprint arXiv:2503.13423. 2025

    2025 arXiv

  30. [38]

    Acemath: Advancing frontier math reasoning with post-training and reward modeling // arXiv preprint arXiv:2412.15084

    Liu Zihan, Chen Yang, Shoeybi Mohammad, Catanzaro Bryan, Ping Wei . Acemath: Advancing frontier math reasoning with post-training and reward modeling // arXiv preprint arXiv:2412.15084. 2024 b

    2024 arXiv

  31. [39]

    The flan collection: Designing data and methods for effective instruction tuning // International Conference on Machine Learning

    Longpre Shayne, Hou Le, Vu Tu, Webson Albert, Chung Hyung Won, Tay Yi, Zhou Denny, Le Quoc V, Zoph Barret, Wei Jason, others . The flan collection: Designing data and methods for effective instruction tuning // International Conference on Machine Learning. 2023. 22631--22648

    2023

  32. [40]

    WizardCoder: Empowering Code Large Language Models with Evol-Instruct

    Luo Ziyang, Xu Can, Zhao Pu, Sun Qingfeng, Geng Xiubo, Hu Wenxiang, Tao Chongyang, Ma Jing, Lin Qingwei, Jiang Daxin . WizardCoder: Empowering Code Large Language Models with Evol-Instruct. 2023

    2023

  33. [41]

    De novo molecular design and generative models // Drug discovery today

    Meyers Joshua, Fabian Benedek, Brown Nathan . De novo molecular design and generative models // Drug discovery today. 2021. 26, 11. 2707--2715

    2021

  34. [42]

    Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? Levenshtein distance, spell checker, hamming distance

    Miller Frederic P, Vandome Agnes F, McBrewster John . Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? Levenshtein distance, spell checker, hamming distance. 2009

    2009

  35. [43]

    ColabFold: making protein folding accessible to all // Nature methods

    Mirdita Milot, Sch \"u tze Konstantin, Moriwaki Yoshitaka, Heo Lim, Ovchinnikov Sergey, Steinegger Martin . ColabFold: making protein folding accessible to all // Nature methods. 2022. 19, 6. 679--682

    2022

  36. [44]

    A general method applicable to the search for similarities in the amino acid sequence of two proteins // Journal of molecular biology

    Needleman Saul B, Wunsch Christian D . A general method applicable to the search for similarities in the amino acid sequence of two proteins // Journal of molecular biology. 1970. 48, 3. 443--453

    1970

  37. [45]

    Training language models to follow instructions with human feedback // Advances in neural information processing systems

    Ouyang Long, Wu Jeffrey, Jiang Xu, Almeida Diogo, Wainwright Carroll, Mishkin Pamela, Zhang Chong, Agarwal Sandhini, Slama Katarina, Ray Alex, others . Training language models to follow instructions with human feedback // Advances in neural information processing systems. 202...

    2022

  38. [46]

    Improving language understanding by generative pre-training.(2018)

    Radford Alec, Narasimhan Karthik, Salimans Tim, Sutskever Ilya, others . Improving language understanding by generative pre-training.(2018). 2018

    2018

  39. [47]

    Twilight zone of protein sequence alignments // Protein engineering

    Rost Burkhard . Twilight zone of protein sequence alignments // Protein engineering. 1999. 12, 2. 85--94

    1999

  40. [48]

    Get Your Atoms in Order: An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm // Journal of chemical information and modeling

    Schneider Nadine, Sayle Roger A, Landrum Gregory A . Get Your Atoms in Order: An Open-Source Implementation of a Novel and Robust Molecular Canonicalization Algorithm // Journal of chemical information and modeling. 2015. 55, 10. 2111--2120

    2015

  41. [49]

    Neural machine translation of rare words with subword units // arXiv preprint arXiv:1508.07909

    Sennrich Rico, Haddow Barry, Birch Alexandra . Neural machine translation of rare words with subword units // arXiv preprint arXiv:1508.07909. 2015

    2015 arXiv

  42. [50]

    A mathematical theory of communication // The Bell system technical journal

    Shannon Claude E . A mathematical theory of communication // The Bell system technical journal. 1948. 27, 3. 379--423

    1948

  43. [51]

    Tokenization counts: the impact of tokenization on arithmetic in frontier llms // arXiv preprint arXiv:2402.14903

    Singh Aaditya K, Strouse DJ . Tokenization counts: the impact of tokenization on arithmetic in frontier llms // arXiv preprint arXiv:2402.14903. 2024

    2024 arXiv

  44. [52]

    Blockwise parallel decoding for deep autoregressive models // Advances in Neural Information Processing Systems

    Stern Mitchell, Shazeer Noam, Uszkoreit Jakob . Blockwise parallel decoding for deep autoregressive models // Advances in Neural Information Processing Systems. 2018. 31

    2018

  45. [53]

    Scaling laws with vocabulary: Larger models deserve larger vocabularies // arXiv preprint arXiv:2407.13623

    Tao Chaofan, Liu Qian, Dou Longxu, Muennighoff Niklas, Wan Zhongwei, Luo Ping, Lin Min, Wong Ngai . Scaling laws with vocabulary: Larger models deserve larger vocabularies // arXiv preprint arXiv:2407.13623. 2024

    2024 arXiv

  46. [54]

    OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data // arXiv preprint arXiv:2410.01560

    Toshniwal Shubham, Du Wei, Moshkov Ivan, Kisacanin Branislav, Ayrapetyan Alexan, Gitman Igor . OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data // arXiv preprint arXiv:2410.01560. 2024

    2024 arXiv

  47. [55]

    Attention is all you need // Advances in neural information processing systems

    Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N, Kaiser ukasz, Polosukhin Illia . Attention is all you need // Advances in neural information processing systems. 2017. 30

    2017

  48. [56]

    Sciriff: A resource to enhance language model instruction-following over scientific literature // arXiv preprint arXiv:2406.07835

    Wadden David, Shi Kejian, Morrison Jacob, Naik Aakanksha, Singh Shruti, Barzilay Nitzan, Lo Kyle, Hope Tom, Soldaini Luca, Shen Shannon Zejiang, others . Sciriff: A resource to enhance language model instruction-following over scientific literature // arXiv preprint arXiv:2406...

    2024

  49. [57]

    Transformers: State-of-the-art natural language processing // Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations

    Wolf Thomas, Debut Lysandre, Sanh Victor, Chaumond Julien, Delangue Clement, Moi Anthony, Cistac Pierric, Rault Tim, Louf R \'e mi, Funtowicz Morgan, others . Transformers: State-of-the-art natural language processing // Proceedings of the 2020 conference on empirical methods ...

    2020

  50. [58]

    WizardLM: Empowering large pre-trained language models to follow complex instructions // The Twelfth International Conference on Learning Representations

    Xu Can, Sun Qingfeng, Zheng Kai, Geng Xiubo, Zhao Pu, Feng Jiazhan, Tao Chongyang, Lin Qingwei, Jiang Daxin . WizardLM: Empowering large pre-trained language models to follow complex instructions // The Twelfth International Conference on Learning Representations. 2024

    2024

  51. [59]

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models // arXiv preprint arXiv:2309.12284

    Yu Longhui, Jiang Weisen, Shi Han, Yu Jincheng, Liu Zhengying, Zhang Yu, Kwok James T, Li Zhenguo, Weller Adrian, Liu Weiyang . MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models // arXiv preprint arXiv:2309.12284. 2023

    2023 arXiv

  52. [60]

    Scoring function for automated assessment of protein structure template quality // Proteins: Structure, Function, and Bioinformatics

    Zhang Yang, Skolnick Jeffrey . Scoring function for automated assessment of protein structure template quality // Proteins: Structure, Function, and Bioinformatics. 2004. 57, 4. 702--710

    2004

  53. [61]

    Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space // arXiv preprint arXiv:2505.15778

    Zhang Zhen, He Xuehai, Yan Weixiang, Shen Ao, Zhao Chenyang, Wang Shuohang, Shen Yelong, Wang Xin Eric . Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space // arXiv preprint arXiv:2505.15778. 2025

    2025 arXiv

  54. [62]

    Wildchat: 1m chatgpt interaction logs in the wild // arXiv preprint arXiv:2405.01470

    Zhao Wenting, Ren Xiang, Hessel Jack, Cardie Claire, Choi Yejin, Deng Yuntian . Wildchat: 1m chatgpt interaction logs in the wild // arXiv preprint arXiv:2405.01470. 2024

    2024 arXiv

  55. [63]

    P, Zhang Hao, Gonzalez Joseph E., Stoica Ion

    Zheng Lianmin, Chiang Wei-Lin, Sheng Ying, Zhuang Siyuan, Wu Zhanghao, Zhuang Yonghao, Lin Zi, Li Zhuohan, Li Dacheng, Xing Eric. P, Zhang Hao, Gonzalez Joseph E., Stoica Ion . Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.