Pith. sign in

REVIEW 4 major objections 5 minor 19 references

LLaMA-Gene: A General-purpose Gene Task Large Language Model Based on Instruction Fine-tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Single gene chat model nears specialist accuracy

desk verdict Promising engineering recipe, but the abstract overclaims SOTA comparability and the paper's own Table 9 shows gaps on every task; worth peer review as a methods paper, not as a SOTA result. read the letter →

arxiv 2412.00471 v1 pith:RRGRGGVX submitted 2024-11-30 q-bio.GN

Wang Liang This is my paper · ORCID
classification q-bio.GN
keywords genelargelanguagemodelinstructionfine-tuningbytepairencodingDNAsequenceproteinmulti-tasklearningLLaMAgenomics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a single large language model can handle many gene-analysis tasks at once, the way ChatGPT handles many text tasks. To do this, the authors take a 7B- or 8B-parameter LLaMA model, expand its vocabulary with a unified byte-pair-encoding tokenizer trained on DNA and protein sequences, continue pretraining on genomic, proteomic, and biology-text data, and then convert five kinds of downstream gene tasks into instruction–response pairs for fine-tuning. They report that the resulting model, LLaMA-Gene, reaches accuracies on DNA classification, structure prediction, sequence-interaction, function prediction, and regression tasks that are close to the published numbers of specialized state-of-the-art models, with a larger gap on protein tasks. The point is that a ChatGPT-style dialogue model, rather than a collection of task-specific models, could serve as a general interface to genomic and proteomic analysis.

What carries the argument

The carrying mechanism is the unified vocabulary: a byte-pair encoding (BPE) tokenizer trained on genome fragments and protein sequences, whose 30,000 DNA tokens and 30,000 protein tokens are merged into LLaMA's original 32,000-token vocabulary to give roughly 91,000 tokens total. Because DNA, protein, and natural language share one tokenizer, no special markers like <protein>…</protein> are needed to tell the model what a sequence is. On top of that, LoRA is used to continue pretraining (about 10% of parameters) on a mix of genomes, proteomes, and biology papers, and then every downstream task is converted into Alpaca-style instruction templates and fine-tuned with the standard causal language-model objective, which is what lets one model answer many task types in dialogue.

What would settle it

Run LLaMA-Gene on the same held-out datasets used by DNABert2 or LucaOne and score it with exact-match accuracy (the output must equal the expected label, with no extra tokens). If exact-match accuracy falls well below both the reported semantic-match scores and the SOTA numbers, or if a majority-class baseline ties the model on any task, the claim of comparability to current state-of-the-art is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a single mixed-language model — natural language plus DNA plus protein in one BPE vocabulary — can be instruction-tuned to answer gene-task questions conversationally and still stay within a few accuracy points of specialized models on DNA tasks and several points behind on protein tasks. The authors claim this validates the feasibility of moving gene large language models from the 'GPT era' of per-task heads to the 'ChatGPT era' of a single model that handles classification, structure, interaction, function, and regression through prompts. They explicitly attribute the remaining protein-task gap to smaller training datasets and computational limits, not to a fundamental flaw in the approach.

Load-bearing premise

The comparison is valid: the published SOTA accuracy numbers were produced under their own evaluation protocols, while LLaMA-Gene's accuracy is scored under a permissive rule that counts an output like 'promoter AGCCGGG' as correct for the label 'promoter'.

Editorial extensions

If this is right

  • One instruction-tuned model replaces separate task-specific heads for DNA classification, structure prediction, interaction detection, function prediction, and regression, so adding a new task only requires a new prompt template and fine-tuning data.
  • The chat interface lets genomic analyses be driven by prompt engineering, retrieval-augmented generation, and agents built for natural-language models.
  • Because no special sequence markers are needed, models can exchange DNA and protein sequences directly in the same token space, simplifying multi-model collaboration.
  • The authors expect larger parameter scales and larger gene datasets to narrow the protein-task gap, since the current gap is attributed to training data size rather than the method.
  • Newer LLaMA versions did not change gene-task results, suggesting the improvements come mainly from the tokenizer and instruction scheme, not the base model's fine-tuning refinements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same unified-tokenizer-plus-instruction recipe could be carried over to other biomolecular modalities, such as RNA, single-cell expression profiles, or epigenetic marks, yielding one conversational model over a wider slice of biology.
  • The semantic-match scoring rule likely overstates the model's precision; an exact-match or token-F1 score would give a stricter measure of how reliably the model follows instructions, and would probably be a better guide for real users.
  • A head-to-head benchmark with identical train/test splits, labels, and scoring would settle whether the unified model genuinely matches specialized models or just comes near them, since cross-paper comparisons currently mix different protocols.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LLaMA-Gene, a 7B/8B LLaMA model whose vocabulary is expanded with BPE-based DNA and protein tokens, followed by continuous pre-training and instruction fine-tuning on a variety of gene tasks converted into a unified prompt format. The central claim, stated in the abstract, is that this mixed-language instruction-tuned model 'achieves results comparable to the current state-of-the-art (SOTA) in tasks such as gene classification and gene sequence interaction.' The paper reports accuracy values in Table 9 across classification, structure prediction, multiple sequence analysis, function prediction, and regression-style tasks for both DNA and protein sequences, and it argues that the instruction-tuning approach offers a ChatGPT-like general-purpose interface for gene tasks.

Significance. If the central claim were substantiated, the contribution would be significant: it would demonstrate that a single general-purpose language model can unify multiple gene task types in a dialogue format, advancing the field from task-specific fine-tuned models toward a unified gene LLM similar to ChatGPT. The design choices—unified BPE tokenization, conversion of diverse tasks into a single instruction format, and LoRA-based continuation of a 7B-scale LLaMA—are reasonable and potentially reusable. The listed public GitHub repository is a positive step for reproducibility. However, the current evidence is internally inconsistent and insufficient: the paper's own Table 9 shows LLaMA-Gene below the cited SOTA on every reported task, with gaps up to 0.24, and the evaluation protocol uses a permissive semantic-matching rule with no described train/test separation. The significance of the work therefore cannot be assessed from the manuscript as written.

major comments (4)
  1. [Abstract and Section 3.2, Table 9] The central claim of SOTA comparability is contradicted by the paper's own results. Table 9 shows LLaMA-Gene below the cited SOTA on all 11 rows, with gaps such as protein multiple sequence 0.63 vs 0.87, DNA+protein multiple sequence 0.71 vs 0.91, and protein classification 0.64 vs 0.72. Section 3.2 itself admits a 'noticeable gap in protein-related tasks.' Therefore the abstract's statement that the model achieves results 'comparable to the current state-of-the-art' is not supported by the reported numbers.
  2. [Section 3.1 and Table 9] The evaluation metric is not comparable across the compared systems. Section 3.1 states that a generated string such as 'promoter AGCCGGG' is counted as correct for the expected label 'promoter,' which is a permissive semantic-match rule that inflates accuracy relative to the exact-match or task-specific metrics typically used by fixed-head classification models. The SOTA values in Table 9 are external and are not accompanied by their evaluation protocols, dataset splits, or even citations to the specific SOTA methods. Without a shared protocol, the comparison in Table 9 cannot support the abstract's SOTA-comparability claim.
  3. [Section 2.2.2/Table 1 and Section 3.2] The paper does not establish that the evaluation data are disjoint from the instruction fine-tuning data. Table 1 lists 'lucaone/convert' as instruction fine-tuning data, and Section 3.2 states that evaluation datasets 'primarily referenced include those from DNABert2, lucaone, InstructionProtein, and BiomedGPT.' No train/test separation, sequence-level deduplication, or task-level split is described. This creates a serious risk that the reported accuracies partly reflect memorization of training instructions rather than generalization, which would invalidate the numerical claims even if the protocol issues were resolved.
  4. [Section 2.2.2.5 and Table 9] The regression tasks are converted to binary classification using a fixed threshold of 0.5, and Table 9 reports accuracy on these binary labels. State-of-the-art regression models are normally evaluated on the original continuous values with metrics such as correlation or RMSE. The reported 'regressiontask' accuracy is therefore not directly comparable to the cited SOTA numbers, further undermining the abstract's general claim of SOTA comparability for these rows.
minor comments (5)
  1. [Section 2.2.2.1] In the example instruction, 'Determine core promoter detection of following dna sequence' is missing an article and uses lowercase 'dna'; for consistency with the rest of the paper, it should read 'Determine the core promoter detection of the following DNA sequence.'
  2. [Section 3.1] The text 'WeUseAccuracy(ACC)asthePrimaryEvaluationMetric' lacks spaces and should be 'We use accuracy (ACC) as the primary evaluation metric.'
  3. [Tables 7 and 8] Table 7 is titled 'generegressiontask' and Table 8 is titled 'gene function prediction task,' but both tables describe the same gene expression level prediction task; the table titles should be consistent and accurate.
  4. [Section 3.2] The paper states that 'The evaluation involves 2 to 3 specific datasets per task' but does not name these datasets, report sample sizes, or provide error bars or confidence intervals, making it impossible to assess the statistical reliability of the reported accuracies.
  5. [References] The reference list is inconsistently formatted: some entries contain 'doi:' prefixes, several citations are incomplete (e.g., reference 1 lacks a title, and editorial numbering artifacts appear in the text such as '1~4' and a stray '2.'), and no DOIs or URLs are given for some entries that would need them for verification.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the central claim is an empirical benchmark claim, and the data-overlap / protocol concerns are validity risks rather than demonstrated reductions of the prediction to its inputs.

full rationale

The central claim is that a LLaMA model expanded with a gene BPE vocabulary, continuously pre-trained on DNA and protein sequences, and instruction-tuned on converted gene task data reaches accuracy comparable to SOTA. This is an empirical system claim, not a quantity derived from a fitted parameter defined in terms of the claimed output. The instruction templates in Section 2.2.2 convert downstream task data into Alpaca-style prompts, and Section 3.1 evaluates by checking whether generated tokens semantically match expected labels. That is a standard supervised fine-tuning and evaluation loop. No equation in the paper defines the reported accuracy in terms of a fitted parameter, and no prior work by the authors is invoked as a uniqueness theorem or as the sole justification for the chosen architecture. The closest concern is that Table 1 lists 'lucaone/convert' as instruction fine-tuning data while Section 3.2 lists 'lucaone' among evaluation datasets, with no explicit train/test split described. However, the paper does not state that the exact evaluation examples were included in the training set, and benchmark datasets normally have standard train/test splits, so without an explicit identity of training and test examples this is a data-hygiene and validity concern rather than a demonstrated circular step. Likewise, Table 9 reports LLaMA-Gene below SOTA on every row and compares against external numbers whose protocols are not fully specified; that is a comparability weakness, not circularity. The paper's own conclusion also acknowledges the gap, saying LLaMA-Gene performs adequately on DNA-related tasks but shows a noticeable gap in protein-related tasks. I therefore find no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on a fitted physical constant or a derived equation. The load-bearing choices are hand-set thresholds, window sizes, and tokenization sizes, plus the assumptions that LoRA pre-training transfers language knowledge to biological sequences and that permissive token matching is a valid accuracy measure. No new entities are introduced.

free parameters (4)
  • Regression binarization threshold = 0.5
    Set by hand in Section 2.2.2.5 to convert normalized expression values into low/high classes; this choice defines the regression evaluation.
  • Secondary structure token assignment tie-break order = H, then E, then C
    Hand-set rule in Section 2.2.2.2 for assigning one structure label to each token when tokens span multiple secondary structures; affects structure prediction accuracy.
  • Context window length for structure prediction = 30 preceding base pairs
    Chosen in Section 2.2.2.2 without justification; the model predicts the structure of the last token given the token plus the previous 30 bp.
  • BPE vocabulary size per sequence type = 30,000 tokens for DNA and 30,000 tokens for protein
    Chosen in Section 2.3.1 to expand the LLaMA vocabulary to roughly 91k tokens; this hand-set size defines the tokenization capacity.
assumptions (4)
  • domain assumption LLaMA's pretrained English language representations transfer to DNA and protein sequences after continued pre-training with LoRA.
    Section 2.3.2 assumes that updating ~10% of parameters via LoRA on 16GB of DNA and 16GB of protein sequences is sufficient to encode biological sequence semantics in a model pretrained on natural language.
  • domain assumption BPE tokenization of DNA and protein sequences produces meaningful tokens without explicit sequence markers.
    Section 2.3.1 expands the LLaMA vocabulary with BPE tokens for DNA and protein; the paper assumes these tokens are learnable and that a single vocabulary can handle both natural language and gene sequences.
  • ad hoc to paper Accuracy based on semantic match of output tokens is a valid evaluation metric.
    Section 3.1 defines accuracy as whether output tokens semantically match expected output, e.g., "promoter AGCCGGG" is scored correct for expected "promoter". This permissive matching is not standard and can inflate scores.
  • domain assumption Instruction fine-tuning on converted downstream task data generalizes to unseen gene tasks.
    Section 2.2.2 and 3.2 assume that fine-tuning on classification, structure, interaction, function, and regression (binarized) instructions yields a general-purpose model; no held-out task generalization test is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLaMA-Gene: A General-purpose Gene Task Large Language Model Based on Instruction Fine-tuning." pith.science (2026). https://pith.science/paper/RRGRGGVX

@misc{pith2026241200471,
  author       = {Pith},
  title        = {Pith review of: LLaMA-Gene: A General-purpose Gene Task Large Language Model Based on Instruction Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRGRGGVX}},
  note         = {Machine review of arXiv:2412.00471}
}
read the original abstract

Building a general-purpose task model similar to ChatGPT has been an important research direction for gene large language models. Instruction fine-tuning is a key component in building ChatGPT, but existing instructions are primarily based on natural language. Natural language and gene sequences have significant differences in tokenization and encoding. Therefore, constructing a multilingual model that can handle both natural language and gene sequences is crucial for solving this problem.In this paper, we expand the capabilities of the LLaMA large language model to include gene language. This involves expanding the vocabulary using the Byte Pair Encoding (BPE) method, specifically tailored for DNA and protein sequences, and conducting further pre-training on these sequences. We then convert various downstream gene task data into a unified format for instruction fine-tuning and further fine-tune the model on this data.Our study demonstrates that a mixed model of gene and natural language, fine-tuned with instructions, achieves results comparable to the current state-of-the-art (SOTA) in tasks such as gene classification and gene sequence interaction. This provides a promising direction for building a unified large language model for gene tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 2 canonical work pages

  1. [1]

    Z.Zhou,etal.,Dnabert-2:Efficientfoundationmodelandbenchmarkformulti-species

    work page

  2. [2]

    genome,arXivpreprintarXiv:2306.15006(2023)

    arXiv 2023

  3. [3]

    Nguyen, et al., Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution,Advancesinneuralinformationprocessingsystems36(2024)

    E. Nguyen, et al., Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution,Advancesinneuralinformationprocessingsystems36(2024)

    2024

  4. [4]

    F.Yang,etal.,scbertasalarge-scalepretraineddeeplanguagemodelforcelltypeannotation ofsingle-cellrna-seqdata,NatureMachineIntelligence4,852–866(2022)

    2022

  5. [5]

    A. Elnaggar, et al., Prottrans: Toward understanding the language of life through selfsupervised learning, IEEE transactions on pattern analysis and machine intelligence 44, 7112–7127(2021)

    2021

  6. [6]

    N.Brandes,D.Ofer,Y.Peleg,N.Rappoport,M.Linial,Proteinbert:auniversaldeeplearning modelofproteinsequenceandfunction,Bioinformatics38,2102–2110(2022)

    2022

  7. [7]

    Lin, et al., Evolutionary-scale prediction of atomic-level protein structure with a language model,Science379,1123–1130(2023)

    Z. Lin, et al., Evolutionary-scale prediction of atomic-level protein structure with a language model,Science379,1123–1130(2023)

    2023

  8. [8]

    & Davuluri, R

    Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT: Pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics 37, 2112–2120(2021).doi:doi.org/10.1093/bioinformatics/btab083

    doi:10.1093/bioinformatics/btab083 2021

Show all 19 references
  1. [9]

    doi: https://doi.org/10.1101/2023.06.12.544594

    Veniamin Fishman, et al.GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences.BioRxiv 2023.06.12.544594. doi: https://doi.org/10.1101/2023.06.12.544594

    2023 doi

  2. [10]

    BioRxiv 2023.07.11.548628; doi: https://doi.org/10.1101/2023.07.11.548628

    Daoan Zhang, et al.DNAGPT: A Generalized Pretrained Tool for Multiple DNA Sequence Analysis Tasks. BioRxiv 2023.07.11.548628; doi: https://doi.org/10.1101/2023.07.11.548628

    2023 doi

  3. [11]

    Wang, B. et al. Pre-trained Language Models in Biomedical Domain: A Systematic Survey. ACM Comput. Surv.56,1–52(2023).doi:doi.org/10.1145/3611651

    2023 doi

  4. [12]

    & Gao, G

    Chen, Z., Wei, L. & Gao, G. Foundation models for bioinformatics. Quant. Biol. (2024) doi:10.1002/qub2.69

    2024 doi

  5. [13]

    Benegas, G., Ye, C., Albors, C., Li, J. C. & Song, Y. S. Genomic Language Models: OpportunitiesandChallenges. arXiv(2024)doi:10.48550/arxiv.2407.11435

    2024 doi arXiv

  6. [14]

    BioRxiv2024.07.01.600583;doi:https://doi.org/10.1101/2024.07.01.600583

    Thomas Hayes, et al.Simulating 500 million years of evolution with a language model. BioRxiv2024.07.01.600583;doi:https://doi.org/10.1101/2024.07.01.600583

    2024 doi

  7. [15]

    LucaOne: Generalized Biological Foundation Model with Unified Nucleic Acid and Protein Language

    Yong He, et al. LucaOne: Generalized Biological Foundation Model with Unified Nucleic Acid and Protein Language. BioRxiv 2024.05.10.592927; doi: https://doi.org/10.1101/2024.05.10.592927

    2024 doi

  8. [16]

    Zhang, K., Zhou, R., Adhikarla, E. et al. A generalist vision–language foundation model for diversebiomedicaltasks.NatMed(2024).https://doi.org/10.1038/s41591-024-03185-2

    2024 doi

  9. [17]

    Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023. ProtST: multi-modality learning of protein sequences and biomedical texts. In Proceedings of the 40th International Conference on Machine Learning (ICML'23), Vol. 202. JMLR.org, Article 1615, 38749–38767

    2023

  10. [18]

    Zeyuan Wang, etc. 2024. InstructProtein: Aligning Human and Protein Language via Knowledge Instruction. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1114–1136, Bangkok, Thailand. AssociationforComputa...

    2024

  11. [19]

    Llama-genegithub,https://github.com/maris205/llama-gene

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.