{"id":"6b44aaec-8973-4ceb-adbf-04cbc10eec56","arxiv_id":"2412.00471","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An instruction-tuned LLaMA with a merged DNA, protein, and English BPE vocabulary handles multiple gene tasks, but stays below SOTA on every reported benchmark.","lead":"LLaMA-Gene is a large language model that adds DNA and protein tokens to LLaMA and fine-tunes it on gene tasks in a chat format. It is a candidate design for a unified gene LLM, but its reported accuracies are below the cited SOTA on every task it compares.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA-comparability claim rests on Table 9, but LLaMA-Gene's scores use a permissive substring match and are compared to external SOTA numbers with unreported protocols; since Table 9 itself shows LLaMA-Gene below SOTA on every task, the comparison is the load-bearing unsupported step.","rationale":"The paper's abstract asserts SOTA-comparable performance, and the only evidence is Table 9. That table has two problems that are directly load-bearing. First, LLaMA-Gene's accuracy is computed with a permissive semantic-match rule (Section 3.1: 'promoter AGCCGGG' counts as 'promoter'), while the cited SOTA numbers come from other papers with unstated evaluation protocols. If those protocols use exact-match or per-position scoring, the comparison is systematically biased in LLaMA-Gene's favor. Second, even under that favorable rule, LLaMA-Gene is below SOTA on every row of Table 9; the protein multi-sequence rows are far outside any reasonable 'comparable' range (0.63 vs 0.87 and 0.71 vs 0.91). Therefore the central claim rests on an unverified assumption that the SOTA numbers were produced under protocols making the comparison fair. A single protocol-reproduction check on one row would settle whether the claim survives. The reader's verdict of REJECT is justified; my analysis adds a concrete falsification test but does not change the verdict.","tokens_in":7178,"tokens_out":4365,"duration_ms":42961,"concrete_test":"Take the protein multiple-sequence row of Table 9 (LLaMA-Gene 0.63 vs SOTA 0.87). Reconstruct the exact evaluation set used for that row, run the released checkpoint with greedy decoding, and score with exact-match after stripping the model's extra tokens. Then recompute the SOTA number on the same examples with the same exact-match rule. If LLaMA-Gene's exact-match accuracy is below the recomputed SOTA by more than the Table 9 margin, or if the SOTA number changes materially, the SOTA-comparability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that instruction fine-tuning a LLaMA model with a unified BPE vocabulary yields accuracy comparable to SOTA on gene tasks. The only quantitative support is Table 9, which lists LLaMA-Gene accuracy against 'SOTA' for 11 task/type rows. Two load-bearing conditions must hold for this table to evidence the claim. First, the LLaMA-Gene accuracies must be measured under the same protocol as the cited SOTA numbers; Section 3.1 shows they are not, because a generated string like 'promoter AGCCGGG' is counted as correct for the label 'promoter', while standard fixed-head classifiers are evaluated with exact-match or task-specific metrics. Second, the comparison must be numerically meaningful; Table 9 shows LLaMA-Gene below SOTA on every row, with protein multi-sequence gaps of 0.63 vs 0.87 and 0.71 vs 0.91. If the two protocols differ, the gap may be larger or smaller, but the paper provides no protocol details, splits, or error bars for either side. The abstract's 'comparable to SOTA' is therefore not established by the paper's own table; it requires an unverified assumption that the external SOTA numbers were generated under conditions that make the comparison favorable. Data overlap is an additional unaddressed risk: instruction fine-tuning uses 'lucaone/convert' data and evaluation uses lucaone, with no train/test split described.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaMA-Gene, a 7B/8B LLaMA model whose vocabulary is expanded with BPE-based DNA and protein tokens, followed by continuous pre-training and instruction fine-tuning on a variety of gene tasks converted into a unified prompt format. The central claim, stated in the abstract, is that this mixed-language instruction-tuned model 'achieves results comparable to the current state-of-the-art (SOTA) in tasks such as gene classification and gene sequence interaction.' The paper reports accuracy values in Table 9 across classification, structure prediction, multiple sequence analysis, function prediction, and regression-style tasks for both DNA and protein sequences, and it argues that the instruction-tuning approach offers a ChatGPT-like general-purpose interface for gene tasks.","tokens_in":7506,"tokens_out":5377,"duration_ms":51839,"significance":"If the central claim were substantiated, the contribution would be significant: it would demonstrate that a single general-purpose language model can unify multiple gene task types in a dialogue format, advancing the field from task-specific fine-tuned models toward a unified gene LLM similar to ChatGPT. The design choices—unified BPE tokenization, conversion of diverse tasks into a single instruction format, and LoRA-based continuation of a 7B-scale LLaMA—are reasonable and potentially reusable. The listed public GitHub repository is a positive step for reproducibility. However, the current evidence is internally inconsistent and insufficient: the paper's own Table 9 shows LLaMA-Gene below the cited SOTA on every reported task, with gaps up to 0.24, and the evaluation protocol uses a permissive semantic-matching rule with no described train/test separation. The significance of the work therefore cannot be assessed from the manuscript as written.","major_comments":[{"comment":"The central claim of SOTA comparability is contradicted by the paper's own results. Table 9 shows LLaMA-Gene below the cited SOTA on all 11 rows, with gaps such as protein multiple sequence 0.63 vs 0.87, DNA+protein multiple sequence 0.71 vs 0.91, and protein classification 0.64 vs 0.72. Section 3.2 itself admits a 'noticeable gap in protein-related tasks.' Therefore the abstract's statement that the model achieves results 'comparable to the current state-of-the-art' is not supported by the reported numbers.","section":"Abstract and Section 3.2, Table 9"},{"comment":"The evaluation metric is not comparable across the compared systems. Section 3.1 states that a generated string such as 'promoter AGCCGGG' is counted as correct for the expected label 'promoter,' which is a permissive semantic-match rule that inflates accuracy relative to the exact-match or task-specific metrics typically used by fixed-head classification models. The SOTA values in Table 9 are external and are not accompanied by their evaluation protocols, dataset splits, or even citations to the specific SOTA methods. Without a shared protocol, the comparison in Table 9 cannot support the abstract's SOTA-comparability claim.","section":"Section 3.1 and Table 9"},{"comment":"The paper does not establish that the evaluation data are disjoint from the instruction fine-tuning data. Table 1 lists 'lucaone/convert' as instruction fine-tuning data, and Section 3.2 states that evaluation datasets 'primarily referenced include those from DNABert2, lucaone, InstructionProtein, and BiomedGPT.' No train/test separation, sequence-level deduplication, or task-level split is described. This creates a serious risk that the reported accuracies partly reflect memorization of training instructions rather than generalization, which would invalidate the numerical claims even if the protocol issues were resolved.","section":"Section 2.2.2/Table 1 and Section 3.2"},{"comment":"The regression tasks are converted to binary classification using a fixed threshold of 0.5, and Table 9 reports accuracy on these binary labels. State-of-the-art regression models are normally evaluated on the original continuous values with metrics such as correlation or RMSE. The reported 'regressiontask' accuracy is therefore not directly comparable to the cited SOTA numbers, further undermining the abstract's general claim of SOTA comparability for these rows.","section":"Section 2.2.2.5 and Table 9"}],"minor_comments":[{"comment":"In the example instruction, 'Determine core promoter detection of following dna sequence' is missing an article and uses lowercase 'dna'; for consistency with the rest of the paper, it should read 'Determine the core promoter detection of the following DNA sequence.'","section":"Section 2.2.2.1"},{"comment":"The text 'WeUseAccuracy(ACC)asthePrimaryEvaluationMetric' lacks spaces and should be 'We use accuracy (ACC) as the primary evaluation metric.'","section":"Section 3.1"},{"comment":"Table 7 is titled 'generegressiontask' and Table 8 is titled 'gene function prediction task,' but both tables describe the same gene expression level prediction task; the table titles should be consistent and accurate.","section":"Tables 7 and 8"},{"comment":"The paper states that 'The evaluation involves 2 to 3 specific datasets per task' but does not name these datasets, report sample sizes, or provide error bars or confidence intervals, making it impossible to assess the statistical reliability of the reported accuracies.","section":"Section 3.2"},{"comment":"The reference list is inconsistently formatted: some entries contain 'doi:' prefixes, several citations are incomplete (e.g., reference 1 lacks a title, and editorial numbering artifacts appear in the text such as '1~4' and a stray '2.'), and no DOIs or URLs are given for some entries that would need them for verification.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a preliminary technical report rather than a fully developed research article. The central claim is contradicted by the paper's own numerical results, and the evaluation lacks the protocol details needed to verify any of the reported numbers. If the authors can re-evaluate with proper train/test separation, use exact-match or task-appropriate metrics, and reposition the claims as a proof-of-concept that is below SOTA, a resubmission might be considered, but the current version does not meet the standards for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this is a classic 'plausible recipe, unsupported headline' paper. The authors build LLaMA-Gene, a 7B LLaMA with a merged BPE vocabulary for DNA, protein, and English, then LoRA-pretrain on sequences and instruction-tune on converted downstream tasks. The unified tokenization without special markers is a real variation on DNAGPT, LucaOne, and ESM3, and the instruction-tuning template for multiple task types is a reasonable engineering contribution. Credit where due: the paper tests classification, structure, interaction, function, and regression, and it is candid that the evaluation is preliminary, saying in Section 3.2 that the datasets 'may not be universally applicable.'\n\nThe soft spots are not minor. The abstract claims results 'comparable to SOTA,' but Table 9 shows LLaMA-Gene below the cited SOTA on all eleven rows, sometimes by 0.24. Section 3.1 defines accuracy with a permissive semantic-match rule: a generated string 'promoter AGCCGGG' is counted correct for the label 'promoter.' That is a different protocol from the fixed-head classifiers that produced the SOTA numbers, so the comparison is not apples-to-apples. The paper gives no error bars, no dataset splits, no train/test separation. There is a real overlap risk because instruction fine-tuning uses lucaone/convert and evaluation uses lucaone. No code or data is provided, despite a GitHub link in the references. So the central claim is unsupported by the paper's own table.\n\nThat said, the paper is coherent on its own terms and the authors are transparent about the limitation. The engineering direction is worth exploring, but this manuscript does not demonstrate that the approach is competitive. A serious referee should ask for code and data, exact-match evaluation, and a revised abstract. I would not cite it yet, but I would send it to peer review rather than desk reject, because the method is relevant and the evaluation flaws are fixable.","headline":"Promising engineering recipe, but the abstract overclaims SOTA comparability and the paper's own Table 9 shows gaps on every task; worth peer review as a methods paper, not as a SOTA result.","tokens_in":8034,"tokens_out":2751,"would_cite":false,"duration_ms":25382,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Single gene chat model nears specialist accuracy","keywords":["gene large language model","instruction fine-tuning","byte pair encoding","DNA sequence","protein sequence","multi-task learning","LLaMA","genomics"],"falsifier":"Run LLaMA-Gene on the same held-out datasets used by DNABert2 or LucaOne and score it with exact-match accuracy (the output must equal the expected label, with no extra tokens). If exact-match accuracy falls well below both the reported semantic-match scores and the SOTA numbers, or if a majority-class baseline ties the model on any task, the claim of comparability to current state-of-the-art is not supported.","tokens_in":6944,"feed_emoji":"🧬","tokens_out":6926,"duration_ms":61970,"temperature":0.7,"pith_summary":"The paper sets out to show that a single large language model can handle many gene-analysis tasks at once, the way ChatGPT handles many text tasks. To do this, the authors take a 7B- or 8B-parameter LLaMA model, expand its vocabulary with a unified byte-pair-encoding tokenizer trained on DNA and protein sequences, continue pretraining on genomic, proteomic, and biology-text data, and then convert five kinds of downstream gene tasks into instruction–response pairs for fine-tuning. They report that the resulting model, LLaMA-Gene, reaches accuracies on DNA classification, structure prediction, sequence-interaction, function prediction, and regression tasks that are close to the published numbers of specialized state-of-the-art models, with a larger gap on protein tasks. The point is that a ChatGPT-style dialogue model, rather than a collection of task-specific models, could serve as a general interface to genomic and proteomic analysis.","feed_headline":"Single gene chat model nears specialist accuracy","feed_subtitle":"A LLaMA model fine-tuned on DNA, protein, and instructions handles five gene-task types in dialogue.","key_machinery":"The carrying mechanism is the unified vocabulary: a byte-pair encoding (BPE) tokenizer trained on genome fragments and protein sequences, whose 30,000 DNA tokens and 30,000 protein tokens are merged into LLaMA's original 32,000-token vocabulary to give roughly 91,000 tokens total. Because DNA, protein, and natural language share one tokenizer, no special markers like <protein>…</protein> are needed to tell the model what a sequence is. On top of that, LoRA is used to continue pretraining (about 10% of parameters) on a mix of genomes, proteomes, and biology papers, and then every downstream task is converted into Alpaca-style instruction templates and fine-tuned with the standard causal language-model objective, which is what lets one model answer many task types in dialogue.","core_discovery":"On its own terms, the paper's discovery is that a single mixed-language model — natural language plus DNA plus protein in one BPE vocabulary — can be instruction-tuned to answer gene-task questions conversationally and still stay within a few accuracy points of specialized models on DNA tasks and several points behind on protein tasks. The authors claim this validates the feasibility of moving gene large language models from the 'GPT era' of per-task heads to the 'ChatGPT era' of a single model that handles classification, structure, interaction, function, and regression through prompts. They explicitly attribute the remaining protein-task gap to smaller training datasets and computational limits, not to a fundamental flaw in the approach.","pith_inferences":["The same unified-tokenizer-plus-instruction recipe could be carried over to other biomolecular modalities, such as RNA, single-cell expression profiles, or epigenetic marks, yielding one conversational model over a wider slice of biology.","The semantic-match scoring rule likely overstates the model's precision; an exact-match or token-F1 score would give a stricter measure of how reliably the model follows instructions, and would probably be a better guide for real users.","A head-to-head benchmark with identical train/test splits, labels, and scoring would settle whether the unified model genuinely matches specialized models or just comes near them, since cross-paper comparisons currently mix different protocols."],"forward_implications":["One instruction-tuned model replaces separate task-specific heads for DNA classification, structure prediction, interaction detection, function prediction, and regression, so adding a new task only requires a new prompt template and fine-tuning data.","The chat interface lets genomic analyses be driven by prompt engineering, retrieval-augmented generation, and agents built for natural-language models.","Because no special sequence markers are needed, models can exchange DNA and protein sequences directly in the same token space, simplifying multi-model collaboration.","The authors expect larger parameter scales and larger gene datasets to narrow the protein-task gap, since the current gap is attributed to training data size rather than the method.","Newer LLaMA versions did not change gene-task results, suggesting the improvements come mainly from the tokenizer and instruction scheme, not the base model's fine-tuning refinements."],"supporting_citations":[{"why":"Supplies the DNA-task baselines and dataset suite (DNABert2, HyenaDNA, ScBert) that define the SOTA numbers for DNA classification and structure prediction.","marker":"1~4"},{"why":"Provides the unified-model precedents (LucaOne, BiomedGPT) whose downstream datasets and evaluation protocols LLaMA-Gene benchmarks against.","marker":"15,16"},{"why":"InstructProtein, the source of protein instruction-tuning data and a protein-task comparison point for the instruction-following design.","marker":"18"},{"why":"ProtST, a protein-text model used as a comparison baseline for protein function and interaction tasks.","marker":"17"},{"why":"ESM3, whose functional keyword tokens exemplify the marker-based alternative that the marker-free unified tokenizer aims to replace.","marker":"13"}],"fun_headline_variants":["One LLaMA model chats DNA and protein tasks","Gene chat model: one model, many tasks","Mixed-language LLaMA tackles gene tasks","From GPT to ChatGPT for gene AI","Instruction-tuned LLaMA speaks gene and language"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison is valid: the published SOTA accuracy numbers were produced under their own evaluation protocols, while LLaMA-Gene's accuracy is scored under a permissive rule that counts an output like 'promoter AGCCGGG' as correct for the label 'promoter'.","fun_headline_variants_meta":{"raw":{"variants":["One LLaMA model chats DNA and protein tasks","Gene chat model: one model, many tasks","Mixed-language LLaMA tackles gene tasks","From GPT to ChatGPT for gene AI","Instruction-tuned LLaMA speaks gene and language"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2293,"prompt_tokens":869,"completion_tokens":1424,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1354}},"tokens_in":485,"tokens_out":1424,"duration_ms":10555,"temperature":1.0,"reasoning_tokens":1354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:20:54.594614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LLaMA-Gene on the same held-out datasets used by DNABert2 or LucaOne and score it with exact-match accuracy (the output must equal the expected label, with no extra tokens). If exact-match accuracy falls well below both the reported semantic-match scores and the SOTA numbers, or if a majority-class baseline ties the model on any task, the claim of comparability to current state-of-the-art is not supported.","supporting_citations":[],"review_version":1}