{"id":"68a6ab0e-822c-48f2-a8f6-ceb114745171","arxiv_id":"2412.11618","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EvoLlama aligns ESM-2 sequence embeddings and ProteinMPNN structure embeddings with Llama-3, improving protein understanding over text-only LLMs on Mol-Instructions and PEER benchmarks.","lead":"EvoLlama is a protein chatbot that feeds both a protein's amino acid sequence and its predicted 3D shape into Llama-3, instead of just pasting the sequence as text. The paper reports that this extra structural and sequence information helps the model answer protein questions and predict protein properties, though the gains over existing protein models are modest and the code is not yet public.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline Mol-Instructions gain is measured against Llama-2 while EvoLlama uses Llama-3; without a text-only Llama-3 baseline on PMol/Mol, the improvement cannot be attributed to the protein encoders.","rationale":"After reading in good faith, the paper's contribution is the multimodal fusion architecture, and the strongest empirical support is the Mol-Instructions SFT result and the PEER comparison. The reader identified the structure encoder's marginal contribution as the weakest assumption; I agree that the ProteinMPNN ablation is thin (Tables 4 and 10 show small, mixed effects), and the limitations section itself concedes that predicted structures are less accurate than wet-lab ones. However, I see a more fundamental confound: the headline Mol-Instructions comparison is not decoder-controlled. EvoLlama uses Llama-3-8B; the SFT baseline is Llama-2-7B-Chat. Because base-model generation quality directly affects ROUGE-L on free-form protein descriptions, the observed 6-point gain is compatible with a 'better base LLM' explanation. The PEER table does include a text-only Llama-3-8B-Instruct baseline, which is a genuine strength and partially supports the multimodal claim on property prediction. However, the paper's boldest quantitative claim is on Mol-Instructions, where that control is absent. The ablation numbers, comparing a 10K-step EvoLlama to a 25K-step headline model, cannot substitute. Thus the conditional verdict remains appropriate: the paper should be accepted only after adding a same-decoder text-only baseline, reporting leakage-filtering criteria, and softening the zero-shot sentence, which is contradicted by Table 1 where EvoLlama zero-shot (0.1400) trails Alpaca (0.1625) and Baize (0.1750). My verdict is UNCHANGED because the reader already assigned CONDITIONAL for overlapping but not identical reasons.","tokens_in":21971,"tokens_out":7646,"duration_ms":65115,"concrete_test":"Fine-tune text-only Llama-3-8B (or Llama-3-8B-Instruct) on the same PMol or PMol+PEER data with the identical 25K-step recipe and evaluate on the same Mol-Instructions test split; if its average ROUGE-L reaches or exceeds 0.5200, the multimodal encoders are not needed for the headline Mol-Instructions gain, whereas if it falls near 0.4675 or below, the encoders contribute.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fusing ProteinMPNN and ESM-2 features into an LLM enhances protein understanding rests on the Mol-Instructions result in Table 1: EvoLlama 0.5200 vs Llama-2-7B-Chat Mol 0.4600. This comparison is confounded in two ways. First, EvoLlama decodes with Llama-3-8B while the baseline uses Llama-2-7B-Chat; Llama-3 is a substantially stronger base model, so the 6-point ROUGE-L gap may reflect decoder choice rather than the protein encoders. Second, the supervised EvoLlama is trained on PMol plus PEER, whereas the Llama-2 Mol baseline is trained only on Mol-Instructions; extra data is another uncontrolled variable. No text-only Llama-3-8B model fine-tuned on PMol or Mol-Instructions is reported, so the contribution of the two protein encoders cannot be isolated on this benchmark. The ablation study does not close the gap: Table 4 reports EvoLlama without ProteinMPNN at 0.4675, close to the Llama-2 Mol score of 0.4600, but these ablation models use a 10K-step recipe while the headline EvoLlama uses 25,000 steps, so the remaining 0.0525 difference is also confounded by training budget. The PEER benchmark does include a text-only Llama-3-8B-Instruct comparison that favors EvoLlama, but the Mol-Instructions benchmark, which carries the paper's headline quantitative claim, lacks the equivalent control. The reader's concern about ProteinMPNN redundancy is valid, but even if ProteinMPNN were informative, the core claim that adding protein encoders to Llama-3 helps would still be unestablished without a same-decoder text-only baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EvoLlama, a multimodal framework that connects a ProteinMPNN (or GearNet) structure encoder and an ESM-2 sequence encoder to a Llama-3-8B decoder through MLP projectors and element-wise addition of residue-level features. Training is performed in two stages: an optional projection tuning stage on Swiss-Prot-derived protein-text pairs, and a supervised fine-tuning stage on protein-oriented instructions from Mol-Instructions (PMol) and the PEER benchmark. The paper evaluates on four protein understanding tasks from Mol-Instructions and six protein property prediction tasks from PEER, reporting that EvoLlama outperforms fine-tuned protein-oriented LLMs in zero-shot settings and surpasses a state-of-the-art baseline after supervised fine-tuning on Mol-Instructions, while achieving competitive results on PEER.","tokens_in":22293,"tokens_out":10055,"duration_ms":78213,"significance":"The architecture is lightweight and modular, with only about 8% of parameters trainable, and the PEER results include a direct comparison to a fine-tuned text-only Llama-3-8B-Instruct model, which EvoLlama beats on all six tasks. This is a meaningful positive result for the value of protein encoders in LLM-based protein understanding. However, the Mol-Instructions headline claim is confounded by the choice of decoder and training data, and the ablation studies are run at a different training budget than the main model, so the specific contribution of the structural encoder remains unclear. The paper would be a solid empirical contribution if these controls were added and the claims were calibrated to the evidence.","major_comments":[{"comment":"The headline result that EvoLlama 'surpasses the state-of-the-art baseline with supervised fine-tuning by an average of 6%' (abstract; Table 1) compares EvoLlama, which uses a Llama-3-8B decoder and is trained on PMol plus PEER, with Llama-2-7B-Chat fine-tuned on the complete Mol-Instructions dataset. No text-only Llama-3-8B model fine-tuned on PMol or Mol-Instructions is reported, so the 0.5200 versus 0.4600 gap cannot be attributed to the protein encoders; it may reflect the stronger base decoder or the additional PEER training data. This control is essential because the PEER benchmark includes a text-only Llama-3-8B-Instruct baseline (Table 2) but Mol-Instructions does not.","section":"§5.1, Table 1 vs. Tables 4 and 7"},{"comment":"The ablation comparing EvoLlama with and without ProteinMPNN (0.4800 versus 0.4675) is run for 10K steps, as stated at the start of Section 5.3, while the headline model in Table 1 is trained for 25,000 steps according to Table 7. The paper does not provide a 25K-step ablation for the sequence-only model, so the 0.0125 difference may be an artifact of training budget. The structure encoder's contribution to the Mol-Instructions results is therefore not established at the training budget used for the main result.","section":"§5.3, Tables 4 and 7"},{"comment":"The abstract claims that EvoLlama 'outperform[s] other fine-tuned protein-oriented LLMs in zero-shot settings by an average of 1%-8%', but Table 1 reports EvoLlama (ProteinMPNN+ESM-2) zero-shot average ROUGE-L of 0.1400, which is lower than Alpaca (0.1625) and Baize (0.1750) fine-tuned on PMol. Only the GearNet variant (0.1700) exceeds those two baselines. The paper's own text in Section 5.1 says 'surpass or approach', so the abstract and introduction should be corrected to match the table.","section":"§5.1, Table 1 and abstract"},{"comment":"The sentence 'compared to Llama-3-8B-Instruct, used as a text decoder in our approach, EVOLLAMA improves performance on all tasks by incorporating the multimodal structure and sequence representations of proteins' attributes the improvement to the multimodal fusion. However, Table 10 shows that at the same 10K-step budget, the model without ProteinMPNN achieves 61.14 average on PEER, already 3.29 points above the fine-tuned text-only Llama-3-8B-Instruct average of 57.85 (Table 2). Adding ProteinMPNN improves the average by only 1.24 points and actually decreases subcellular localization (68.05 to 68.63 without the structure encoder) and fold classification (6.18 to 7.94). The evidence thus indicates that the sequence encoder, not the structural encoder, is the primary source of the gains over a text-only LLM.","section":"§5.2 and Table 10"}],"minor_comments":[{"comment":"The task groupings appear swapped: 'Protein Understanding Tasks' is listed with PEER benchmark tasks (Solubility, Subcellular Localization, Binary Localization, Fold Classification, Yeast PPI, Human PPI), while 'Protein Property Prediction Tasks' is listed with Mol-Instructions (PMol) tasks. This makes the experimental setup confusing.","section":"Table 6"},{"comment":"The sentence 'Yeast PPI prediction ... and human PPI prediction ... are defined as binary localization tasks' should read 'binary classification tasks' to avoid confusion with the earlier 'binary localization' task.","section":"§5.2, first paragraph"},{"comment":"The notation Xa,i and Xa,<i in Equation (1) is not defined; the paper should specify that i indexes tokens of the answer sequence.","section":"§3.1, Eq. (1)"},{"comment":"The figure contains the fragment 'AlphaFold ESMFold' without a connecting verb; the caption should state explicitly that structures are predicted by AlphaFold-2 during projection tuning and by ESMFold during supervised fine-tuning.","section":"Figure 1"},{"comment":"The abstract states 'We will release our code in a future version'; for reproducibility it would be helpful to release the code, the exact data filtering procedure for the 369K Swiss-Prot subset, and the data splits at the time of publication.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper has a positive core result on PEER: the fine-tuned multimodal model beats a fine-tuned text-only Llama-3-8B-Instruct on all six tasks. The main weaknesses are the confounded Mol-Instructions comparison (no text-only Llama-3 control), the 10K-versus-25K ablation mismatch, and the overstatement of zero-shot results in the abstract. These are fixable with additional experiments and a recalibration of the claims, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing you should know: the headline '6% over SOTA' on Mol-Instructions is not a clean result. EvoLlama decodes with Llama-3-8B, the baseline is Llama-2-7B-Chat, and there is no text-only Llama-3-8B fine-tuned on the same data. The stress-test note is correct that the gap could be mostly decoder strength plus the extra PEER training data. The ablated model without ProteinMPNN, at 0.4675 with a 10K-step recipe, is close to the Llama-2 Mol score of 0.46, so the claimed contribution is fragile.\n\nThat said, the paper is not without merit. The architecture is a clean LLaVA-style two-encoder-plus-projector, and the element-wise fusion of residue-aligned structure and sequence features is a sensible way to halve token count. The ablations are honestly reported: Table 4 and Table 10 show both encoders help, though ProteinMPNN's marginal contribution is small (0.48 vs 0.4675 on PMol; 62.38 vs 61.14 on PEER). The optional projection tuning stage, and the finding that it hurts when followed by supervised fine-tuning due to AlphaFold-2 vs ESMFold structure mismatch, is a useful empirical observation. They also show scaling with ESM-2 size, and the PEER benchmark includes a same-decoder comparison against Llama-3-8B-Instruct in Table 2, where EvoLlama averages 64.41 vs 57.85. That is not a fully controlled fine-tuned baseline, but it does indicate the multimodal features add something beyond the base LLM.\n\nSoft spots, in proportion: the zero-shot claim in the abstract is overstated. Table 1's EvoLlama zero-shot (0.14) trails Alpaca (0.1625) and Baize (0.1750) fine-tuned on PMol, so the '1%-8%' claim does not hold for those baselines. The leakage filtering for the Swiss-Prot projection data is described only as 'filtered based on downstream tasks' with no criteria; that needs detail. The closest multimodal baselines, ProteinGPT and ProtLLM, are discussed but not evaluated. The ROUGE-L evaluation is modified by removing non-critical parts, which is a minor metric concern; they do report the re-evaluation, so it's not hidden. Code and data are not public yet.\n\nWho gets value: someone working on protein LLMs who wants a concrete example of structure-sequence fusion with Llama-3, plus a useful ablation on the projection-tuning trade-off. It is a solid subfield contribution, not a field reset. I would send it to a referee, but with the request to require a same-decoder text-only baseline on Mol-Instructions and clear leakage criteria before acceptance.","headline":"EvoLlama's headline Mol-Instructions gain is confounded by the Llama-3 vs Llama-2 decoder swap, but the PEER evidence and honest ablations make it worth a referee's time.","tokens_in":22893,"tokens_out":4441,"would_cite":true,"duration_ms":35214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EvoLlama claims that adding a protein structure encoder and a protein sequence encoder to Llama-3, fused through a lightweight MLP projector, enhances the LLM's protein understanding beyond what sequence-only protein LLMs achieve.","keywords":["protein understanding","multimodal LLM","protein language model","structure encoder","sequence encoder","instruction tuning","Mol-Instructions","PEER benchmark"],"falsifier":"A decisive check would be to train EvoLlama and an ESM-2-only variant with identical data and compute budget over many random seeds and tasks; if the full model's average ROUGE-L and PEER accuracy never exceed the sequence-only model by a margin larger than the reported 0.0125 ROUGE-L difference on Mol-Instructions, the structure branch's contribution would be negligible. More directly, running the same pipeline with experimentally determined crystal structures instead of AlphaFold-2/ESMFold predictions and observing no performance change would indicate the geometric features are not carrying genuine structural signal.","tokens_in":21735,"feed_emoji":"🧬","tokens_out":6628,"duration_ms":47954,"temperature":0.7,"pith_summary":"EvoLlama claims that feeding an LLM both the amino acid sequence and the predicted 3D structure of a protein, encoded by pre-trained protein models and fused into the text embedding space, produces a protein-understanding model that is stronger and more sample-efficient than fine-tuning a text-only LLM on protein instructions. The paper builds a Llama-3 decoder with an ESM-2 sequence encoder and a ProteinMPNN structure encoder, connected by MLP projectors that add the two feature streams element-wise. In zero-shot evaluation on Mol-Instructions the model matches or beats several fully fine-tuned protein LLMs, and after supervised fine-tuning it exceeds Llama-2-7B-Chat by about six ROUGE-L points on the same benchmark. On the PEER protein property prediction tasks it reaches accuracies competitive with task-specific baselines. The paper's claim is that protein structure and sequence are complementary modalities a general-purpose LLM can absorb through a lightweight projector.","feed_headline":"Protein structure and sequence fusion lifts LLM protein test scores","feed_subtitle":"EvoLlama adds ESM-2 and ProteinMPNN encoders to Llama-3 and beats sequence-only baselines on protein tasks.","key_machinery":"The load-bearing object is the multimodal fusion path: ProteinMPNN's encoder produces per-residue structural features, ESM-2 produces per-residue sequence features, and two MLPs project each into the Llama-3 word-embedding space; the projected tokens are then combined by element-wise addition so the LLM attends over one protein token per residue instead of two. This element-wise fusion halves the number of protein tokens the decoder attends to, cutting inference latency by about 20%, while the two-stage training (optional projection tuning on Swiss-Prot descriptions, then supervised fine-tuning on verbalized Mol-Instructions and PEER tasks) teaches the projector to align protein features with language. The same architecture is shown to work with GearNet in place of ProteinMPNN, and with smaller ESM-2 encoders, which the paper uses to argue the design is plug-and-play.","core_discovery":"The central claim is that a multimodal representation of proteins, combining an ESM-2 sequence embedding with a ProteinMPNN geometric embedding through simple MLP projection and element-wise addition, can be injected into a frozen Llama-3 decoder and substantially improve its ability to answer protein questions. The paper reports that EvoLlama (ProteinMPNN+ESM-2) achieves 0.52 average ROUGE-L on Mol-Instructions after supervised fine-tuning, versus 0.46 for Llama-2-7B-Chat fine-tuned on the full Mol-Instructions dataset, and that in zero-shot settings it outperforms other fine-tuned protein-oriented LLMs by 1% to 8%. On PEER, it beats or approaches prior state-of-the-art results on solubility, binary localization, and human protein-protein interaction prediction. The authors attribute the gain to the fused structure-sequence representation, noting that ablations removing ProteinMPNN or ESM-2 both hurt performance on Mol-Instructions and PEER.","pith_inferences":["The paper's own ablation numbers (removing ProteinMPNN drops Mol-Instructions ROUGE-L from 0.4800 to 0.4675 and PEER average from 62.38 to 61.14) suggest the structure branch contributes a small, consistent margin; whether that margin is worth its compute is an open question the paper does not fully settle.","The same MLP-projection plus element-wise fusion recipe could be applied to other biomolecular modalities such as small-molecule graphs or DNA sequences; the authors list this as future work.","The reported sensitivity to AlphaFold-2 versus ESMFold structures implies that zero-shot generalization depends on matching the structure encoder's feature distribution between training and inference; a testable extension would be fine-tuning on ESMFold structures only and measuring transfer back to AlphaFold-2 structures.","Because the fused model improves on Llama-3-8B-Instruct across all PEER tasks, the fusion gain may be largest when the base text decoder already has strong instruction-following; one could test whether the gain shrinks for weaker or stronger decoders."],"forward_implications":["After only projection tuning, EvoLlama can follow human protein instructions in zero-shot settings, reducing the need for task-specific fine-tuning.","The 6-point ROUGE-L gain over full-parameter fine-tuned Llama-2-7B-Chat suggests that frozen-LLM multimodal instruction tuning can beat full fine-tuning on protein text tasks while training far fewer parameters.","On PEER, the fused model approaches or beats task-specific baselines on several properties, indicating that a single protein LLM can replace separate classifiers for solubility, localization, and protein-protein interaction prediction.","Because the architecture is plug-and-play, swapping in different structure encoders or sequence encoders should extend the same recipe to other protein representations."],"supporting_citations":[{"why":"Supplies the ProteinMPNN structure encoder whose geometric features form the structure branch.","marker":"(Dauparas et al., 2022)"},{"why":"Provides the ESM-2 sequence encoder and ESMFold structure prediction used for supervised fine-tuning data.","marker":"(Lin et al., 2022)"},{"why":"Supplies the Llama-3 8B text decoder that EvoLlama aligns protein features to.","marker":"(Dubey et al., 2024)"},{"why":"Provides the Mol-Instructions (PMol) dataset and the fine-tuned baselines EvoLlama is compared against.","marker":"(Fang et al., 2023)"},{"why":"Provides the PEER benchmark tasks and task-specific baseline results for protein property prediction.","marker":"(Xu et al., 2022)"},{"why":"Supplies AlphaFold-2 predicted structures used for Swiss-Prot projection tuning data.","marker":"(Jumper et al., 2021)"},{"why":"Prot2Text is the multimodal structure-sequence baseline EvoLlama outperforms in zero-shot protein function prediction.","marker":"(Abdine et al., 2024)"},{"why":"ProLLaMA is the protein LLM baseline used for zero-shot domain/motif comparison.","marker":"(Lv et al., 2024)"}],"fun_headline_variants":["EvoLlama fuses protein structure and sequence to beat sequence-only LLMs","Structure-aware EvoLlama tops protein benchmarks","Fusing protein structure and sequence boosts LLM protein skills","Mix of structure and sequence cues lifts protein understanding in LLMs","EvoLlama: structure+sequence beats sequence-only protein LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the assumption that AlphaFold-2 and ESMFold predicted structures are accurate enough that ProteinMPNN's geometric features add information beyond what ESM-2 sequence embeddings already capture, so that the structure branch is not merely redundant.","fun_headline_variants_meta":{"raw":{"variants":["EvoLlama fuses protein structure and sequence to beat sequence-only LLMs","Structure-aware EvoLlama tops protein benchmarks","Fusing protein structure and sequence boosts LLM protein skills","Mix of structure and sequence cues lifts protein understanding in LLMs","EvoLlama: structure+sequence beats sequence-only protein LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3129,"prompt_tokens":1024,"completion_tokens":2105,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2027}},"tokens_in":640,"tokens_out":2105,"duration_ms":14946,"temperature":1.0,"reasoning_tokens":2027,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:45:34.658527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to train EvoLlama and an ESM-2-only variant with identical data and compute budget over many random seeds and tasks; if the full model's average ROUGE-L and PEER accuracy never exceed the sequence-only model by a margin larger than the reported 0.0125 ROUGE-L difference on Mol-Instructions, the structure branch's contribution would be negligible. More directly, running the same pipeline with experimentally determined crystal structures instead of AlphaFold-2/ESMFold predictions and observing no performance change would indicate the geometric features are not carrying genuine structural signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ProteinMPNN structure encoder whose geometric features form the structure branch."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prot2Text is the multimodal structure-sequence baseline EvoLlama outperforms in zero-shot protein function prediction."}],"review_version":1}