{"id":"7c7628e5-79f6-47d8-bd50-50a4008f1c66","arxiv_id":"2509.07983","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Activation steering can guide protein language models to generate and optimize sequences with higher predicted thermostability, solubility, or GFP brightness, but only in surrogate-based evaluation.","lead":"The authors adapt an LLM technique called activation steering to protein language models, adding a steering vector during sequence generation to push outputs toward desired properties like thermostability. The method needs no retraining, but the reported gains are measured with the same predictor used to build the steering signal, so independent validation is still needed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains are measured by the same surrogate predictors used to define the steering target; without an independent oracle, ASPO's 'precise control' could be predictor-gaming rather than biological improvement.","rationale":"The reader's weakest_assumption identified the same issue: surrogate predictors are used both to construct steering targets and to evaluate success. I agree with that assessment. The steering mechanism itself is not internally inconsistent; the problem is external validity. Since all headline numbers come from the same predictors, a positive control (e.g., checking that generated sequences preserve structure or that independent predictors agree) is missing. The baseline comparison is also asymmetric: ASPO is optimized toward the same predictor used for evaluation, whereas AdaLead/PEX/GGS are deliberately stripped of ground-truth updates. These issues jointly make the paper's central claim unsupported as written. Because the reader already reached REJECT for essentially this reason, my stress-test does not change the verdict.","tokens_in":15551,"tokens_out":4209,"duration_ms":52087,"concrete_test":"Take a random sample of ~50 sequences produced by ESM2+ASPO and ESM3+ASPO in Table 2 (or regenerate them from the released code) and measure their melting temperatures experimentally (e.g., nanoDSF or differential scanning fluorimetry). Compare the measured Tm distribution with the predicted values and with the AdaLead/PEX baseline sequences under the same assay. If the predicted 80-88°C values do not reproduce, or if ASPO no longer dominates the baselines, the reported gains are surrogate artifacts and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the fitness scores in Tables 1-4 track real protein properties. That requirement is not met, because the same surrogate predictors define the steering target and measure success. Section 4.1.1 constructs the positive/negative sets from predicted thermostability/solubility values; Section A.2 introduces those predictors (thermostability Spearman 0.76; solubility accuracy 0.708; GFP predictor from [13]); and Sections 4.1-4.2 report 'Fitness' using the same predictors. A mean-difference steering vector therefore learns the direction from 'predictor says low' to 'predictor says high,' and the evaluation rewards movement along exactly that direction. A surrogate with moderate global accuracy can contain systematic, exploitable biases (compositional or motif artifacts, length effects, calibration errors), so the reported improvements do not establish real thermostability/solubility/GFP gains. The GFP experiment is the most exposed: both the task split and the evaluation predictor come from [13]. The optimization comparison does not rescue the claim because AdaLead/PEX/GGS are modified to run with a static surrogate and no ground-truth fitness updates (Section 4.2.1), removing the mechanism those methods were designed for. Thus the paper demonstrates control of surrogate scores, not precise protein engineering.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts activation steering (ActAdd) to protein language models (PLMs). Steering vectors are computed as the mean activation difference between positive and negative protein sets defined by surrogate predictors of thermostability and solubility. The vectors are added to internal activations during generation for autoregressive (ProLLaMA) and auto-encoding (ESM2, ESM3) PLMs, and the approach is extended to a protein optimization framework (ASPO) that selects mutation sites by projecting token representations onto the steering vector. Experiments report improvements in predicted fitness, diversity, novelty, and optimization metrics over fine-tuning and search-based baselines, including a GFP brightness task. The paper claims training-free, precise control over protein generation and optimization.","tokens_in":15845,"tokens_out":4723,"duration_ms":55599,"significance":"If the reported fitness gains reflect genuine improvements in real protein properties, the method would be a valuable lightweight, architecture-agnostic alternative to fine-tuning and iterative optimization. The paper's strengths include a simple formulation, evaluation across multiple PLM architectures, sensitivity analyses for key hyperparameters, and a multi-property steering extension. However, the central empirical claim rests entirely on surrogate predictors that define both the steering target and the evaluation metric, so the current evidence does not distinguish real biological improvement from predictor exploitation. That concern, plus the modified baseline setup, substantially limits the significance of the results as presented.","major_comments":[{"comment":"The positive/negative sets for steering vector extraction are constructed using predicted thermostability/solubility values (Section 4.1.1, Data), and the reported fitness metrics in Tables 1-4 are computed with the same predictors (Section A.1, Fitness). This creates a circular evaluation: the steering vector is the mean activation difference between sequences labeled high and low by the predictor, and the success metric is that predictor's output on steered sequences. If the predictor is approximately linear in the representation layer, adding the steering vector moves activations in the direction that increases the predicted score by construction. The observed gains therefore do not establish that the generated or optimized sequences have increased real thermostability, solubility, or GFP brightness. The authors should evaluate on independent experimental data (e.g., the experimental GFP brightness values underlying the [13] benchmark, or held-out Meltome proteins) or at least use a different predictive model for evaluation. Without such validation, the central claim of precise protein engineering is unsupported.","section":"4.1.1 and A.2"},{"comment":"The comparison with AdaLead, PEX, and GGS is not a fair comparison of the methods as originally designed. Section 4.2.1 states that because the original AdaLead and PEX update their surrogate fitness predictor using ground-truth fitness from wet-lab experiments each round, the authors assume no access to ground-truth fitness and do not update the surrogate for these baselines. This removes the intended mechanism of those algorithms, whereas ASPO does not require ground-truth updates. The result is a comparison against degraded variants of the baselines. To support the claim that ASPO 'significantly outperforms all baselines' (Section 4.2.2), the baselines should either be run with their native feedback mechanism or the paper should explicitly frame the comparison as being against static-surrogate variants only. As presented, the optimization results do not demonstrate superiority in a realistic setting.","section":"4.2.1"},{"comment":"The premise verification in Section 3.1 uses t-SNE visualizations of PLM activations for proteins partitioned into positive and negative sets defined by the same surrogate predictors used later for steering. The observed cluster separation is therefore partly a consequence of the predictors' signal already being present in the PLM representations, and it does not independently establish that PLMs 'inherently encapsulate intrinsic knowledge' about the properties. The authors should demonstrate cluster separation using experimentally labeled data (e.g., Meltome or the GFP dataset) rather than the surrogate labels. This would provide a stronger and non-circular justification for the approach.","section":"3.1"}],"minor_comments":[{"comment":"The text references 'Fig. 4.2.2' in the sensitivity to steering strength discussion; this appears to be an incorrect figure reference, likely intended to be Figure 4(a) and Figure 4(d).","section":"4.3"},{"comment":"The column headers 'Dissiminit' and 'Dissimhigh' should be typeset as 'Dissim_init' and 'Dissim_high' for readability, and for consistency with the metric definitions in Section A.1.","section":"Tables 2 and 3"},{"comment":"For AE-PLMs, the generation procedure is described as randomly masking and regenerating 10% of tokens from a reference sequence (Section 4.1.1, Base Models). This is more accurately a mutagenesis or denoising procedure than de novo sequence generation; the terminology should be clarified to avoid overstating the generation capability.","section":"4.1.1"},{"comment":"The conclusion states that the method does not require 'explicit fitness predictors' because it does not use them during inference. However, the steering vectors are constructed using predictions from property-specific predictors. This should be acknowledged explicitly to avoid misleading readers about the reliance on proxy labels.","section":"5"},{"comment":"The paper does not state whether code will be released. For reproducibility, please provide code or detailed hyperparameters for the mutation site selection, especially the choice of layer ℓ used for relatedness scores.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The circular evaluation is a serious issue that will need substantive new experiments or a reframing of the claims, but the method itself is interesting and the problem is not inherently unfixable. I would encourage the editor to consider a revision that includes independent validation rather than outright rejection, provided the authors can address the baseline fairness and provide non-circular evidence. If independent validation is outside the authors' capacity, the claims should be narrowed to controlling surrogate predictors, which would considerably reduce the significance of the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Activation steering for PLMs: the paper takes ActAdd, computes steering vectors from mean activation differences between sequences a surrogate labels high and low, and injects them into hidden states at inference. They also introduce ASPO, which selects mutation sites by cosine similarity to the steering vector. The method is simple, training-free, and they test it on ESM2, ESM3, ProLLaMA, with sensitivity analysis. That's a genuine adaptation, and it will probably work as a quick bias knob.\n\nThe problem is the evaluation. The positive/negative sets are defined using predicted thermostability and solubility (Section 4.1.1), and 'fitness' is then measured with those same predictors (Section A.1). The steering vector points from 'predictor says low' to 'predictor says high,' so the reported gains are largely a self-fulfilling prophecy. GFP is the worst: both the task split and the evaluation predictor come from [13]. The thermostability predictor has Spearman 0.76; decent accuracy, but not ground truth. There is no wet-lab, no structure prediction, no held-out predictor anywhere. The abstract and conclusion claim 'precise protein engineering,' but the evidence shows control of surrogate scores. The conclusion even says the method works 'without explicit fitness predictors,' which is misleading—they use predictors to define the sets.\n\nThe optimization baseline comparison is also tilted. AdaLead and PEX are run without ground-truth fitness updates, which removes the mechanism they were designed for. So the comparison is 'who can exploit a static, biased surrogate best,' not 'who designs better proteins.' Diversity and novelty are nice, but they don't fix the core issue.\n\nI'd want the authors to either validate with an independent predictor or experiments, or reframe the claims as controlling predicted properties. As it stands, it's an incremental adaptation with a load-bearing flaw. Still, I'd let a serious referee look at it. The method is reproducible, the experiments are thorough, and the topic is timely. A referee can push for better validation or softer claims. So: send it out, but expect major revision or rejection if the circularity isn't addressed.\n\nWho should read it? People working on PLM controllability, as a baseline. I wouldn't cite it yet in my own work.","headline":"Activation steering for PLMs is a neat trick, but the evaluation is circular: the same predictors define the steering target and measure success.","tokens_in":16315,"tokens_out":5602,"would_cite":false,"duration_ms":62393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Activation steering—adding a mean-difference vector to a protein language model's internal activations during inference—enables training-free control of sequence generation and, with mutation-site selection, outperforms search-based…","keywords":["activation steering","protein language models","protein optimization","thermostability","solubility","GFP brightness","mutation site identification","training-free control"],"falsifier":"Measure the melting temperature, solubility, or fluorescence of ASPO-generated top-scoring sequences in a wet-lab assay; if the predicted gains over the initial set do not appear under direct experimental measurement, the central claim that steering pushes generation toward genuinely improved proteins collapses.","tokens_in":15355,"feed_emoji":"🧬","tokens_out":7828,"duration_ms":75907,"temperature":0.7,"pith_summary":"The paper claims that protein language models already encode information about properties such as thermostability, solubility, and GFP brightness in their internal representations, and that this knowledge can be tapped at inference time by adding a steering vector to the activations. The method, adapted from activation addition in text LLMs, shifts generation toward sequences with the desired property without any weight updates or fine-tuning. The paper extends this to protein optimization through ASPO, which first identifies the amino-acid positions most opposed to the target property and then regenerates them under steering. If the results hold, a lightweight, training-free control mechanism could replace or complement fine-tuning and search-based methods in protein engineering.","feed_headline":"Steering vectors direct protein AI toward desired traits","feed_subtitle":"Inference-time activation editing beats fine-tuning on thermostability, solubility, and GFP brightness tests.","key_machinery":"The central object is the steering vector $v_l$, computed at layer $l$ as the mean activation of the positive set minus the mean activation of the negative set, averaged over tokens for auto-encoding PLMs and using the last token for autoregressive PLMs. During inference the activation is edited as $\\tilde h_l = h_l + \\alpha v_l$ and renormalized to the original norm, with $\\alpha$ controlling steering strength. ASPO adds a relatedness score $s_k = \\cos(h_k^l, v_l)$ at the most informative layer, selected by linear classifier validation accuracy, and masks the $T$ tokens with lowest scores for regeneration under steering, repeated for $R$ rounds.","core_discovery":"On the paper's own terms, the central discovery is that activation steering works for protein language models across both auto-encoding architectures (ESM2, ESM3) and an autoregressive architecture (ProLLaMA). Steering vectors computed as the mean activation difference between positive and negative protein sets, added at every layer except the input and followed by norm rescaling, reliably raise predicted thermostability, solubility, and GFP brightness of generated lysozyme-like sequences while maintaining or improving diversity and novelty. For optimization, ASPO selects mutation sites by the cosine relatedness of token representations to the steering vector and then masks and regenerates those positions under steering, achieving the highest fitness in medium- and hard-difficulty optimization tasks for thermostability, solubility, and GFP brightness when compared with AdaLead, PEX, and GWG, while staying close to the initial sequence set.","pith_inferences":["We infer that the same recipe should transfer to other protein families and other measurable properties, such as binding affinity or immunogenicity, whenever a contrast set can be constructed from a predictor or an assay.","We infer that the strong dependence on surrogate predictors sets the practical ceiling of the method: if the predictors are smooth or biased in ways the steering vector can exploit, wet-lab validation is the only way to know whether the gains are real biological improvement.","The multi-property experiment in the appendix suggests steering vectors for different properties can be combined linearly, pointing toward compositional, multi-objective control of protein generation.","Because the method needs no training and only a few hundred labeled examples, it may make targeted protein design accessible to groups that lack the compute or data for fine-tuning large models."],"forward_implications":["Activation steering can be integrated into both auto-encoding and autoregressive protein language models without retraining, preserving the model's general knowledge while biasing outputs toward a target property.","Steering outperforms fine-tuning and the unmodified model on lysozyme-like thermostability and solubility generation across ProLLaMA, ESM2, and ESM3, and improves novelty and diversity in several settings.","ASPO achieves the highest fitness on thermostability, solubility, and GFP brightness optimization at both medium and hard difficulty, with lower dissimilarity to the initial set than the search-based baselines.","The method is sample-efficient, with 100 contrast sequences giving near-peak performance, and it is sensitive to steering strength, with over-steering at $\\alpha = 20$ collapsing solubility performance.","On ESM2-3B, activation steering yields larger gains than on ESM2-650M, indicating that larger protein language models leverage activation editing even more effectively."],"supporting_citations":[{"why":"Supplies the activation addition technique the paper adapts: steering vectors as mean differences and inference-time injection.","marker":"[40]"},{"why":"ESM2 is a base auto-encoding PLM used for generation and optimization, and the feature extractor for the thermostability predictor.","marker":"[17]"},{"why":"ESM3-open is the second auto-encoding PLM used to test steering and ASPO.","marker":"[8]"},{"why":"ProLLaMA is the autoregressive PLM whose superfamily control restricts generation to lysozyme-like sequences.","marker":"[20]"},{"why":"Provides the GFP brightness predictor, the GFP data split, and the GWG baseline ASPO is compared against.","marker":"[13]"},{"why":"Meltome Atlas data trains the thermostability predictor that defines positive and negative sets and evaluates fitness.","marker":"[12]"},{"why":"Supplies the independent solubility test set used to evaluate the solubility predictor.","marker":"[4]"},{"why":"PEX is a proximal search baseline for protein optimization used in the comparison.","marker":"[31]"},{"why":"AdaLead is the adaptive greedy search baseline used in the optimization comparison.","marker":"[33]"}],"fun_headline_variants":["Activation steering guides protein AI with no retraining","Steer protein language models to achieve target properties","Inference-time activation editing controls protein generation","Protein AI steered to hit thermostability and solubility targets","Activation steering offers training-free protein property control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the surrogate predictors used to define the positive and negative sets and to score the generated sequences—thermostability Spearman 0.76, solubility accuracy 0.708, and the GFP brightness predictor—are accurate enough that steering toward their labels and measuring success with the same predictors reflects real biological improvement.","fun_headline_variants_meta":{"raw":{"variants":["Activation steering guides protein AI with no retraining","Steer protein language models to achieve target properties","Inference-time activation editing controls protein generation","Protein AI steered to hit thermostability and solubility targets","Activation steering offers training-free protein property control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00052,"raw_usage":{"total_tokens":2470,"prompt_tokens":850,"completion_tokens":1620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":1546}},"tokens_in":466,"tokens_out":1620,"duration_ms":18000,"temperature":1.0,"reasoning_tokens":1546,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:04:36.916935+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the melting temperature, solubility, or fluorescence of ASPO-generated top-scoring sequences in a wet-lab assay; if the predicted gains over the initial set do not appear under direct experimental measurement, the central claim that steering pushes generation toward genuinely improved proteins collapses.","supporting_citations":[{"cited_title":"Activation addition: Steering language models without optimization","cited_arxiv_id":null,"evidence_quote":"Supplies the activation addition technique the paper adapts: steering vectors as mean differences and inference-time injection."},{"cited_title":"Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q","cited_arxiv_id":null,"evidence_quote":"ESM3-open is the second auto-encoding PLM used to test steering and ASPO."},{"cited_title":"Improving protein optimization with smoothed fitness landscapes","cited_arxiv_id":null,"evidence_quote":"Provides the GFP brightness predictor, the GFP data split, and the GWG baseline ASPO is compared against."},{"cited_title":"Meltome at- las—thermal proteome stability across the tree of life.Nature methods, 17(5):495–503, 2020","cited_arxiv_id":null,"evidence_quote":"Meltome Atlas data trains the thermostability predictor that defines positive and negative sets and evaluates fitness."},{"cited_title":"Bioinformatics approaches for improved recombinant protein production in escherichia coli: protein solubility prediction.Briefings in bioinformatics, 15(6):953–962, 2014","cited_arxiv_id":null,"evidence_quote":"Supplies the independent solubility test set used to evaluate the solubility predictor."},{"cited_title":"Proximal exploration for model-guided protein sequence design","cited_arxiv_id":null,"evidence_quote":"PEX is a proximal search baseline for protein optimization used in the comparison."}],"review_version":1}