{"id":"09ea2eab-1e68-4bd0-bb6a-03ac7f6f5ab0","arxiv_id":"2505.20052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Ankh3 shows that combining multiple masking probabilities with sequence completion improves protein language model performance downstream, but the causal role of multi-task pretraining is not proven by a matched ablation.","lead":"A new protein language model, Ankh3, is pretrained on two objectives at once: masked language modeling at three masking rates and sequence completion. It reports gains over earlier sequence-only protein models on four benchmarks, though the comparisons lack a same-scale control for the multi-task claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central causal claim that multi-task pretraining enhances representations is unsupported: no same-scale single-task control is run, so gains could stem from larger scale, longer training, or the evaluation pipeline rather than the multi-task objective.","rationale":"I focused on the missing same-scale single-task ablation because the title and abstract make a strong causal claim ('Multi-Task Pretraining ... Enhances Protein Representations'), and this control is the one experiment that would directly support or refute it. The baseline-pipeline mismatch identified by the reader is also serious, but it primarily affects the magnitude of the reported gains rather than the causal mechanism. My reading also surfaced an internal tension: for Ankh3-Large, the model is worse than the original Ankh Large on several metrics in Table 5, so the Section 4 sentence 'Ankh3 performed better in all tasks' is not true for the smaller model. This reinforces that the claimed benefit may be scale-related rather than objective-related. The proposed test, a same-scale and same-budget MLM-only control evaluated under the identical benchmarking pipeline, would settle whether the multi-task objective itself adds value and would also remove the pipeline confound for that comparison. I therefore agree with the reader's conditional verdict: the released model and weights are useful assets, but the central causal claim requires this control before it can be accepted as demonstrated.","tokens_in":7870,"tokens_out":6805,"duration_ms":59044,"concrete_test":"Pre-train a matched control with the Ankh3-XL architecture, the same UniRef50 data and training budget (approximately 5M steps, batch size 1024, sequence length 512), but using only the MLM objective with the three masking rates (15%, 20%, 30%) and no completion task. Freeze the encoder and evaluate on SSP-3 and SSP-8 for CASP12 and CASP14, fluorescence, GB1, and contact prediction with the exact ConvBERT pipeline and the three seeds used for Ankh3. If the multi-task model does not outperform this control on a clear majority of the eight metrics, the central claim that multi-task pretraining enhances representations is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4 assert that jointly optimizing MLM at 15%/20%/30% masking plus 50% sequence completion makes Ankh3 representations richer and more generalizable. The only evidence is Table 5, which compares Ankh3-L/XL against Ankh and ESM2 numbers 'sourced from the original Ankh paper' (Section 3.1), i.e., produced with a different downstream head/pooling, and against ESM3 run by the authors. No experiment isolates the pretraining objective: Ankh3 is never compared with a same-architecture, same-data, same-budget control trained on MLM only or completion only. The scaling argument in Section 4 (Ankh3-XL improves over Ankh3-L unlike ProtT5-XL to ProtT5-XXL) cannot substitute for an ablation because ProtT5 differs in data, architecture, and training setup. Moreover, the paper's own numbers undermine the unqualified claim that 'Ankh3 performed better in all tasks': Ankh3-L NLU gets 78.03 on CASP12 SSP-3 versus 80.81 for Ankh Base and 83.59 for Ankh Large, and trails Ankh Large on ProteinNet P@L/5 (69.42 vs 73.49) and CASP14 P@L/5 (26.65 vs 29.91). Thus the benefit is at best scale-dependent, and the causal attribution to the multi-task objective is untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Ankh3, an encoder-decoder protein language model pre-trained on UniRef50 with a multi-task objective combining masked language modeling at three masking probabilities (15%, 20%, 30%) and a 50% sequence-completion task. Two model sizes are trained (Ankh3-Large and Ankh3-XL), with architectures based on T5 and training details fully specified. The models are evaluated on secondary structure prediction (CASP12, CASP14), fluorescence (TAPE), GB1 fitness (FLIP), and contact prediction (ProteinNet, CASP14), reporting three-seed averages for NLU and S2S token prompts. The authors claim that this multi-task pre-training yields richer, more generalizable protein representations from sequence-only input and that Ankh3 outperforms previous sequence-only PLMs such as Ankh and ESM2, while being competitive with ESM3 on structure-related tasks.","tokens_in":8137,"tokens_out":5253,"duration_ms":55705,"significance":"If the causal claim were established, this would be a practically valuable result: it would show that a purely sequence-based multi-task objective can improve protein representations without adding structural or other modalities, and the paper openly releases model weights, hyperparameters, and dataset links. The three-seed averaging with standard deviations for Ankh3 is good experimental hygiene, and the comparison against ESM3, despite its variability, is informative. However, the central attribution of the reported gains to the multi-task objective is not identified in any controlled experiment, and several internal results contradict the unqualified claim that Ankh3 performs better in all tasks. The manuscript therefore documents a promising model and a plausible hypothesis, but the evidence as presented does not yet support the paper's headline conclusion.","major_comments":[{"comment":"The paper's central causal claim—that jointly optimizing MLM at multiple masking probabilities and sequence completion yields richer representations—is not supported by a controlled experiment. Ankh3 is never compared with a same-architecture, same-data, same-training-budget model trained on MLM alone or on completion alone; every comparison in Table 5 changes at least one of architecture, scale, data, or downstream pipeline. The ProtT5 scaling comparison invoked in §4 cannot serve as a control because ProtT5 differs in data, architecture, and training setup. Please train at least an Ankh3-Large MLM-only baseline and an Ankh3-Large completion-only baseline, matching the multi-task run in data, optimizer, and step count, and report them in Table 5.","section":"§3.1, Table 5; §4"},{"comment":"The Ankh and ESM2 baseline numbers in Table 5 are \"sourced from the Ankh paper\" and were produced with a different downstream evaluation pipeline than the ConvBERT head with global average pooling described in §3 and Table 4. Because downstream head and pooling choices can change results substantially, differences in Table 5 may reflect the evaluation pipeline rather than the pretraining objective. Please rerun the sequence-only baselines (at minimum Ankh Base/Large and ESM2-650M) through the identical evaluation pipeline used for Ankh3, and report those numbers alongside.","section":"§3.1, Table 5 footnote"},{"comment":"Several statements in the Abstract and §4 (\"Ankh3 performed better in all tasks\", \"multi-task pre-training demonstrated... improved performance in downstream tasks\") are contradicted by the paper's own Table 5. Ankh3-L NLU scores 78.03 on CASP12 SSP-3 versus 83.59 for Ankh Large, 69.42 versus 73.49 on ProteinNet P@L/5, and 26.65 versus 29.91 on CASP14 P@L/5. The benefit is therefore task- and scale-dependent, and the unqualified claims should be revised and, where possible, accompanied by significance tests across seeds.","section":"Table 5; Abstract; §4"},{"comment":"The sentence in §4 attributing the performance boost to \"the addition of multiple masking probabilities and sequence completion\" as the \"main contributors\" is a causal attribution that the experiments do not establish, because no ablation isolates either factor and the comparison to ProtT5 changes multiple variables. Please remove this attribution or rephrase it as a hypothesis to be tested in future work.","section":"§4"}],"minor_comments":[{"comment":"The line \"encoder-decoder models ()\" contains an empty citation; please add the intended reference.","section":"§1"},{"comment":"With embedding dim 1536 and 16 heads, the per-head dimension is 96, not the listed KV dim 64; for the XL model it would be 80. Please clarify whether KV dim denotes a different quantity (e.g., a grouped-query setting) or correct the table.","section":"Table 1"},{"comment":"The row label \"ProteinNet (L/1)\" should be \"P@L\" to match the text's definition of Precision@L; as printed it is easy to misread as a length-1 normalization.","section":"Table 5"},{"comment":"The text says \"the average performance of each model with both [NLU] and [S2S] tokens\" but Table 5 reports separate columns; please clarify whether the average is over seeds only, with tokens shown separately, or whether the two columns themselves are averaged.","section":"§3.1"},{"comment":"The ProtT5 numbers quoted in §4 (84.4%, 79.2%, 81.4%) are not presented in a table or directly sourced in this manuscript; please add a citation or reproduce the relevant source values for verifiability.","section":"§4"},{"comment":"For reproducibility of the completion task, please specify how the 50% split is performed at the token level, how odd-length sequences are handled, and how sentinel tokens are introduced in the completion task.","section":"§2.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent about the source of baselines and releases model weights, which is commendable. The main issue is evidentiary: the headline causal claim is untested because no same-scale single-task ablation is run, and the baseline comparison pipeline differs. This is fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would not recommend accepting in the current form because the abstract and discussion assert more than the data show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what's worth taking from this paper: they train two T5-style protein models with a multi-task objective borrowed from UL2—MLM at three masking rates plus 50% sequence completion—and release the weights. That's a concrete, reproducible resource for the protein LM community. Training details are documented well, they report three seeds, and the comparison to ESM3, while noisy, is a reasonable thing to include. The observation that S2S helps on classification tasks like GB1/fluorescence while NLU helps on structure prediction is a useful empirical lead. The citation pattern is also fine; prior work, especially UL2 and the Ankh line, is credited properly.\n\nThe soft spots are real and fairly large. The central claim in the abstract and Section 4—that multi-task pretraining produces richer, more generalizable representations—is never actually tested. There is no same-scale, same-data control trained with MLM only or completion only. The scaling argument in Section 4 does not substitute for an ablation because ProtT5 differs in data, architecture, and setup. On top of that, the baselines for Ankh and ESM2 are lifted from the original Ankh paper, so they come from a different evaluation pipeline (different head, pooling, and training details). And the text says 'Ankh3 performed better in all tasks' against sequence-only models, but their own Table 5 shows Ankh3-L NLU below Ankh Base/Large on multiple CASP structure and contact benchmarks. That is not a minor wording issue; it is the headline result.\n\nThe ESM3 comparison also deserves a caution flag: they rely on one seed (42) producing very high variance to claim competitiveness on contact prediction. That is within the data they report, so it is not cherry-picking exactly, but the conclusion is fragile. No evaluation code is released either, which makes the baseline comparison hard to reproduce.\n\nWhat the paper does establish: a specific training recipe at 1.9B and 5.7B scale yields competitive or better numbers than ESM2 and Ankh on several tasks, and the weights are public. That is worth having. But it does not establish that the multi-task objective is the cause.\n\nFor peer review: yes, send it out. The release is serious, the idea is timely, and the community would benefit from a version where the causal claim is either backed by a proper ablation or downgraded to a descriptive statement. The authors should add at least one same-scale single-task control, rerun the Ankh/ESM2 baselines in their own pipeline, fix the 'all tasks' sentence, and release evaluation code. I would engage with a revised version.","headline":"Useful model release and an interesting pretraining recipe, but the causal claim is untested and the paper overstates its own results.","tokens_in":8692,"tokens_out":2772,"would_cite":true,"duration_ms":28343,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Protein language models trained jointly on masked denoising and sequence completion learn richer representations than single-task models.","keywords":["Ankh3","protein language model","multi-task pretraining","masked language modeling","sequence completion","secondary structure prediction","contact prediction","fitness prediction"],"falsifier":"Re-run Ankh and ESM2 through the exact ConvBERT evaluation pipeline used for Ankh3, or train an Ankh3-sized model with only a single masking probability and no completion task. If the Table 5 gaps shrink to noise or the single-task model matches Ankh3, the multi-task claim is not supported.","tokens_in":7667,"feed_emoji":"🧬","tokens_out":9687,"duration_ms":94704,"temperature":0.7,"pith_summary":"Ankh3 asks whether a protein language model trained on more than one self-supervised objective can learn richer representations from protein sequences alone. The paper argues that it can: jointly optimizing masked language modeling at 15%, 20%, and 30% masking with a sequence-completion task that predicts the second half of a protein from the first half produces embeddings that outperform the earlier single-task models Ankh and ESM2 on secondary structure prediction, fluorescence, GB1 fitness, and contact prediction. Against ESM3, which was pretrained with structure tokens, Ankh3 is competitive on secondary structure and contact tasks and substantially better on fluorescence and GB1 fitness, tasks neither model saw during pretraining. If this is right, multi-task pretraining is a route to stronger protein representations without adding data modalities.","feed_headline":"Combining denoising and completion boosts protein model accuracy","feed_subtitle":"Sequence-only Ankh3 rivals multimodal ESM3 on structure tasks and beats it on fluorescence and fitness.","key_machinery":"The load-bearing mechanism is a two-task pretraining objective based on the UL2 scheme, implemented in an encoder-decoder T5 architecture. The first task is masked language modeling in which tokens are masked at 15%, 20%, or 30% rates, sampled uniformly per step, and the model is trained to reconstruct the corrupted spans. The second task is sequence completion, in which the encoder reads the first half of a protein sequence and the decoder must generate the remaining half. Each sequence is randomly assigned to one of the two tasks every time it is sampled, so the same protein is seen under both denoising and generation objectives over the course of training. At evaluation, an [NLU] or [S2S] token is prepended, and the frozen backbone representation is pooled and fed to a fixed ConvBERT head.","core_discovery":"On the paper's own terms, the central discovery is that combining variable-rate masked denoising with completion of the missing half of a sequence produces a protein language model whose frozen representations transfer better than those of prior sequence-only protein language models. The evidence is a set of benchmarks in which Ankh3-Large and Ankh3-XL beat the published Ankh and ESM2 scores on every task tested, and in which Ankh3-XL outperforms the multimodal ESM3 on fluorescence and GB1 fitness while matching it on most secondary-structure and contact entries. The authors read the scaling behavior as supporting the same conclusion: Ankh3-Large to Ankh3-XL improves on both objectives, whereas the earlier ProtT5 line plateaued when scaled, so the gains are attributed to the multi-task objective and variable masking rather than to model size alone.","pith_inferences":["The paper does not isolate how much of the gain comes from multi-task training versus simply varying the masking probability; a same-scale ablation of single-rate MLM, multi-rate MLM, and completion alone would settle that.","The large seed-42 variance reported for ESM3 suggests that single-seed comparisons with strong multimodal baselines can be misleading, so more seeds would make the competitive claim sturdier.","If sequence completion is the active ingredient, variable completion percentages and longer contexts are a natural next experiment that could improve both generation and embedding quality.","The same two-task recipe could transfer to other biological sequence domains, such as RNA or antibody repertoires, where denoising alone has been the default pretraining objective."],"forward_implications":["If the central claim is right, sequence-only protein language models can be improved by adding a second self-supervised task instead of adding structure or evolution data.","The mixture of 15%, 20%, and 30% masking implies that no single masking rate dominates across downstream tasks, making variable denoising a reasonable default for future pretraining.","Because Ankh3-XL improves over Ankh3-Large on both objectives, the multi-task setup appears to give larger models useful capacity to exploit, unlike the ProtT5 scaling plateau the paper cites.","The observation that [NLU] and [S2S] tokens lead to different rankings across tasks means users may need to select the task token per application.","Strong fluorescence and GB1 fitness results suggest the multi-task representation transfers to fitness landscapes that were not part of pretraining."],"supporting_citations":[{"why":"supplies the Ankh architecture, the published baseline scores in Table 5, and the earlier result that different masking probabilities suit different tasks.","marker":"(Elnaggar et al., 2023)"},{"why":"defines the UL2 multi-task objective whose R-denoiser and S-denoiser tasks Ankh3 adopts.","marker":"(Tay et al., 2022)"},{"why":"provides the T5 encoder-decoder architecture and denoising formulation that Ankh3 is built on.","marker":"(Raffel et al., 2020)"},{"why":"is the multimodal ESM3 model that Ankh3 is compared against on structure-related benchmarks.","marker":"(Hayes et al., 2024)"},{"why":"defines the UniRef50 clustering used to construct the pretraining dataset.","marker":"(Suzek et al., 2015)"},{"why":"supplies the TAPE fluorescence dataset used as a downstream evaluation task.","marker":"(Rao et al., 2019)"},{"why":"supplies the FLIP GB1 fitness landscape benchmark used for evaluation.","marker":"(Dallago et al., 2022)"},{"why":"provides the CASP14 data used for secondary structure and contact prediction benchmarks.","marker":"(Kryshtafovych et al., 2021)"}],"fun_headline_variants":["Multi-task pretraining: denoising plus completion sharpens protein models","Protein language model gains from joint denoising and completion","Ankh3: dual objectives better protein representations","Multi-task pretraining yields stronger protein embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Ankh and ESM2 scores quoted from the earlier Ankh paper are directly comparable to the new Ankh3 scores, even though they come from a different downstream evaluation setup, and that the performance gap is caused by the pretraining objective rather than by model scale or other training changes.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task pretraining: denoising plus completion sharpens protein models","Protein language model gains from joint denoising and completion","Ankh3: dual objectives better protein representations","Multi-task pretraining yields stronger protein embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000602,"raw_usage":{"total_tokens":2782,"prompt_tokens":886,"completion_tokens":1896,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":502,"tokens_out":1896,"duration_ms":14092,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:59:48.299991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Ankh and ESM2 through the exact ConvBERT evaluation pipeline used for Ankh3, or train an Ankh3-sized model with only a single masking probability and no completion task. If the Table 5 gaps shrink to noise or the single-task model matches Ankh3, the multi-task claim is not supported.","supporting_citations":[],"review_version":1}