{"id":"4c098f47-eae2-4011-83ba-f0a71c3ae880","arxiv_id":"2505.12711","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A tri-modal pathology pretraining framework with any-to-any input flexibility, evaluated on four downstream tasks, but with an untested missing-modality claim and a data leakage concern.","lead":"This paper introduces ALTER, a pretraining method that aligns whole-slide pathology images, gene expression profiles, and diagnostic reports in one model. It reports performance gains on survival prediction, subtyping, mutation prediction, and report generation across ten datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ALTER's central any-to-any claim is unsupported by the experiments: pretraining uses only complete WSI-omic-report triplets and no evaluation measures performance under missing modality input, so the main contribution is an architectural assertion.","rationale":"The reader's weakest assumption is the untested missing-modality premise, and I agree it is load-bearing. The architecture may be modular, but transformer training is sensitive to input distribution; a sequence model that always sees three streams may not allocate attention correctly when one stream is absent, and modality-specific experts cannot produce outputs for an absent modality. No experiment addresses this. The leakage concern with LUAD strengthens the case for rejection but is secondary to the central claim. Thus the current evidence does not support 'any-to-any' pretraining, and the paper should not be accepted as is. The proposed dropout check would directly test whether the concern lands; since this does not change the reader's verdict of REJECT, I mark the verdict as UNCHANGED.","tokens_in":20060,"tokens_out":7389,"duration_ms":79818,"concrete_test":"Pretrain ALTER on the same 6,850 TCGA triplets with a randomized modality-dropout schedule (e.g., 20% WSI-only, 20% omics-only, 20% report-only, 20% WSI+omics, 20% complete triplets), keeping all losses and hyperparameters fixed. Then fine-tune and evaluate the four survival cohorts under every feasible input subset (WSI-only, omics-only, WSI+genomics, WSI+report, full triplets) with the same 5-fold protocol. Compare against the original tri-modal-only checkpoint. If the original checkpoint loses more than 3 points C-index when omics is dropped, or if dropout-pretraining changes average C-index by that margin, the missing-modality claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ALTER can pretrain with any subset of modalities and handle missing modalities at inference. Section 3.3 asserts that the two-stage fusion 'inherently supports missing modality scenarios' because attention is computed over the available subset, and Section 3.4 claims the three-tier losses enable training on arbitrary modality combinations. The experimental section never tests either half. Pretraining is described as 6,850 WSI-omic-report pairs, i.e., complete triplets; no run drops a modality during pretraining. The only partially observed uses are downstream unimodal tasks (subtyping, mutation, report generation), where WSI-only inputs are used with the fusion blocks frozen (Sec. 4.3, Fig. 4). Freezing sidesteps the distribution shift the paper claims to handle, because the shared attention layers have only been trained on full triplets and have never seen an absent stream at the fusion input. There is also no inference-time ablation that removes genomics or reports from the multimodal survival model. A compounding issue is that Sec. 4.1 says four validation cancer types were held out of pretraining, yet TCGA-LUAD is used for gene mutation prediction (Table 3) and LUAD is among the 29 TCGA pretraining sites, leaving Table 3 vulnerable to leakage. Together these gaps put the 'any-to-any' contribution and one of the four headline evaluations on unverified ground.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ALTER, a tri-modal pretraining framework for computational pathology that jointly models whole-slide images (WSIs), gene expression profiles, and diagnostic reports. The method uses modality-specific encoders (TransMIL for WSIs, a Performer for genes, BioBERT for reports), a universal sequence Transformer with shared attention followed by modality-specific expert decoupling, and three pretraining objectives: intra-modal masked language modeling, inter-modal CLIP-style contrastive alignment, and inter-sample triplet loss based on cancer type. The authors claim that this design enables pretraining and inference with any subset of modalities, and they evaluate ALTER on survival prediction, cancer subtyping, gene mutation prediction, and report generation across 10 public datasets, reporting superior or comparable performance to several baselines.","tokens_in":20389,"tokens_out":7308,"duration_ms":82306,"significance":"If the any-to-any capability were validated, ALTER would be a useful foundation model for computational pathology, addressing realistic missing-modality scenarios and unifying unimodal and multimodal downstream tasks. The paper has notable strengths: it evaluates on a broad set of public benchmarks, uses external datasets for cancer subtyping, holds out four TCGA cancer types for survival prediction, and compares against several recent state-of-the-art methods. However, the central any-to-any claim is not tested by the experiments, and the gene mutation prediction experiment is compromised by data leakage. The reported results may still indicate that multimodal pretraining improves downstream performance, but they do not currently substantiate the paper's main contribution.","major_comments":[{"comment":"No experiment in the paper trains or evaluates the model with partial modality sets. Pretraining is performed on 6,850 complete WSI-omic-report triplets; the MLM procedure that 'randomly select[s] a modality' still feeds all three modalities as input, only the masked-prediction target is chosen per epoch. The statement in §3.3 that 'the two-stage design inherently supports missing modality scenarios' is an architectural assertion, not a demonstrated property. The central claim of any-to-any learning therefore rests on an untested premise. Please add experiments that (i) pretrain with artificially dropped modalities, (ii) evaluate inference under modality dropout on a multimodal task such as survival prediction, and (iii) compare the any-to-any pretraining variant against complete-triplet-only pretraining.","section":"§3.3–3.4, Algorithm 1"},{"comment":"The gene mutation prediction experiment on TCGA-LUAD is not a held-out evaluation. Section 4.1 states that four cancer types—UCEC, STAD, KIRP, and KIRC—were held out for validation, while 'all data corresponding to the remaining 29 cancer types' were used in pretraining. LUAD is among those remaining cancer types, so the TP53 and EGFR mutation results in Table 3 can benefit from pretraining on the same patients' WSIs, gene expression, and reports. This leaks information and undermines one of the paper's four headline results. The authors should either exclude LUAD from pretraining and retrain, or replace this experiment with a truly held-out dataset.","section":"§4.1, Table 3"},{"comment":"For the unimodal downstream tasks—cancer subtyping, gene mutation prediction, and report generation—the fusion layers are frozen during fine-tuning. This design sidesteps exactly the distribution shift that the any-to-any claim concerns: the shared attention layers have never seen an absent stream at the fusion input, and the frozen-fusion setup does not exercise the fusion transformer with partial inputs. These experiments therefore do not provide evidence for missing-modality inference. The authors should report results with the fusion layers active under single-modality input, or explicitly justify why freezing is consistent with the claimed any-to-any capability.","section":"§4.3, Figure 4"}],"minor_comments":[{"comment":"The abstract contains typographical errors: 'and and additional modalities' should be 'and additional modalities', and 'cnnecessitating' should be 'necessitating'.","section":"Abstract"},{"comment":"The dataset description says 'Cervical Kidney Renal Papillary Cell Carcinoma (KIRP)'; KIRP stands for Kidney Renal Papillary Cell Carcinoma, so the word 'Cervical' appears to be an error.","section":"§4.1"},{"comment":"The contributions list refers to 'gene expression prediction', but the experiments and the rest of the text describe gene mutation prediction (TP53 and EGFR). Please align the terminology.","section":"§1, §4.1"},{"comment":"In Algorithm 1, the final loss is written as L = L_CLIP + L_triplet + L_MLM, but Section 3.4 defines L = αL_MLM + βL_CLIP + L_triplet with α and β set to 1. The algorithm should include α and β to be consistent with the main text.","section":"Algorithm 1"},{"comment":"Figure 4 lacks axis labels and numerical values, and the text does not state which metric is plotted. Please add explicit axes and quantitative results, preferably with error bars.","section":"Figure 4"},{"comment":"The survival results are reported as mean C-index with standard deviations, but no statistical significance tests are provided. Given that some confidence intervals overlap across methods, additional paired significance tests would strengthen the claim of consistent improvement.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central contribution of the paper is currently an assertion rather than a demonstrated capability. The missing-modality experiments and the LUAD leakage issue are serious, but both are fixable with additional experiments and retraining. I would be willing to reconsider a revised version that includes modality-dropout pretraining and inference evaluations, and that either removes LUAD from pretraining or replaces the mutation experiment with a clean held-out evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: ALTER is the first pathology pretraining framework I know that seriously attempts WSI + genomics + report fusion with a modality-flexible design, and several pieces are done well. The two-stage fusion with modality-shared attention plus modality-specific experts is reasonable; the region-level WSI aggregation and pathway-level gene pooling are sensible ways to cut sequence length; and the three-tier loss (MLM, CLIP, triplet) covers intra-, inter-, and sample-level signal. The evaluation is broad—survival, subtyping, mutation, report generation, ten datasets—and the gains over baselines are consistent, often by healthy margins. Using the same UNI features for all baselines is a fair touch, and the freeze-vs-finetune fusion ablation is the kind of analysis that helps readers.\n\nBut the load-bearing claim doesn't survive contact with the experiments. The paper says ALTER supports pretraining with any subset and robust inference with missing modalities. Section 3.3 states that the two-stage design 'inherently supports missing modality scenarios.' Yet pretraining is done only on 6,850 complete triplets; no run drops a modality during pretraining; no inference-time ablation removes genomics or reports from the multimodal survival model. The unimodal downstream tasks freeze the fusion blocks, so the shared attention layers are never tested on absent streams. The Algorithm 1 pseudocode includes conditionals for missing modalities, but those code paths are never exercised. That means 'any-to-any' is an architectural assertion, not a measured capability. The fix is straightforward—train on partial subsets and evaluate inference with dropout—so I wouldn't sink the paper over this alone, but as written the abstract overstates results.\n\nThere's also a real data-leakage problem in the mutation prediction. Section 4.1 says four cancer types were held out for validation; those are UCEC, STAD, KIRP, KIRC, all used for survival. TCGA-LUAD is used for TP53/EGFR mutation prediction, and LUAD is among the 29 pretraining sites. So Table 3 is likely inflated by pretraining on the same cancer type. The paper never mentions this. It needs to be rerun on a held-out cancer type or explicitly justified.\n\nMinor points: alpha=beta=1 with no sensitivity analysis; no code or model release mentioned; report generation is patch-level rather than slide-level, which limits the comparison. The limitations section honestly discusses single-source training but does not address the missing-modality evaluation gap.\n\nOverall: the paper is useful and the method is salvageable. I'd send it to peer review with a clear request: directly test missing modalities, fix or re-frame the mutation experiment, and temper the 'any-to-any' language to match evidence.","headline":"ALTER is a well-engineered tri-modal pathology pretraining pipeline, but the 'any-to-any' claim is never actually tested and one evaluation leaks pretraining data; worth peer review, not acceptance as-is.","tokens_in":20918,"tokens_out":2834,"would_cite":false,"duration_ms":29473,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ALTER claims that triplet pretraining on whole-slide images, genomic profiles, and pathology reports produces a single model that accepts any subset of modalities and transfers to subtyping, survival, mutation, and report-generation tasks.","keywords":["computational pathology","multimodal pretraining","whole-slide images","genomic profiles","pathology reports","any-to-any learning","contrastive learning","missing modalities"],"falsifier":"Fine-tune ALTER for survival prediction with one modality absent, for example WSI plus reports but no genomics, and compare against ALTER with full inputs and against unimodal baselines. If the model cannot beat a unimodal baseline or collapses under modality dropout, the claim that the architecture inherently supports missing modalities would be falsified.","tokens_in":19859,"feed_emoji":"🧬","tokens_out":8959,"duration_ms":92989,"temperature":0.7,"pith_summary":"The paper proposes ALTER, a pretraining framework that learns from three modalities at once: whole-slide images, genomic expression profiles, and pathology reports. ALTER is designed so that a single pretrained model can later be fine-tuned for tasks that use any one of these modalities or any combination of them, rather than requiring a separate model per task. The authors claim this any-to-any flexibility follows from a two-stage fusion transformer, shared attention followed by modality-specific experts, together with three pretraining objectives that align modalities pair by pair and by cancer type. If the claim holds, it would let clinical models be built from partially paired data, where some patients have slides and reports but no genomics, or slides only. Under the paper's view, this matters because real pathology datasets are fragmented, yet downstream tasks range from image-only subtyping to multimodal survival prediction.","feed_headline":"Pretraining on slides, genes, and reports yields one any-to-any model","feed_subtitle":"One pretrained model beats or matches baselines on survival, subtyping, mutation, and report-generation tasks.","key_machinery":"The mechanism is a modality-adaptive fusion stack. Each modality receives its own encoder: a two-layer transformer with region-wise aggregation for whole-slide images, a pathway-grouped gene encoder for expression profiles, and a biomedical language model for diagnostic reports. Their outputs are concatenated and passed through a universal sequence transformer with two stages per layer: modality-shared self-attention followed by modality-specific mixture-of-experts decoupling. Three pretraining losses operate at three levels: masked-modality reconstruction within each modality, CLIP-style contrastive alignment among every pair of modality [CLS] tokens, and triplet loss on cancer type using a sample-level token. The two-stage fusion is the piece intended to carry the any-to-any property: attention runs over whatever modalities are present, and each expert decodes that modality's structure from the shared latent space.","core_discovery":"The central discovery asserted is that triplet pretraining over WSI, genomics, and report text produces representations that transfer to both unimodal and multimodal downstream tasks better than task-specific supervised baselines. On four TCGA survival cohorts, ALTER reaches a mean C-index of 0.762, outperforming the second-best method by the margin reported in the paper, and it improves cancer subtyping in three of four benchmarks, gene mutation prediction for both TP53 and EGFR, and all six report-generation metrics. The authors also show that freezing the fusion layers helps unimodal downstream tasks while fine-tuning them helps multimodal survival prediction, which they interpret as evidence that cross-modal knowledge is stored in the fusion blocks. On the paper's own terms, these results demonstrate that a pretrained any-to-any model can handle image-only, image-plus-genomics, and image-to-text tasks without being redesigned for each task.","pith_inferences":["The missing-modality claim is architectural and not yet demonstrated: Section 3.3 asserts that the two-stage design inherently supports partial inputs, but all reported pretraining uses complete triples, and no experiment trains with a modality dropped or evaluates inference under modality dropout.","The authors' own limitation section notes that pretraining is confined to TCGA; testing on external or multi-institutional datasets would show whether the learned alignment survives scanner, staining, and reporting-style variation.","The freeze/fine-tune experiment implies the fusion layers hold task-relevant cross-modal knowledge even for unimodal use; probing those layers with attention analyses could turn the reported heatmap observation into a causal story about what multimodal pretraining stores.","The same two-stage shared-attention-plus-experts recipe could be dropped onto other modality pairs, such as radiology images, clinical notes, or immunohistochemistry stains, as a general fusion pretraining pattern; the paper sketches but does not test that generality."],"forward_implications":["A single ALTER checkpoint can be fine-tuned for image-only tasks (cancer subtyping, gene mutation prediction), image-plus-genomics tasks (survival prediction), and image-to-text tasks (report generation), replacing task-specific architectures with one shared fusion backbone.","Pretraining can proceed from partially paired data: if the any-to-any design works as claimed, any subset of WSI, genomics, and report pairs contributes to the same model, reducing the need for complete tri-modal alignment in clinical datasets.","The gains on report generation, with BLEU-1/2/3/4, METEOR, and ROUGE-L all improved, suggest that seeing genomics and slides together during pretraining improves later image-to-text generation even when only the WSI is available at fine-tuning time.","The survival-prediction results across four cancer types support the claim that cross-modal pretraining transfers to cancer types not seen during pretraining, with average C-index 0.762."],"supporting_citations":[{"why":"Provides the pretrained patch encoder that turns every WSI into the slide feature bag used throughout the paper.","marker":"[6]"},{"why":"Provides the two-layer transformer aggregator used to compress slide patches into a [CLS] token and region features.","marker":"[14]"},{"why":"Provides the masked gene encoder used to embed discrete gene-expression tokens before pathway pooling.","marker":"[57]"},{"why":"Provides the biomedical language model used to embed diagnostic report tokens.","marker":"[58]"},{"why":"Defines the paired contrastive loss that aligns every pair of modality [CLS] tokens during pretraining.","marker":"[59]"},{"why":"Supplies the pathway-token recipe for genomic aggregation and is the strongest multimodal survival baseline ALTER is compared against.","marker":"[4]"},{"why":"Supplies the pathway-level gene grouping used to pool thousands of genes into a few hundred pathway features.","marker":"[37]"},{"why":"Supplies the report generation framework, the TCGA report preprocessing convention, and the unimodal generation baseline.","marker":"[68]"},{"why":"Supplies the visual-language report generation baseline with local-global hierarchical encoding that ALTER outperforms.","marker":"[40]"},{"why":"Defines the closest three-modal prior work that ALTER positions itself against: it uses all three modalities but lacks flexible any-to-any input.","marker":"[3]"}],"fun_headline_variants":["Any-to-any pretraining makes one model fit all pathology tasks","Triplet pretraining fuses slides, genes, and text for pathology","One pretrained model matches or beats baselines in pathology","Adaptive pretraining handles missing modalities and multiple tasks","ALTER: any-to-any triplet pretraining for pathology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The any-to-any promise rests on an untested premise: the two-stage fusion transformer works correctly when one or two modalities are missing, even though every pretraining run and every reported experiment uses complete triples or fixed modality sets.","fun_headline_variants_meta":{"raw":{"variants":["Any-to-any pretraining makes one model fit all pathology tasks","Triplet pretraining fuses slides, genes, and text for pathology","One pretrained model matches or beats baselines in pathology","Adaptive pretraining handles missing modalities and multiple tasks","ALTER: any-to-any triplet pretraining for pathology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000906,"raw_usage":{"total_tokens":3892,"prompt_tokens":939,"completion_tokens":2953,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2868}},"tokens_in":555,"tokens_out":2953,"duration_ms":20298,"temperature":1.0,"reasoning_tokens":2868,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:28:10.810013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune ALTER for survival prediction with one modality absent, for example WSI plus reports but no genomics, and compare against ALTER with full inputs and against unimodal baselines. If the model cannot beat a unimodal baseline or collapses under modality dropout, the claim that the architecture inherently supports missing modalities would be falsified.","supporting_citations":[{"cited_title":"Song, Richard J","cited_arxiv_id":null,"evidence_quote":"Supplies the pathway-level gene grouping used to pool thousands of genes into a few hundred pathway features."},{"cited_title":"Wsicaption: Multiple instance generation of pathology reports for gigapixel whole-slide images","cited_arxiv_id":null,"evidence_quote":"Supplies the report generation framework, the TCGA report preprocessing convention, and the unimodal generation baseline."},{"cited_title":"Histgen: Histopathol- ogy report generation via local-global feature encoding and cross-modal context interaction","cited_arxiv_id":null,"evidence_quote":"Supplies the visual-language report generation baseline with local-global hierarchical encoding that ALTER outperforms."}],"review_version":1}