Pith. sign in

REVIEW 4 major objections 6 minor 2 references

Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?

T0 review · 4 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Tabular in-context models trained on synthetic tables become strong biomolecular predictors when paired with expressive fixed embeddings.

desk verdict Solid empirical audit: TabPFN3/TabICL + ESMC are competitive few-shot protein heads; the ProteinGym SOTA wording is ahead of the external verification the paper itself flags. read the letter →

arxiv 2606.31126 v2 pith:AC5FP4BX submitted 2026-06-30 cs.LG q-bio.QMstat.ML

Davy Guan , Lu Zhang , Asiri Wijesinghe , Allen Zhu , He Zhao , Helen Power , F. Hafna Ahmed , Andrew Warden
show 2 more authors
Cheng Soon Ong Daniel M. Steinberg
This is my paper · ORCID
classification cs.LGq-bio.QMstat.ML
keywords tabularfoundationmodelsin-contextlearningproteinfitnesspredictionsmall-moleculepropertyfew-shotGymESMCembeddingsECFPRDKitdescriptors
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Predicting protein fitness or small-molecule properties from few labels is a core bottleneck once good pretrained embeddings already exist. This paper shows that tabular foundation models—pretrained only on synthetic tables from random causal graphs—can fill that role surprisingly well. When sequences are encoded with fixed ESMC embeddings, the models match or exceed strong baselines on ProteinGym and a diverse esterase family, including in few-shot regimes. With ECFP/RDKit descriptors they stay competitive across ADMET, MoleculeNet, FS-Mol, and DrugOOD without dominating every family. The practical claim is that the bottleneck has shifted: choose a representation that already exposes local task structure, then let an amortized tabular in-context learner read support labels at inference time.

What carries the argument

The predictor–representation pair: a frozen domain encoder (ESMC for proteins; ECFP, RDKit, or both for molecules) produces fixed-length rows of a table; TabICL or TabPFN3 then performs in-context prediction by conditioning on labeled support rows without task-specific gradient updates.

What would settle it

On the same ProteinGym assays and support sizes, replace ESMC with a deliberately weak or random fixed embedding of equal dimension and check whether TabPFN3/TabICL still beat ridge and HistGradientBoosting; if they collapse to baseline or worse, the claim that the tabular prior transfers once geometry is good fails.

Watch

Extended reading notes

Core claim

Tabular foundation models (TabPFN3 and TabICL), despite a prior with no obvious biological correspondence, act as strong data-efficient predictors for biomolecular property tasks when treated as predictor–representation pairs: ESMC embeddings plus either model achieve competitive or better ProteinGym and PpEST protein-fitness results, and ECFP/RDKit pairings stay competitive with task-specific methods on small-molecule classification, especially under few-shot budgets.

Load-bearing premise

The fixed pretrained embeddings already place similar-labeled molecules near each other so a tabular prior trained only on synthetic causal tables can interpolate without learning new structure or using graphs.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper empirically tests whether tabular in-context learners (TabPFN3, TabICL), pretrained on synthetic causal tables, transfer to biomolecular property prediction when used as frozen predictors over fixed domain representations. For proteins, sequences are encoded with ESMC and evaluated on ProteinGym (217 DMS assays; random/modulo/contiguous holdouts; full-train and few-shot) and the PpEST esterase dataset. For small molecules, TabICL/TabPFN3/XGBoost are paired with ECFP, RDKit, or ECFP+RDKit descriptors and compared to ChemProp and ChemProp+CheMeleon across TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD (including ID/OOD gaps). The authors report strong protein results under fixed ESMC, competitive but non-dominant molecular results that depend heavily on representation, and conclude that tabular foundation models are viable biomolecular predictors only as learner–representation pairs.

Significance. The work addresses a practical bottleneck—data-efficient prediction once strong frozen encoders exist—and documents a counter-intuitive transfer of synthetic tabular ICL priors to protein and molecular feature tables. Strengths include multi-assay coverage, official ProteinGym split schemes, few-shot learning curves with explicit support-set protocols, paired TabPFN3–TabICL deltas, fixed-representation controls that isolate the predictor, DrugOOD OOD analysis, and an honest limitations section. If the protein results hold under external ProteinGym evaluation and careful claim language, the paper is a useful empirical audit for protein engineering and cheminformatics practice, even without a new architecture. The learner–representation-pair framing is methodologically clean and should influence how tabular foundation models are reported in scientific domains.

major comments (4)
  1. [Abstract; §4.1; Table 3] Abstract and opening claim that TabPFN3/TabICL + ESMC “achieve or exceed state-of-the-art results on ProteinGym,” but §4.1 and Table 3 explicitly treat public leaderboard numbers as “external context rather than as a new public ranking claim,” because submission packages still require ProteinGym’s external process. Local random 5-fold means (Table 1: Spearman 0.767/0.753) sit above the contextual public figures (Kermut 0.745, ProteinNPT 0.741), yet this ordering is not yet externally secured. The abstract’s SOTA language should be tempered to match the body’s caution until external evaluation is complete, or the external results should be included.
  2. [Abstract; Table 2; Figure 3; §7] Table 2 shows large drops under official modulo and contiguous schemes (TabPFN3 mean Spearman 0.744 random → 0.595 modulo → 0.522 contiguous). The paper correctly notes that random-split numbers are not the only generalization estimate, but the abstract and high-level protein claims still lead with the strongest random-split aggregates. Load-bearing claims should foreground split dependence (or report all three schemes as co-primary) so readers do not take random-split SOTA-style numbers as the main ProteinGym result.
  3. [§4.1; Limitations] Two of the largest ProteinGym assays (HIS7_YEAST_Pokusaeva_2019, SPG1_STRSG_Olson_2014) use PCA128 rescue (and 8 estimators for TabPFN3) rather than the same full-feature configuration as the remaining suite (§4.1; Limitations). Because these assays can move aggregate means, the paper should quantify sensitivity: report aggregates with and without the rescued assays, and/or with a uniform PCA setting across all methods, so the headline ProteinGym ranking is not partly an artifact of incomplete full-feature coverage.
  4. [Abstract; Table 7; §6; Limitations] For molecules, the abstract’s “competitive with the existing task-specific state-of-the-art” is only weakly anchored: Table 7 reports best internal pairs by family, MoleculeNet TabPFN3 coverage excludes PCBA (667/806), and there is little direct numerical comparison to published benchmark SOTA numbers (as opposed to the paper’s own ChemProp/XGBoost baselines). Either add a compact external-SOTA reference table per family or soften the abstract to “competitive with strong descriptor and graph baselines under our protocol,” which the body already supports.
minor comments (6)
  1. [Abstract; Table 5] The abstract says models “outperform task-specific supervised regressors on a diverse esterase catalytic activity dataset,” while Table 5 shows TabICL/TabPFN3 trading wins with each other and beating FT ESM/HGBR overall—fine, but “outperform” could specify rank vs MSE and that FT ESM is the strongest non-tabular baseline.
  2. [§4 Evaluation metrics; Figures 4, 6–8] Equations (1)–(3) define monotone best-so-far envelopes for plots; this is reasonable for legibility, but figure captions should state more prominently that hollow markers are raw means and that source tables retain full variability, so readers do not over-read envelope smoothness.
  3. [Table 1; §4.1] RBF sampler collapse on ESMC (Table 1 Spearman ≈ 0) is attributed to median-heuristic scaling; a one-sentence note on whether alternative kernel scales were tried would prevent the impression that kernel methods were dismissed after a single default.
  4. [§4 Datasets; References] PpEST is introduced via Ahmed et al. 2026 with overlapping authorship; a brief disclosure that this is a related experimental resource (not an independent third-party benchmark) would improve transparency without diminishing its value as a diverse-sequence test.
  5. [Abstract; §2] Minor consistency: the abstract uses “TabPFN” generically while the body standardizes on TabPFN3; align naming in the abstract and keywords.
  6. [Figure 5] Figure 5 (assay size vs Spearman) is under-interpreted; a short quantitative statement (e.g., correlation of performance with assay size) would make the diagnostic more useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pure empirical audit of pretrained tabular ICL models on external biomolecular benchmarks with frozen third-party representations; no derivation reduces to its own inputs.

full rationale

This paper contains no first-principles derivation, uniqueness claim, or fitted-parameter-as-prediction chain. Its load-bearing content is an empirical evaluation: frozen ESMC embeddings (Hayes et al.) or ECFP/RDKit descriptors are fed as fixed feature tables to off-the-shelf TabPFN3/TabICL (pretrained on synthetic causal tables by other groups), then scored by MSE/Spearman or ROC-AUC on public suites (ProteinGym, TDC ADMET, MoleculeNet, FS-Mol, DrugOOD) plus one additional esterase dataset. Performance numbers are obtained by direct measurement against held-out labels; they are not algebraically forced by any equation that re-uses the same quantities being predicted. The sole author-overlapping citation (Ahmed et al. 2026 for PpEST) supplies an independent experimental resource, not a theorem or ansatz that underwrites the method. Table 3’s leaderboard context is explicitly caveated as non-ranking, and PCA-rescue rows are disclosed as engineering necessities rather than definitional moves. Consequently the central claim—that tabular ICL + expressive representations is competitive—stands or falls on external numbers and is free of circular reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The work is empirical. Load-bearing premises are standard ML evaluation assumptions plus the domain choice to freeze pretrained embeddings/descriptors and treat tabular ICL as a black-box head. No new physical entities or fitted universal constants drive the central claim; free choices are experimental (support sizes, PCA rescue, feature concatenations).

free parameters (3)
  • few-shot support sizes {8,16,32,64} (proteins) and up to 512 (molecules)
    Hand-chosen low-data grid used for learning curves and headline few-shot claims; not derived from theory.
  • PCA128 compression + 8 estimators for two large ProteinGym assays
    Ad-hoc rescue configuration when full-feature TabPFN3/TabICL exceeded memory/time; affects completeness of the 217-assay aggregate.
  • molecular feature views (ECFP, RDKit, ECFP+RDKit)
    Descriptor choices treated as part of the model pair; rankings depend on this hand-selected feature set.
assumptions (4)
  • domain assumption Frozen ESMC 960-d embeddings capture transferable structural/functional constraints useful for fitness regression across DMS assays.
    Invoked throughout §3–4; all protein methods share this representation so predictor comparisons assume its adequacy.
  • domain assumption ECFP/RDKit fixed descriptors are sufficient tabular views of molecular structure for fair comparison to graph models on ADMET/activity tasks.
    §3 molecule features and §5–6; representation choice is acknowledged as first-order yet still underpins the competitive claim.
  • ad hoc to paper In-context prediction from synthetic causal-table pretraining can transfer to real biomolecular feature tables without task-specific gradient updates.
    Central transfer hypothesis stated in abstract/intro; tested empirically rather than derived.
  • standard math Standard supervised metrics (Spearman/MSE for fitness; ROC-AUC for classification) and public split protocols are appropriate success criteria.
    Used throughout evaluation sections; conventional for these benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?." pith.science (2026). https://pith.science/paper/AC5FP4BX

@misc{pith2026260631126,
  author       = {Pith},
  title        = {Pith review of: Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AC5FP4BX}},
  note         = {Machine review of arXiv:2606.31126}
}
read the original abstract

Predicting biomolecular properties from limited labeled data is a central bottleneck in protein engineering and small-molecule design. As strong pretrained encoders now supply rich fixed-length representations, the difficulty has shifted from representation learning to building a data-efficient predictor for the few-shot regime. Tabular foundation models such as TabPFN and TabICL are unlikely candidates for this role: they are in-context learners pretrained on synthetic tables drawn from random causal graphs, a generative prior with no obvious correspondence to the processes that produce protein sequences or molecular graphs. That this tabular, causal inductive bias should transfer to biomolecular data at all is counter-intuitive, yet we find it does. Treating each method as a predictor-representation pair, we evaluate across two domains. We find that on protein fitness regression tasks these in-context learning models coupled with ESM Cambrian representations achieve or exceed state-of-the-art results on ProteinGym, and outperform task-specific supervised regressors on a diverse esterase catalytic activity dataset. For small-molecule classification with ECFP/RDKit descriptors, no single predictor-representation pairing dominates across TDC ADMET, MoleculeNet, FS-Mol, and DrugOOD, but they are competitive with the existing task-specific state-of-the-art. Crucially, on both protein and small-molecule few-shot tasks, these predictor-representation pairs offer strong performance. We conclude that tabular foundation models can be strong biomolecular predictors, but only when coupled with expressive representations.

Figures

Figures reproduced from arXiv: 2606.31126 by the authors.

Figure 1
Figure 1. ProteinGym random 5-fold method comparison across 217 assays. Bars report mean Spearman [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Assay-level paired difference in validation performance between TabPFN3 and TabICL on Prote [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. ProteinGym performance across official random, modulo, and contiguous holdout schemes. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: ProteinGym few-shot performance as support-set size increases. Curves show best-so-far task [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: ProteinGym assay-size diagnostic. Mean Spearman is plotted against assay variant count for [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Few-shot PpEST performance measured by Spearman correlation. Bold curves are monotone best [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Few-shot PpEST performance measured by MSE. Bold curves are monotone best-so-far envelopes [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Few-shot small-molecule learning curves. The x-axis is the number of labeled support molecules. The [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Full-train small-molecule benchmark summary. Bars show mean ROC-AUC across tasks; DrugOOD [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: DrugOOD full-train ID/OOD generalization gap. Bars report ID-test ROC-AUC minus OOD-test ROC [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Paired few-shot comparison of TabPFN3 and TabICL on molecules. Positive values indicate higher [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

  1. [1]

    doi:10.64898/2026.06.02.729712. J. Burns, A. Zalte, and W. H. Green. Descriptor-based foundation models for molecular property prediction, 2025. Preprint, arXiv:2506.15792. T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 7...

    work page doi:10.64898/2026.06.02.729712 2026

  2. [2]

    doi:10.1039/C7SC02664A. K. Yang, K. Swanson, W. Jin, C. Coley, P. Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea, A. Palmer, V. Settels, T. Jaakkola, K. Jensen, and R. Barzilay. Analyzing learned molecular representations for property prediction.Journal of Chemical Information and Model- ing, 59(8):3370–3388, 2019. doi:10.1021/acs.jcim.9b...

    doi:10.1039/c7sc02664a 2019

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.