Averaging top-K attention heads selected from 10 proteins recovers contacts in one forward pass and matches or exceeds the Categorical Jacobian on bidirectional protein LMs.
Martin Steinegger and Johannes Söding
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
fields
cs.LG 2years
2026 2verdicts
UNVERDICTED 2representative citing papers
Case study applies SAE probing with enstrophy triage to a continuum-dynamics foundation model and reports intermittent feature consistency that does not align with standard physics while linking some output discrepancies to specific feature changes.
citing papers explorer
-
Protein contacts are already in the attention: a single-forward-pass alternative to the Categorical Jacobian
Averaging top-K attention heads selected from 10 proteins recovers contacts in one forward pass and matches or exceeds the Categorical Jacobian on bidirectional protein LMs.
-
Sparse probes and murky physics: a case study of interpretability challenges in a foundation model for continuum dynamics
Case study applies SAE probing with enstrophy triage to a continuum-dynamics foundation model and reports intermittent feature consistency that does not align with standard physics while linking some output discrepancies to specific feature changes.