After training, a 110M-parameter power-law attention model's learned scoring operator becomes nearly input-invariant, so inference can cache it; the paper proves this collapse conditionally, measures it at 1e-6 and below, and machine-checks selected proofs in Lean 4.
Power Law Graph Transformer for Machine Translation and Representation Learning
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We present the Power Law Graph Transformer, a transformer model with well defined deductive and inductive tasks for prediction and representation learning. The deductive task learns the dataset level (global) and instance level (local) graph structures in terms of learnable power law distribution parameters. The inductive task outputs the prediction probabilities using the deductive task output, similar to a transductive model. We trained our model with Turkish-English and Portuguese-English datasets from TED talk transcripts for machine translation and compared the model performance and characteristics to a transformer model with scaled dot product attention trained on the same experimental setup. We report BLEU scores of $17.79$ and $28.33$ on the Turkish-English and Portuguese-English translation tasks with our model, respectively. We also show how a duality between a quantization set and N-dimensional manifold representation can be leveraged to transform between local and global deductive-inductive outputs using successive application of linear and non-linear transformations end-to-end.
fields
cs.LG 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
After training, a 110M-parameter power-law attention model's learned scoring operator becomes nearly input-invariant, so inference can cache it; the paper proves this collapse conditionally, measures it at 1e-6 and below, and machine-checks selected proofs in Lean 4.