Pith. sign in

REVIEW 4 major objections 7 minor 35 references

SAOBP adds one-step belief propagation to self-attention, suppressing attention collapse and improving accuracy in small Transformers; gains are strongest below 50M parameters.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 22:23 UTC pith:PPSYQZJN

load-bearing objection A repulsive attention regularizer that gives small-model gains, wrapped in an unsupported multi-hop story. the 4 major comments →

arxiv 2509.07324 v1 pith:PPSYQZJN submitted 2025-09-09 cs.CL cs.AI

Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation

Nakyung Lee , Yeongoon Kim , Minhae Oh , Suhwan Kim , Jin Woo Koo , Hyewon Jo , Jungwoo Lee This is my paper
classification cs.CL cs.AI
keywords self-attention localizationattention entropy collapsebelief propagationrepulsive Potts priorGlobal Token Dependencysmall-scale Transformersmulti-hop dependenciesattention regularization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to fix "attention localization"—the tendency of Transformer self-attention to concentrate on very few tokens and lose long-range context—and argues that one round of belief propagation can inject the missing global structure. The proposed mechanism, SAOBP with the BP-High variant, treats each row of the attention matrix as a node in a factor graph, passes messages between rows through a repulsive Potts compatibility function, and replaces the attention distribution with the resulting one-step belief. Because the refinement targets the self-attention computation itself, models must be pretrained with it. Across BERT-Mini, BERT-Small, BERT-Medium, and decoder-only GPT-2 variants, the paper reports higher attention entropy, consistently better accuracy on GLUE, SQuAD, HellaSwag, and RACE-Middle, and gains that are largest in the smallest models. A companion diagnostic, Global Token Dependency (GTD), quantifies the multi-hop attention mass and correlates with task performance, giving practitioners a layer-wise readout of where attention collapses.

Core claim

The paper's central claim is that a single belief-propagation step applied to raw attention scores—before the value aggregation—is enough to give Transformer attention a globally aware, multi-hop character. The update multiplies each attention row by messages aggregated from every other row, with a repulsive Potts prior that rewards attention on dissimilar tokens and thwarts concentrated, low-entropy distributions. The authors argue that this one-step process, run alongside standard parameter updates during pretraining, prevents entropy collapse in deeper layers, keeps GTD at task-appropriate levels, and thereby improves downstream performance—most visibly in models under roughly 50M paramet

What carries the argument

The load-bearing object is a factor graph built from the attention matrix: each token row is a variable node, each row pair shares a factor node whose potential is a repulsive Potts term ψ(r,k)=exp(λ) for r≠k and 1 for r=k. One step of message passing computes messages m_{f_ij→j}(k)=A_ik+e^λ(1−A_ik), and the final belief is the normalized product of every such incoming message with the original row A_jk. This closed-form update is what injects all-to-all context at negligible overhead. The companion diagnostic GTD(A)=‖Σ_{t=2}^K β^{t−1} A^t‖_F^2/(‖A‖_F^2+‖G‖_F^2) measures the relative mass of paths of length at least two, letting the authors detect per-layer attention collapse and show that S

Load-bearing premise

The method assumes that one round of message passing between attention rows carries true multi-hop (global) information; if a single step merely spreads attention mass without encoding indirect token relationships, SAOBP would be a mild regularizer rather than a global-context mechanism.

What would settle it

Take a model pretrained with SAOBP and, on the same inputs, compare the refined attention matrix to the first few powers of the original attention matrix A^t, which encode genuine multi-hop paths. If the SAOBP attention assigns no more mass to token pairs connected by two- or three-step paths than to unconnected pairs once row sums are matched, the claim that one-step belief propagation injects multi-hop dependency is falsified; the gains would instead be attributable to a repulsive regularizer.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Small Transformer models pretrained with SAOBP should close part of the accuracy gap to larger models on long-range reasoning tasks, since explicit multi-hop regularization compensates for reduced depth.
  • GTD can serve as a cheap, layer-wise diagnostic during pretraining or fine-tuning: heads whose GTD falls below roughly 0.5 or above roughly 0.85 are the ones most worth inspecting or regularizing.
  • The choice of repulsive compatibility matters: BP-High outperforms similarity-based BP-ElemMul and repulsion-suppressing BP-Low, so attention diversity, not just any message passing, drives the reported gains.
  • Because the update is a single vectorized pass over attention rows, SAOBP adds modest per-step cost and requires no architectural change, making it suitable for resource-constrained training and inference.
  • The benefit shrinks as model size grows, implying that larger models already learn global context internally and need less explicit multi-hop pressure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I would test whether the gains come from genuine multi-hop path information or from repulsion alone: replace the belief-product with a simpler row-wise anti-concentration transform and compare; if both work equally, the BP framing is incidental to a regularizer.
  • GTD's strong checkpoint-level correlation with accuracy could be turned into an early-stopping criterion or a per-head masking and pruning signal, a use the paper suggests but does not develop.
  • The claim that one step suffices hints that further iterations may add noise rather than context—the paper's own limitation note allows this—so a controlled sweep of one, two, and three steps would settle whether the mechanism is genuinely message passing or just one-shot smoothing.
  • A natural extension is to make the repulsive strength λ adaptive per layer or head, or to anneal it during training, which the paper explicitly leaves to future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes SAOBP, a modification of Transformer self-attention that applies a one-step belief-propagation-style update with a repulsive Potts prior to the attention matrix, with the stated goal of injecting multi-hop token dependencies and mitigating attention localization and entropy collapse. The authors also introduce Global Token Dependency (GTD), a diagnostic computed from powers of the attention matrix, and report that SAOBP increases GTD and improves downstream accuracy on GLUE, SQuAD, HellaSwag, and RACE-Middle for BERT-Mini/Small/Medium, with larger gains in smaller models. Decoder-only experiments are included in an appendix, along with computational-cost measurements.

Significance. If the empirical findings are reproducible, the paper offers a simple and low-cost attention reweighting that improves small-scale Transformer performance and counters entropy collapse, which would be practically useful. The release of code, the breadth of evaluations across three encoder-only and three decoder-only model sizes, and the explicit computational-cost analysis are strengths. GTD is also potentially interesting as a descriptive diagnostic. However, the significance is conditional: the central mechanism claim—that SAOBP injects multi-hop relationships—is not supported by the equations, and the algorithm as written is not reproducible from the manuscript. The paper is best viewed, pending revision, as an empirical regularization study rather than a belief-propagation method.

major comments (4)
  1. [§4.4, Algorithm 1] The pseudocode does not implement Eq. (11). In the inner loop, line 4 defines M_i as a vector, line 5 makes P_i a scalar, and line 6 overwrites MP_i on each iteration; there is no accumulation of a product over i. After the loop, UP_j = A_j ⊙ MP uses only the final vector MP_L. Moreover, Eq. (11) includes the i=j factor, while the comment 'Exclude self-message' suggests it should be removed, yet no conditional skip is shown. As written, the method is ambiguous and not reproducible from the manuscript text.
  2. [§4.3, Eq. (11)] The update is not multi-hop propagation. Expanding the right side of Eq. (11) yields products of single-hop attention weights A_{ik} for a fixed column k, e.g., A_{i1,k}A_{i2,k}...; no term of the form A_{j,m}A_{m,k} with m≠j,k appears. Thus the operation does not compute powers of the attention matrix and cannot 'inject multi-hop relationships' as claimed in the abstract and contributions. The later increase in GTD (Eq. 12, defined via matrix powers) is therefore not caused by the asserted mechanism; it likely reflects a broader attention mass. The factor function in Eq. (7) is also independent of j, so the 'pairwise' structural interpretation is not substantive. Additionally, Eq. (10) multiplies the prior by only one incoming message, contrary to the BP belief update in Eq. (5), before Eq. (11) switches to a product over all i.
  3. [§4, §8] The claim that a single message-passing step is 'sufficient to introduce global contextual information' is an unsupported premise. No experiment varies the number of BP iterations or compares one-step with multi-step updates; the Limitations section even acknowledges that multi-step message passing could 'enhance—or potentially degrade—model representational quality.' Without such an ablation, the central claim that SAOBP's benefit comes from global/multi-hop context rather than from entropy regularization is not established.
  4. [§5, §7] GTD is a diagnostic computed after training; it is not part of the loss or any selection criterion. The repulsive strength λ is fixed by model size in §6, not chosen by GTD or by task performance. Therefore phrases like 'adaptively maintains GTD at task-appropriate levels' (Abstract) and the 'optimal GTD range' of 0.6–0.8 are post hoc descriptions, not a mechanism. The correlation evidence in Fig. 3 is aggregated over checkpoints and does not control for other covariates, so it supports GTD as a descriptive metric but not as the driver of SAOBP's gains.
minor comments (7)
  1. [Tables 2 and 8] The reported GLUE averages for BERT-Small are inconsistent: Table 2 lists 54.03 (Original) and 57.61 (High), while Table 8 lists 56.63 and 57.99. Please reconcile or explain the discrepancy.
  2. [Eq. (14)] The notation eGij is confusing. The text says eGij = Gij / Σ_k Gik, but the symbol suggests an exponential. Use \tilde{G}_{ij} for the row-normalized matrix.
  3. [Algorithm 1, line 3] Since softmax normalization makes S_i = Σ_k A_ik identically 1, the computation is redundant as written. If the method is intended to accept unnormalized inputs, state this explicitly.
  4. [Appendix C] The decoder-only modification is described only verbally. Please provide the exact masking and message-passing equations used to block future tokens.
  5. [Fig. 2 caption] Please clarify how the 40-node subgraph and threshold τ=10^{-4} are selected, and whether CC and BC are averaged over heads or computed per head.
  6. [Terminology] The paper uses 'SAOBP' for the framework and 'BP-High' for a variant, but sometimes the terms are interchanged. Define the relationship explicitly, e.g., SAOBP is the framework, BP-High is the repulsive-Potts instantiation.
  7. [Table 9] The inference latency and throughput columns appear to report measurements from possibly different settings. Please state the hardware and batching conditions for both training and inference tables.

Circularity Check

0 steps flagged

No significant circularity: the core claims rest on external benchmark comparisons, and the GTD diagnostic is independent of training and hyperparameter selection.

full rationale

The paper's derivation chain is self-contained. SAOBP is defined by Eq. (11) and evaluated against an original baseline and prior regularization methods on external benchmarks (GLUE, SQuAD, HellaSwag, RACE), so the main performance claims are not forced by construction. GTD, defined in Eqs. (12)-(13), is a post-hoc diagnostic based on powers of the attention matrix; it is not used as a training loss, nor are its hyperparameters (β=0.9, K=4) tuned to maximize reported metrics. The repulsive strength λ is hand-set per model size (0.2/0.08/0.05) rather than fitted to the reported accuracies, so there is no fitted-input-called-prediction pattern. The paper contains no self-citations from the author team that carry argumentative weight; the cited BP and Potts results are standard external mathematics. The concern that Eq. (11) does not literally contain A_{jm}A_{mk} path terms is a mechanistic/correctness critique, not a circularity: the observed GTD increase is a contingent empirical property of the refined matrix, not an identity between Eq. (11) and Eq. (12). Accordingly, no circular step can be exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The central method depends on a hand-set repulsive strength lambda scaled by model size, and the diagnostic GTD uses manually chosen constants beta and K. The key domain assumptions are that attention rows are valid probability distributions for belief propagation and that a single message-passing step injects global context. GTD is a new invented construct with no independent validation outside the paper.

free parameters (3)
  • repulsive strength lambda = 0.2 (Mini), 0.08 (Small), 0.05 (Medium)
    Hand-set per model size; controls the repulsion strength in the BP-High factor. It is a free hyperparameter of the method, not derived from theory.
  • GTD discount factor beta = 0.9
    Fixed constant for the GTD diagnostic; not fit to data, but chosen ad hoc.
  • GTD maximum multi-hop step K = 4
    Fixed for GTD; choice limits summed powers of the attention matrix.
axioms (6)
  • standard math Softmax rows sum to one, used to simplify Eq. (9) to S_i = 1.
    Invoked when deriving the message update in Section 4.3.
  • domain assumption Attention rows can be treated as categorical distributions over token labels for belief propagation.
    Section 4.1 interprets each row A_i as a variable node with a probability distribution over labels.
  • ad hoc to paper One-step message passing is sufficient to introduce global contextual information.
    Section 4 states 'our experiments suggest that a single-step message passing... is sufficient', with no theoretical guarantee.
  • domain assumption The repulsive Potts potential with parameter lambda drives attention diversification and mitigates localization.
    Eq. (7) defines the factor function and Section 4.1 asserts this effect.
  • domain assumption GTD with beta=0.9 and K=4 is a meaningful measure of multi-hop dependency.
    Eq. (12) defines GTD; the constants are set in Section 6 without derivation.
  • ad hoc to paper The optimal GTD range of 0.6-0.8 is task-appropriate and can be used to interpret model quality.
    Section 7 infers this range from the same data used to evaluate SAOBP; it is not an independent benchmark.
invented entities (1)
  • Global Token Dependency (GTD) no independent evidence
    purpose: Diagnostic metric quantifying relative multi-hop attention mass
    GTD is only evaluated on the authors' own trained checkpoints and tasks; no external validation beyond these models.

pith-pipeline@v1.3.0-alltime-deepseek · 13984 in / 15399 out tokens · 155983 ms · 2026-08-04T22:23:59.898471+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation." pith.science (2026). https://pith.science/paper/PPSYQZJN

@misc{pith2026250907324,
  author       = {Pith},
  title        = {Pith review of: Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PPSYQZJN}},
  note         = {Machine review of arXiv:2509.07324}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization, where attentions collapse onto a limited subset of tokens and fail to capture long-range dependencies. To address this issue, we propose Self-Attention One-step Belief Propagation (SAOBP), a refinement framework that injects multi-hop relationships through a belief propagation process. To interpret and quantify these interactions, we introduce Global Token Dependency (GTD) that captures the relative contribution of multihop connections within the attention graph. Empirical results indicate that SAOBP helps prevent entropy collapse in deeper layers and adaptively maintains GTD at task-appropriate levels, thereby supporting improvements in model performance. Importantly, we observe competitive gains in small-scale models, highlighting its potential for improving inference quality in resource-constrained scenarios.

Figures

Figures reproduced from arXiv: 2509.07324 by Hyewon Jo, Jin Woo Koo, Jungwoo Lee, Minhae Oh, Nakyung Lee, Suhwan Kim, Yeongoon Kim.

Figure 1
Figure 1. Figure 1: Comparison between original self-attention (left) and proposed SAOBP algorithm (right): The standard self [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Correlation between GTD and graph-theoretic indices CC (left), BC (right) during pretraining. For each sample-head pair in the RACE-Middle dev set, we compute average GTD and project the corresponding attention matrix to a token graph (τ = 10−4 ) and measure CC, BC on a 40- node subgraph. Histograms on the margins show the empirical density of each variable, with the pearson coefficient rCC = 0.70, rBC = −… view at source ↗
Figure 3
Figure 3. Figure 3: Correlation between GTD and model performance in BERT-Mini. We report Pearson correlation coefficients com￾puted over 10–20 checkpoints sampled from the pretraining or finetuning stages (p < 0.02). Higher GTD values correlate positively with accuracy on SST2, HellaSwag, RACE tasks, and negatively with perplexity in WikiText. Model Algorithm Glue Hellaswag RACE-Middle SQuAD Mini Original 53.37 28.06 28.48 1… view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise GTD and Entropy under Different Factor Functions on 1024 samples of RACE-Middle. Each row corresponds to a model size (top = BERT-Mini, middle = BERT-Small, bottom = BERT-Medium). Columns report, from left to right: (1) GTD ratio per layer, (2) indirect entropy per layer, (3) mean positional entropy across layers, and (4) mean head entropy in the final layer. Curves compare four factor functions… view at source ↗
Figure 5
Figure 5. Figure 5: Correlation between GTD and graph-theoretic [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of attention map sparsity between the original model (top) and BP-High (bottom). Attention [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GTD value, indirect entropy, layer-wise and head-wise entropy distriutions on SQuAD datasets [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GTD value, indirect entropy, layer-wise and head-wise entropy distriutions on SST2 (GLUE) datasets [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pearson correlation between GTD and model performance for BERT-Small and BERT-Medium across [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 24 canonical work pages

  1. [1]

    Han Bao, Ryuichiro Hataya, and Ryo Karakida. 2024. Self-attention networks localize when qk-eigenspectrum concentrates. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), pages 2903--2922. PMLR

    work page 2024

  2. [2]

    Alain Barrat, Marc Barth\'elemy, Romualdo Pastor-Satorras, and Alessandro Vespignani. 2004. The architecture of complex weighted networks. Proceedings of the National Academy of Sciences, 101(11):3747--3752

    work page 2004

  3. [3]

    Yuri Boykov and Marie - Pierre Jolly. 2001. https://doi.org/10.1109/ICCV.2001.937505 Interactive graph cuts for optimal boundary & region segmentation of objects in n\!-\!d images . In Proceedings of the 8th IEEE International Conference on Computer Vision (ICCV), pages 105--112. IEEE

    work page arXiv 2001

  4. [4]

    Haoyue Dai, Bo Dai, and Le Song. 2016. Discriminative embeddings of latent variable models for structured data. In Proceedings of the 33rd International Conference on Machine Learning, pages 2702--2711, New York, NY, USA. PMLR

    work page 2016

  5. [5]

    Zico Kolter

    Zihang Dong, Huan Zhang, Kaili Xu, Jiajun Li, Hongyuan Zha, and J. Zico Kolter. 2024. https://openreview.net/forum?id=0bGkVp1jEq Transformer training instability of softmax and lipschitz kernel attentions . In Proceedings of the 12th International Conference on Learning Representations (ICLR)

    work page 2024

  6. [6]

    Linton C. Freeman. 1977. A set of measures of centrality based on betweenness. Sociometry, 40(1):35--41

    work page 1977

  7. [7]

    Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. https://skylion007.github.io/OpenWebTextCorpus/

    work page 2019

  8. [8]

    Nandan Kumar Jha and Brandon Reagen. 2025. Entropy-guided attention for private llms. arXiv preprint arXiv:2501.03489

    Pith/arXiv arXiv 2025

  9. [9]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Findings of EMNLP, pages 4163--4174

    work page 2020

  10. [10]

    Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. Structured attention networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark. Association for Computational Linguistics

    work page 2017

  11. [11]

    Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models : Principles and Techniques . MIT Press, Cambridge, MA

    work page 2009

  12. [12]

    Jonathan Kuck, Shuvam Chakraborty, Hao Tang, Rachel Luo, Jiaming Song, Ashish Sabharwal, and Stefano Ermon. 2020. Belief propagation neural networks. In Advances in Neural Information Processing Systems 33, pages 1--9, Vancouver, BC, Canada. Curran Associates, Inc

    work page 2020

  13. [13]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    work page 2017

  14. [14]

    Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra

    Zechun Liu, Changsheng Zhao, Forrest N. Iandola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, Liangzhen Lai, and Vikas Chandra. 2024. https://arxiv.org/abs/2402.14905 Mobilellm: Optimizing sub‐billion parameter language models for on‐device use cases . In Proceedings of the 41st International ...

    Pith/arXiv arXiv 2024

  15. [15]

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. https://arxiv.org/abs/1609.07843 Pointer sentinel mixture models . In Proceedings of the 34th International Conference on Machine Learning

    Pith/arXiv arXiv 2016

  16. [16]

    Marc Mézard and Andrea Montanari. 2009. Information, Physics, and Computation, 1st edition. Oxford University press

    work page 2009

  17. [17]

    Judea Pearl. 1988. Probabilistic Reasoning in Intelligent Systems : Networks of Plausible Inference . Morgan Kaufmann, San Mateo, CA

    work page 1988

  18. [18]

    Renfrey B. Potts. 1952. https://doi.org/10.1017/S0305004100027419 Some generalized order--disorder transformations . Mathematical Proceedings of the Cambridge Philosophical Society, 48(1):106--109

    doi:10.1017/s0305004100027419 1952

  19. [19]

    Xianbiao Qi, Yelin He, Jiaquan Ye, Chun-Guang Li, Bojia Zi, Xili Dai, Qin Zou, and Rong Xiao. 2025. https://openreview.net/forum?id=GeUK3zGreN Taming transformer without using learning rate warmup . In International Conference on Learning Representations (ICLR)

    work page 2025

  20. [20]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1--67

    work page 2020

  21. [21]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    work page 2016

  22. [22]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distil BERT , a distilled version of BERT : Smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Representation Learning for NLP (BlackBoxNLP). Association for Computational Linguistics

    work page 2019

  23. [23]

    Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang, Zhenguo Li, and James Tin-Yau Kwok. 2021. https://proceedings.mlr.press/v139/shi21a.html Sparsebert: Rethinking the importance analysis in self-attention . In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 9547--9557. PMLR

    work page 2021

  24. [24]

    Khan, Hisham Cholakal, Rao M

    Omkar Thawakar, Ashmal Vayani, Salman H. Khan, Hisham Cholakal, Rao M. Anwer, Michael Felsberg, Tim Baldwin, Eric P. Xing, and Fahad Shahbaz Khan. 2024. https://arxiv.org/abs/2402.16840 Mobillama: Towards accurate and lightweight fully transparent gpt . Preprint, arXiv:2402.16840

    Pith/arXiv arXiv 2024

  25. [25]

    Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962

    Pith/arXiv arXiv 2019

  26. [26]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc

    work page 2017

  27. [27]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop on BlackboxNLP

    work page 2018

  28. [28]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, and Suhang Wang. 2024. https://arxiv.org/abs/2411.03350 A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and...

    Pith/arXiv arXiv 2024

  29. [29]

    Yedidia, William T

    Jonathan S. Yedidia, William T. Freeman, and Yair Weiss. 2001. Generalized belief propagation. In Advances in Neural Information Processing Systems\;13, pages 689--695

    work page 2001

  30. [30]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)

    work page 2019

  31. [31]

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Josh Susskind. 2023. Stabilizing transformer training by preventing attention entropy collapse. In Proceedings of the 40th International Conference on Machine Learning, pages 40770--40803. PMLR

    work page 2023

  32. [32]

    Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, and Dong Yu. 2024. https://arxiv.org/abs/2412.16545 Attention entropy is a key factor: An analysis of parallel context encoding with full-attention-based pre-trained language models . arXiv preprint arXiv:2412.16545

    Pith/arXiv arXiv 2024

  33. [33]

    Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. https://arxiv.org/abs/1506.06724 Aligning books and movies: Towards story-like visual explanations by watching movies and reading books . In Proceedings of the IEEE International Conference on Computer Vision (ICCV)

    Pith/arXiv arXiv 2015

  34. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  35. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...