Pith. sign in

REVIEW 4 major objections 5 minor 69 references

Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SurvBESA applies self-attention to Beran survival curves and reports better concordance than bagging, RSF, or GBM.

desk verdict The ensemble idea is real, but the advertised quadratic-program special case flips the C-index objective and substitutes a point value for an integral, so the paper's central methodological claim is unsupported as written. read the letter →

arxiv 2506.07933 v1 pith:YB2JJVAT submitted 2025-06-09 cs.LG stat.ML

Lev V. Utkin , Semen P. Khomets , Vlada A. Efremenko , Andrei V. Konstantinov , Natalya M. Verbova This is my paper
classification cs.LGstat.ML
keywords survivalanalysisBeranestimatorself-attentionensemblelearningC-indexcensoreddataHubercontaminationmodelquadraticoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SurvBESA, an ensemble survival model in which each weak learner is a Beran estimator of the conditional survival function and the base survival functions are aggregated by a self-attention mechanism instead of by simple averaging. The authors' central claim is that this attention-weighted aggregation smooths away noisy or anomalous survival functions produced by unlucky bootstrap samples, and that the resulting model achieves higher concordance indices than a single Beran estimator, bagged Beran estimators, random survival forests, and gradient-boosted Cox and AFT models. A special case using Huber's contamination model for the attention weights is shown to turn parameter training into a quadratic or linear optimization problem. The claim matters because ensemble survival methods are widely deployed, so a stable and cheap improvement in ranking accuracy would transfer directly to medical and reliability settings with censored data.

What carries the argument

The central object is the Beran estimator, a kernel-weighted product-limit estimator of the conditional survival function $\widehat{S}(t\mid x)$. Around it the paper builds two moves: first, an ensemble of $M$ Beran estimators trained on bootstrap subsets; second, a self-attention layer in which the query, key, and value are the predicted survival functions rather than feature vectors. Attention weights are derived from the Kolmogorov-Smirnov distance between pairs of survival functions through a Gaussian kernel, so neighboring survival curves receive large weight and anomalous curves are pulled toward the ensemble. Training maximizes a sigmoid-smoothed C-index over the attention parameters. The special case replaces the attention weight with Huber's $\epsilon$-contamination form $(1-\epsilon)\cdot\mathrm{softmax}(\cdot)+\epsilon\cdot\theta$, which, combined with a hinge loss, reduces parameter fitting to a quadratic program with linear constraints in $\theta$ and auxiliary variables $\xi_{ij}$.

What would settle it

Compute both sides of the substitution for a real ensemble, $\widehat{T}_i^{(k)} = \sum_{l} S_l^{(k)}(x_i)(t_{l+1}-t_l)$ versus $S^{(k)}(T_i\mid x_i)$, then compare the sign of $R_{ij}(\theta)$ under each choice; a material disagreement would show that the quadratic program optimizes a different objective than the C-index it claims.

Watch

Extended reading notes

Core claim

The paper's claim is that self-attention can be applied to the outputs of an ensemble, not to its input features, and that this is enough to stabilize and improve survival predictions. Each weak learner is a Beran estimator returning a step-function survival curve $S^{(k)}(t\mid x)$. The model computes attention weights from the Kolmogorov-Smirnov distance between pairs of curves, forms denoised curves as weighted sums of neighboring curves, and averages the denoised curves to obtain the ensemble prediction. The weights are trained by maximizing a sigmoid surrogate of the C-index; under Huber's contamination model the training problem becomes quadratic or linear. The authors interpret their synthetic and real-data experiments as showing that SurvBESA outperforms a single Beran estimator, bagged Beran estimators, random survival forests, and gradient-boosted Cox and AFT models.

Load-bearing premise

The quadratic special case assumes that replacing the expected survival time with the survival probability at the observed event time does not change which rankings the training objective optimizes, and the paper does not justify that swap.

Editorial extensions

If this is right

  • Attention-weighted aggregation should raise the concordance index relative to simple bagging whenever bootstrap subsets produce divergent survival curves.
  • The Huber special case gives a training route that avoids backpropagation through the survival curves, at the price of the substitution described below.
  • On the ten real benchmarks studied, SurvBESA reports the highest mean C-index on most datasets, and the paired t-test over datasets is reported as significant with $p=0.0017$.
  • The method should be most valuable on small datasets with multi-cluster structure, where individual bootstrap samples can be unrepresentative of a test point's true neighborhood.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same self-attention-over-predictions scheme to ensembles of other nonparametric survival estimators, since nothing in the mechanism depends on the Beran product-limit form.
  • The attention weights depend only on pairwise survival-curve distances, so the correction could be applied as a post-processing step to any existing ensemble of predicted survival functions without retraining the base models.
  • The quadratic special case's equivalence to the original C-index objective depends on the unstated substitution of $S^{(k)}(T_i\mid x_i)$ for the expected time $\widehat{T}_i^{(k)}$; testing that substitution on a real dataset would separate a genuine algorithmic contribution from an approximation artifact.
  • One could check whether the full matrix of per-pair temperature parameters is needed or a single shared temperature captures the gain, which would make the method cheaper and easier to tune.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SurvBESA, an ensemble method that applies a self-attention mechanism to the survival functions produced by bagged Beran estimators. Attention weights are trained by maximizing a sigmoid-smoothed C-index, and a special case based on Huber's epsilon-contamination model is claimed to reduce training to a quadratic program. Experiments on synthetic and real datasets compare SurvBESA with bagged Beran, single Beran, RSF, GBM Cox, and GBM AFT, reporting higher C-index values for SurvBESA; the code is publicly available.

Significance. If the claims are correct, the idea of applying self-attention to predicted survival functions rather than to raw feature vectors is a useful and non-standard contribution, and the public implementation is a practical strength. The empirical results look promising at face value. However, the derivation of the special-case quadratic program in Section 3.3 is internally inconsistent as written, and the paper does not state whether the experiments actually use that special case or the general sigmoid surrogate. These issues must be resolved before the claimed contributions can be evaluated.

major comments (4)
  1. [Section 3.3, Eq. (22)] The hinge-loss surrogate is sign-flipped. The C-index in Eq. (18) counts pairs with R_ij(theta) > 0, so maximizing concordance requires penalizing non-positive margins, for example by minimizing sum max(0, -R_ij(theta)) or a margin version such as max(0, 1 - R_ij(theta)). The stated objective, minimizing sum max(0, R_ij(theta)), actively drives R_ij negative and thus rewards discordant orderings. This is a load-bearing algebraic error in the advertised QP simplification.
  2. [Section 3.3, Eqs. (26)-(28)] The derivation silently replaces the predicted expected time bT^(k)_i, which is an integral of the step survival curve defined in Eq. (13), with the point value S^(k)(T_i | x_i) in Eqs. (26) and (27). These are mathematically different quantities, and the substitution is neither stated nor justified. Using bT^(k)_i instead of S^(k)(T_i|x_i) preserves the affine structure in theta, so the QP formulation could survive a correction, but as written Eq. (28) does not optimize the C-index surrogate defined in Eqs. (17)-(18).
  3. [Section 3.3, Eqs. (20) and (24)-(25)] The attention weights are not properly normalized. With beta(l,k)_j defined as (1-epsilon)*softmax(...) + epsilon*theta_{l,k} and with theta constrained to the M x M unit simplex, the row sums of beta are not equal to 1 for each query l, so the weights do not define a valid attention distribution over the M base estimators. The constraints in Eqs. (24)-(25) also mix the notation theta in Delta_M and theta in Delta_{M x M}, making the feasible set ambiguous.
  4. [Section 3.4 and Section 3.5] The paper never states whether the numerical experiments use the QP special case of Section 3.3 or the general sigmoid surrogate of Eq. (19). Section 3.4 says that 'the training parameters are optimized using Adam with 100 epochs,' which suggests the sigmoid objective, not the QP. Since the abstract and the conclusion advertise the QP/linear simplification as a key contribution, the authors must clarify which objective is used in each experiment; if the flawed QP was used, the reported results need to be re-examined in light of the sign error.
minor comments (5)
  1. [Section 3.3] The hyperparameter eta is introduced but never defined; the following equations use phi instead.
  2. [Section 3.4 and Section 3.5] The hyperparameter w appears in the SurvBESA and Bagging lists but is never defined in the model; it may be intended as the regularization lambda.
  3. [Section 3.5] The t-test is described as proposed by Demsar, but Demsar (2006) recommends rank-based tests such as Wilcoxon signed-rank or Friedman. Comparing against the 'best results provided by other models' in each dataset inflates the risk of false significance.
  4. [Overall structure] The section numbering is inconsistent: the numerical-experiments section appears to be a top-level heading after Section 3.3, yet its subsections are numbered 3.4-3.6, and the conclusion is numbered 4.
  5. [Eq. (15)] There appears to be a typo in Eq. (15): the second factor in the sum should be bT^(k) rather than bT^(l) to match the definition in Eq. (17).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: attention weights are fit on training data and benchmark comparisons are held out; Section 3.3 concerns are algebraic, not circular.

full rationale

The paper's claimed derivation chain is not circular. The self-attention weights beta in Eqs. (7) and (10) are learned by maximizing a sigmoid surrogate of the C-index (Eq. (19)) on training data, and the special-case weights in Eq. (20) are taken from the authors' earlier contamination-model paper [27]; supervised fitting and reuse of a published construction are not circular reductions. The benchmark claim is supported by held-out test-set C-index results in Table 1 compared with standard external baselines (RSF, GBM Cox, GBM AFT), with hyperparameters tuned on a validation split, so the central empirical claim is not equivalent to its inputs. The paper does cite two earlier works by the same group ([27], [28]) for the epsilon-contamination attention form, but those citations are not invoked to forbid alternatives or to import a uniqueness theorem, and the Huber contamination model itself is attributed to standard external references [25], [26]. The sign convention and the bT-vs-S(T) substitution in Section 3.3 are potential correctness defects in the quadratic/linear special case, but they are algebraic consistency issues rather than cases where an output reduces by definition to an input. Accordingly, no self-definitional, fitted-input, or load-bearing self-citation circularity is present.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method's performance depends on several tuned hyperparameters and a learned attention matrix. The special case adds epsilon, phi, and lambda, each fitted from validation data. No new physical or scientific entities are introduced.

free parameters (6)
  • Attention weights theta = Learned via hinge-loss QP on training data
    M(M-1) entries of the contamination distribution; these are the main trainable parameters of the special case.
  • Kernel temperature tau = Tuned via Optuna over {1e-2,...,1e3}
    Controls the neighborhood size in each Beran estimator; directly affects survival curve shape.
  • Contamination parameter epsilon = Tuned in [0,1]
    Balances the fixed softmax component and the learned theta component in the attention weights.
  • Softmax temperature phi = Tuned
    Scales the KS distances before softmax in the attention kernel.
  • Regularization lambda = Tuned
    Strength of the L2 penalty on theta in the QP objective.
  • Ensemble size M and subset fraction = Tuned (M in [5,50], fraction in [0.1,0.7])
    Number of Beran estimators and bootstrap sample size; chosen per dataset by Optuna.
assumptions (4)
  • standard math The Beran estimator provides a valid conditional survival function estimate
    Kernel-based survival curve estimation is an established method; paper cites Beran [14] and does not re-derive it.
  • domain assumption C-index is a suitable objective for training survival models
    Used as both evaluation metric and (through surrogate) training loss; standard in survival analysis though not strictly convex.
  • domain assumption The sigmoid and hinge surrogates replace the non-differentiable indicator without changing the optimal ranking
    Assumed in Eq. (19) and (22); common practice but not proved in the paper.
  • ad hoc to paper Huber contamination model is a meaningful representation of attention weight uncertainty
    Borrowed from the authors' prior work [27,28]; no independent biological or statistical justification in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions." pith.science (2026). https://pith.science/paper/YB2JJVAT

@misc{pith2026250607933,
  author       = {Pith},
  title        = {Pith review of: Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YB2JJVAT}},
  note         = {Machine review of arXiv:2506.07933}
}
read the original abstract

Survival analysis predicts the time until an event of interest, such as failure or death, but faces challenges due to censored data, where some events remain unobserved. Ensemble-based models, like random survival forests and gradient boosting, are widely used but can produce unstable predictions due to variations in bootstrap samples. To address this, we propose SurvBESA (Survival Beran Estimators Self-Attended), a novel ensemble model that combines Beran estimators with a self-attention mechanism. Unlike traditional methods, SurvBESA applies self-attention to predicted survival functions, smoothing out noise by adjusting each survival function based on its similarity to neighboring survival functions. We also explore a special case using Huber's contamination model to define attention weights, simplifying training to a quadratic or linear optimization problem. Numerical experiments show that SurvBESA outperforms state-of-the-art models. The implementation of SurvBESA is publicly available.

Figures

Figures reproduced from arXiv: 2506.07933 by the authors.

Figure 1
Figure 1. Dependence of the C-index on the parameter [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Dependence of the C-index on the number of points in the training set [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Dependence of the C-index on the proportion of uncensored data [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Dependence of the C-index on the number of weak models [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Dependence of the C-index on the subset size [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Dependence of the C-index on the parameter [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Illustration of the SurvBESA training process for the Veteran dataset [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Illustration of the SurvBESA training process for the Breast Cancer dataset [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Illustration of the SurvBESA training process for the AIDS dataset [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: An illustrative example of the SF transformation produced by the weak Beran models [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 53 canonical work pages

  1. [1]

    P. Wang, Y. Li, and C.K. Reddy. Machine learning for survival analysis: A survey. ACM Computing Surveys (CSUR) , 51(6):1–36, 2019

    work page 2019

  2. [2]

    A Deep Active Survival Analysis Approach for Precision Treatment Recommendations: Application of Prostate Cancer

    M.Z. Nezhad, N. Sadati, K. Yang, and D. Zhu. A deep active survival analysis approach for precision treatment recommendations: Application of prostate cancer. arXiv:1804.03280v1, April 2018. 18

    work page Pith review arXiv 2018

  3. [3]

    Ibrahim, A

    N.A. Ibrahim, A. Kudus, I. Daud, and M.R. Abu Bakar. Decision tree for competing risks survival probability in breast cancer study. International Journal Of Biological and Medical Research, 3(1):25–29, 2008

    work page 2008

  4. [4]

    Mogensen, H

    U.B. Mogensen, H. Ishwaran, and T.A. Gerds. Evaluating random forests for survival analysis using prediction error curves. Journal of Statistical Software , 50(11):1–23, 2012

    work page 2012

  5. [5]

    Schmid, M.N

    M. Schmid, M.N. Wright, and A. Ziegler. On the use of harrell’s c for clinical risk prediction via random survival forests. Expert Systems with Applications , 63:450–459, 2016

    work page 2016

  6. [6]

    Wang and L

    H. Wang and L. Zhou. Random survival forest with space extensions for censored data. Artificial intelligence in medicine , 79:52–61, 2017

    work page 2017

  7. [7]

    Utkin, A.V

    L.V. Utkin, A.V. Konstantinov, V.S. Chukanov, M.V. Kots, M.A. Ryabinin, and A.A. Meldo. A weighted random survival forest. Knowledge-Based Systems, 177:136–144, 2019

    work page 2019

  8. [8]

    Utkin and A.V

    L.V. Utkin and A.V. Konstantinov. Random survival forests incorporated by the nadaraya-watson regression. Informatics and Automation , 21(5):851–880, 2022

    work page 2022

Show all 69 references
  1. [9]

    Wright, T

    M.N. Wright, T. Dankowski, and A. Ziegler. Unbiased split variable selection for random survival forests using maximally selected rank statistics. Statistics in Medicine , 36(8):1272–1284, 2017

    2017

  2. [10]

    A gradient boosting algorithm for survival analysis via direct optimization of concordance index

    Yifei Chen, Zhenyu Jia, Dan Mercola, and Xiaohui Xie. A gradient boosting algorithm for survival analysis via direct optimization of concordance index. Computational and mathematical methods in medicine, 2013(1):873595, 2013

    2013

  3. [11]

    Hitboost: survival analysis via a multi-output gradient boosting decision tree method

    Pei Liu, Bo Fu, and Simon X Yang. Hitboost: survival analysis via a multi-output gradient boosting decision tree method. IEEE Access, 7:56785–56795, 2019

    2019

  4. [12]

    Meier, S

    R. Meier, S. Graw, J. Usset, R. Raghavan, J. Dai, P. Chalise, S. Ellis, B. Fridley, and D. Koestler. An ensemble-based Cox proportional hazards regression framework for predicting survival in metastatic castration-resistant prostate cancer (mcrpc) patients. F1000Research, 5:2677, 2016

    2016

  5. [13]

    D.R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society, Series B (Methodological), 34(2):187–220, 1972

    1972

  6. [14]

    R. Beran. Nonparametric regression with randomly censored survival data. Technical report, University of California, Berkeley, 1981

    1981

  7. [15]

    A survey on ensemble learning

    Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science , 14:241–258, 2020

    2020

  8. [16]

    Ferreira and M.A.T

    A.J. Ferreira and M.A.T. Figueiredo. Boosting algorithms: A review of methods, theory, and applications. In C. Zhang and Y. Ma, editors, Ensemble Machine Learning: Methods and Appli- cations, pages 35–85. Springer, New York, 2012

    2012

  9. [17]

    Y. Ren, L. Zhang, and P. N. Suganthan. Ensemble classification and regression-recent devel- opments, applications and future directions [review article]. IEEE Computational Intelligence Magazine, 11(1):41–53, 2016

    2016

  10. [18]

    Sagi and L

    O. Sagi and L. Rokach. Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(e1249):1–18, 2018. 19

    2018

  11. [19]

    Wozniak, M

    M. Wozniak, M. Grana, and E. Corchado. A survey of multiple classifier systems as hybrid systems. Information Fusion, pages 3–17, 2014

    2014

  12. [20]

    Z.-H. Zhou. Ensemble Methods: Foundations and Algorithms . CRC Press, Boca Raton, 2012

    2012

  13. [21]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998–6008, 2017

    2017

  14. [22]

    Transformer-based deep survival analysis

    Shi Hu, Egill Fridgeirsson, Guido van Wingen, and Max Welling. Transformer-based deep survival analysis. In Survival Prediction-Algorithms, Challenges and Applications , pages 132–148. PMLR, 2021

    2021

  15. [23]

    Explainable survival analysis with uncertainty using convolution- involved vision transformer

    Zhihao Tang, Li Liu, Zongyi Chen, Guixiang Ma, Jiyan Dong, Xujie Sun, Xi Zhang, Chaozhuo Li, Qingfeng Zheng, Lin Yang, et al. Explainable survival analysis with uncertainty using convolution- involved vision transformer. Computerized Medical Imaging and Graphics , 110:102302, 2023

    2023

  16. [24]

    Survtrace: Transformers for survival analysis with competing events

    Zifeng Wang and Jimeng Sun. Survtrace: Transformers for survival analysis with competing events. In Proceedings of the 13th ACM International Conference on Bioinformatics, Computa- tional Biology and Health Informatics , pages 1–9, 2022

    2022

  17. [25]

    P.J. Huber. Robust Statistics. Wiley, New York, 1981

    1981

  18. [26]

    P. Walley. Statistical Reasoning with Imprecise Probabilities. Chapman and Hall, London, 1991

    1991

  19. [27]

    Utkin and A.V

    L.V. Utkin and A.V. Konstantinov. Attention-based random forest and contamination model. Neural Networks, 154:346–359, 2022

    2022

  20. [28]

    Utkin, A.V

    L.V. Utkin, A.V. Konstantinov, and S.R. Kirpichenko. Attention and self-attention in random forests. Progress in Artificial Intelligence , 12:257–273, 2023

    2023

  21. [29]

    Ridgeway

    G. Ridgeway. The state of boosting. Computing science and statistics , 31:172–181, 1999

    1999

  22. [30]

    Barnwal, H

    A. Barnwal, H. Cho, and T. Hocking. Survival regression with accelerated failure time model in xgboost. Journal of Computational and Graphical Statistics , 31(4):1292–1302, 2022

    2022

  23. [31]

    Deep learning for survival analysis: a review

    Simon Wiegrebe, Philipp Kopper, Raphael Sonabend, Bernd Bischl, and Andreas Bender. Deep learning for survival analysis: a review. Artificial Intelligence Review , 57(65):1–34, 2024

    2024

  24. [32]

    Katzman, U

    J.L. Katzman, U. Shaham, A. Cloninger, J. Bates, T. Jiang, and Y. Kluger. Deepsurv: Person- alized treatment recommender system using a Cox proportional hazards deep neural network. BMC medical research methodology, 18(24):1–12, 2018

    2018

  25. [33]

    M. Luck, T. Sylvain, H. Cardinal, A. Lodi, and Y. Bengio. Deep learning for patient-specific kidney graft survival analysis. arXiv:1705.10245, May 2017

    2017 arXiv

  26. [34]

    J. Yao, X. Zhu, F. Zhu, and J. Huang. Deep correlational learning for survival prediction from multi-modality data. In Medical Image Computing and Computer–Assisted Intervention – MIC- CAI 2017, volume 10434 of Lecture Notes in Computer Science , pages 406–414. Springer, Cham,...

    2017

  27. [35]

    Deep recurrent survival analysis

    Kan Ren, Jiarui Qin, Lei Zheng, Zhengyu Yang, Weinan Zhang, Lin Qiu, and Yong Yu. Deep recurrent survival analysis. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 4798–4805, 2019

    2019

  28. [36]

    Haarburger, P

    C. Haarburger, P. Weitz, O. Rippel, and D. Merhof. Image-based survival analysis for lung cancer patients using CNNs. arXiv:1808.09679v1, Aug 2018

    2018 arXiv

  29. [37]

    Hothorn, P

    T. Hothorn, P. B¨ uhlmann, S. Dudoit, A. Molinaro, and M.J. van der Laan. Survival ensembles. Biostatistics, 7(3):355–373, 2006

    2006

  30. [38]

    Cheng, L

    J. Cheng, L. Dong, and M. Lapata. Long short-term memory-networks for machine reading. arXiv:1601.06733, Jan 2016

    2016 arXiv

  31. [39]

    Parikh, O

    A. Parikh, O. Tackstrom, D. Das, and J. Uszkoreit. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255. Association for Computational Linguistics, 2016

    2016

  32. [40]

    Z. Lin, M. Feng, C.N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio. A structured self- attentive sentence embedding. In The 5th International Conference on Learning Representations (ICLR 2017) , pages 1–15, 2017

    2017

  33. [41]

    Devlin, M.W

    J. Devlin, M.W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, Oct 2018

    2018 arXiv

  34. [42]

    F. Wu, A. Fan, A. Baevski, Y. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations (ICLR 2019) , pages 1–14, 2019

    2019

  35. [43]

    Povey, H

    D. Povey, H. Hadian, P. Ghahremani, K. Li, and S. Khudanpur. A time-restricted self-attention layer for asr. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5874–5878. IEEE, 2018

    2018

  36. [44]

    K. Shim, J. Choi, and W. Sung. Understanding the role of self attention for efficient speech recognition. In The Tenth International Conference on Learning Representations (ICLR), volume https://openreview.net/forum?id=AvcfxqRy4Y, pages 1–19, 2022

    2022

  37. [45]

    Z. Chen, L. Xie, J. Niu, X. Liu, and L. Wei. Joint self-attention and scale-aggregation for self-calibrated deraining network. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2517–2525, 2020

    2020

  38. [46]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, and J. Uszkoreit. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929, Oct 2020

    2010 arXiv

  39. [47]

    Guo, Z.-N

    M.-H. Guo, Z.-N. Liu, T.-J. Mu, and S.-M. Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv:2105.02358, May 2021

    2021 arXiv

  40. [48]

    Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierar- chical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012–10022, 2021. 21

    2021

  41. [49]

    Z. Shen, I. Bello, R. Vemulapalli, X. Jia, and C.H. Chen. Global self-attention networks for image recognition. arXiv:2010.03019, Oct 2020

    2010 arXiv

  42. [50]

    Soydaner

    D. Soydaner. Attention mechanism in neural networks: where it comes and where it goes. Neural Computing and Applications , 34(16):13371–13385, 2022

    2022

  43. [51]

    F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang. Residual attention network for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3156–3164, 2017

    2017

  44. [52]

    X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7794–7803, 2018

    2018

  45. [53]

    H. Zhao, J. Jia, and V. Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10076–10085, 2020

    2020

  46. [54]

    Brauwers and F

    G. Brauwers and F. Frasincar. A general survey on attention mechanisms in deep learning. IEEE Transactions on Knowledge and Data Engineering , 35(4):3279–3298, 2021

    2021

  47. [55]

    T. Lin, Y. Wang, X. Liu, and X. Qiu. A survey of transformers. arXiv:2106.04554, Jul 2021

    2021 arXiv

  48. [56]

    Goncalves, I

    T. Goncalves, I. Rio-Torto, L.F. Teixeira, and J.S. Cardoso. A survey on attention mechanisms for medical applications: are we moving towards better algorithms? arXiv:2204.12406, Apr 2022

    2022 arXiv

  49. [57]

    Hassanin, S

    M. Hassanin, S. Anwar, I. Radwan, F.S. Khan, and A. Mian. Visual attention methods in deep learning: An in-depth survey. Information Fusion, 108:102417, 2024

    2024

  50. [58]

    S. Khan, M. Naseer, M. Hayat, S.W. Zamir, F.S. Khan, and M. Shah. Transformers in vision: A survey. ACM Computing Surveys (CSUR) , pages 1–38, 2022

    2022

  51. [59]

    Santana and E

    A. Santana and E. Colombini. Neural attention models in deep learning: Survey and taxonomy. arXiv:2112.05909, Dec 2021

    2021 arXiv

  52. [60]

    Soydaner

    D. Soydaner. Attention mechanism in neural networks: Where it comes and where it goes. arXiv:2204.13154, Apr 2022

    2022 arXiv

  53. [61]

    Y. Xu, H. Wei, M. Lin, Y. Deng, K. Sheng, M. Zhang, F. Tang, W. Dong, F. Huang, and C. Xu. Transformers in computational visual media: A survey. Computational Visual Media, 8(1):33–62, 2022

    2022

  54. [62]

    Deep gated neural network with self-attention mechanism for survival analysis

    Xulin Yang and Hang Qiu. Deep gated neural network with self-attention mechanism for survival analysis. IEEE Journal of Biomedical and Health Informatics , 2024

    2024

  55. [63]

    Hosmer, S

    D. Hosmer, S. Lemeshow, and S. May. Applied Survival Analysis: Regression Modeling of Time to Event Data . John Wiley & Sons, New Jersey, 2008

    2008

  56. [64]

    Harrell, R

    F. Harrell, R. Califf, D. Pryor, K. Lee, and R. Rosati. Evaluating the yield of medical tests. Journal of the American Medical Association , 247:2543–2546, 1982

    1982

  57. [65]

    Uno, Tianxi Cai, M.J

    H. Uno, Tianxi Cai, M.J. Pencina, R.B. D’Agostino, and Lee-Jen Wei. On the c-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in medicine, 30(10):1105–1117, 2011. 22

    2011

  58. [66]

    R. Vidal. Attention: Self-expression is all you need. ICLR 2022, OpenReview.net. https://openreview.net/forum?id=MmujBClawFo, 2022

    2022

  59. [67]

    Van Belle, K

    V. Van Belle, K. Pelckmans, J.A.K. Suykens, and S. Van Huffel. Support vector machines for survival analysis. In Proceedings of the Third International Conference on Computational Intelligence in Medicine and Healthcare (CIMED2007) , pages 1–8, 2007

    2007

  60. [68]

    Akiba, S

    T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama. Optuna: A next-generation hyperparam- eter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2623–2631, 2019

    2019

  61. [69]

    J. Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1–30, 2006. 23

    2006

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.