Pith. sign in

REVIEW 4 major objections 5 minor 49 references

Financial Risk Assessment via Long-term Payment Behavior Sequence Folding

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LBSF claims that folding long payment behavior sequences by merchant, with multi-field encoding and hierarchical Transformer melding, yields more accurate default-risk predictions than standard long-sequence models.

desk verdict Merchant-level folding is a genuinely new idea and the paper is clearly written, but the central performance claim is currently undercut by an unstated temporal boundary between behavior sequences and default labels. read the letter →

arxiv 2411.15056 v1 pith:HW4XPG66 submitted 2024-11-22 cs.CY cs.AI

Yiran Qiao , Yateng Tang , Xiang Ao , Qi Yuan , Ziming Liu , Chen Shen , Xuehao Zheng This is my paper · ORCID
classification cs.CYcs.AI
keywords financialriskassessmentlong-termbehaviorsequencefoldingmerchant-levelmodelingmulti-fieldencodingdefaultpredictionpaymentbehaviorsTransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that default risk in online inclusive finance can be predicted more accurately when models use long payment histories rather than short windows, and that the best way to use those long histories is to reorganize them by merchant before feeding them to a Transformer. It proposes LBSF, a method that folds each user's payment sequence into merchant-level sub-sequences, encodes payment description, time, and amount jointly, then melds within-merchant and across-merchant information to build a user representation. On a large real-world dataset, LBSF outperforms 13 baselines on 90-day and 180-day windows, with the biggest gain coming from the merchant-folding component. If this is right, platforms without traditional credit records can extract more signal from everyday payment behavior, and other sequential-modeling tasks on long behavior logs have a cheap, knowledge-free reorganization to try.

What carries the argument

The load-bearing object is merchant-level sequence folding: the user's raw payment sequence is regrouped into sub-sequences by merchant, keeping chronological order within each merchant, so the chaotic global timeline becomes a structured set of per-merchant histories. This folding is what makes parallel Transformer processing feasible and, more importantly, exposes shifts in spending patterns, such as moving from high-end dining to convenience-store purchases, which the paper argues are markers of financial stress. It is paired with multi-field behavior encoding, using description text, sine-cosine periodicity embeddings for month, day, week, and hour, and transaction amount, plus a two-stage Transformer melding with a ViT-style [CLS] token for across-merchant relational learning.

What would settle it

Take the 180-day subset, fix a cutoff date well before each user's default event, discard all payments after it, and re-run LBSF versus Mamba; also rerun with merchants chosen by a label-blind rule such as top-frequency. If AUC and Recall@10% margins vanish, the longer-window gains are leakage or merchant-selection artifacts rather than genuine folding benefits.

Watch

Extended reading notes

Core claim

The central claim is that the internal structure of payment behavior itself, specifically the merchant a user pays, is a better grouping cue for long-sequence modeling than external hierarchies such as web-page structure or extracted events. LBSF reorganizes the chronological payment sequence by merchant, preserving time order inside each merchant, encodes each payment through concatenated description, sine-cosine time, and amount embeddings, and then applies two levels of Transformer aggregation: one to summarize each merchant's sub-sequence, and a second, ViT-style pass over merchant embeddings with a [CLS] token to produce the user representation for a binary default classifier. The paper reports that this folding consistently beats sequential baselines on 90-day and 180-day windows, reaching AUC 0.7575 and 0.7678 respectively, and that removing merchant folding causes the largest drop in an ablation study, supporting the claim that merchant-level organization carries the signal.

Load-bearing premise

The result stands on an unstated boundary: the payment behaviors used for prediction must end before the default event is determined, because if the behavior window overlaps the label window, the model is reading the default rather than predicting it; a second unstated premise is that the 74 merchants chosen per user are selected without using the default label.

Editorial extensions

If this is right

  • Longer payment windows, from 45 days to 90 days to 180 days, monotonically improve AUC across nearly all compared models, so platforms should retain and use longer behavior histories rather than truncating them.
  • Merchant-level folding contributes more than any single field embedding: ablating it drops 90-day AUC from 0.7575 to 0.7063, a larger loss than removing amount, timing, or description.
  • Because folding is field-based and knowledge-free, it can be applied on top of any end-to-end sequential backbone, including newer state-space models.
  • LBSF's attention weights over merchants give a pattern-like explanation, such as a shift from high-end spending to consumer-finance-funded shopping, which could support operational review of flagged users.
  • On 45-day histories LBSF does not beat Mamba, so the folding advantage appears primarily when histories are long enough to contain meaningful within-merchant trends.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves unresolved whether the behavior window ends before the default event is determined; if the windows overlap, part of the observed gain could come from label leakage rather than modeling.
  • The rule for choosing the 74 merchants per user is never stated; if merchants are selected by frequency, the method may partly reproduce recency and frequency signals that simpler feature-based baselines could also capture.
  • The same folding recipe could transfer to other long behavior-log tasks such as churn prediction or fraud detection wherever a merchant or category field provides a natural grouping, though the paper does not test this.
  • A testable extension is to compare folding by merchant against folding by category, time-of-day, or amount band to see whether the grouping criterion or the two-level structure itself drives the gain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LBSF (Long-term Payment Behavior Sequence Folding) for financial default prediction from long payment histories. LBSF reorganizes each user's payment sequence by merchant, encodes payment descriptions, timestamps, and amounts jointly, melds behavior embeddings within each merchant with a Transformer, then applies a second Transformer plus a ViT-style [CLS] token to perform relational learning across merchants. The method is evaluated on a proprietary Tencent Mobile Payment dataset of 458,744 users with 45-day, 90-day, and 180-day behavior windows, compared against 13 baselines (basic, sequential, and pre-training models). The main reported result is that LBSF outperforms all baselines on the 90-day and 180-day subsets, with the largest gain on 180 days (AUC 0.7678 vs. 0.7554 for Mamba, Recall@10% 0.3467 vs. 0.3208). An ablation study and two qualitative case studies are also presented. The paper concludes that folding long behavior sequences by merchant effectively models long-term financial patterns and yields more accurate user financial profiles.

Significance. If the central claim holds, LBSF offers a practical, architecture-agnostic way to model very long payment sequences: merchant-level folding provides an internal grouping criterion that enables parallel processing, and multi-field encoding exploits rich payment details. The method is evaluated end-to-end on a held-out test split, so there is no derivation-versus-fit circularity, and the attention-based case studies give some interpretability. However, the empirical significance is currently constrained by three load-bearing gaps: the temporal relationship between the behavior window and the default label is never stated, the rule for selecting M=74 merchants per user is unspecified, and the experimental evidence rests on single runs of a proprietary dataset without code or error bars. These issues must be resolved before the reported LBSF-vs-Mamba gap can be attributed to merchant folding rather than to leakage or selection artifacts.

major comments (4)
  1. [Section III-A and Section V-A] The label timing assumption is never stated. Section III-A frames the task as predicting default before granting access to financial services, and Section V-A labels users who 'default after being granted access' as positive, but the paper does not say that the payment-behavior window is truncated at the service-access date or at any fixed offset before the default event. Table I reports only calendar splits of users into training/validation/test sets, not per-user observation-versus-outcome windows. If a positive user's 180-day behavior window overlaps the period after access or around the default event, the model can exploit direct signals of the default (missed payments, balance drops, unusual merchant patterns), and the claimed 1.24pp AUC gap in Table II (0.7678 vs. 0.7554 for Mamba) could be an artifact of label leakage. The authors must specify the exact cutoff between the behavior window and the default determination, and ideally report results under truncated windows as a robustness check.
  2. [Section V-C and Algorithm 1] The selection rule for M=74 merchants per user is not described. The paper states only that 'we set M to 74,' with no explanation of how the merchants {m'_n1,...,m'_nM} are chosen, whether the merchant list is global or per-user, or whether selection happens before or after label assignment. If merchant selection uses label information or future behavior, the reported improvements would be inflated. The authors need to specify the selection procedure and demonstrate that it is label-independent and time-independent.
  3. [Table II and Section VI-A] The evidence for consistent gains across windows is incomplete. XGBoost and MLP are reported only on the 90-day subset (with '——' on 45-day and 180-day), yet the text claims a consistent trend across all models with longer sequences. Additionally, no error bars, confidence intervals, or multiple-seed variance are reported; the 45-day case shows LBSF behind Mamba by 0.46%, and the headline 180-day AUC gap of 1.24pp may be within run-to-run noise. Reporting results over multiple seeds and completing the missing baseline cells is necessary to support the central empirical claim.
  4. [Section V-A and Table I] The dataset description and the table are inconsistent. The text says the dataset is sampled 'to maintain a positive rate among all included users of approximately 10.0%,' but Table I reports positive rates of 7.2% for training, 12.2% for validation, and 12.2% for testing. The authors should reconcile this discrepancy and state whether the sampling was done within each split or globally.
minor comments (5)
  1. [Eq. (2)] Equation (2) has a formatting error: it is missing a closing parenthesis (it reads '(cos(2πt/T, sin(2πt/T )') and the comma placement after 'cos(2πt/T' is wrong; it should be the two-component vector (cos(2πt/T), sin(2πt/T)).
  2. [Section IV-C] The notation is confusing: the sentence 'the embeddings of d_ni, t_ni are denoted by d_ni, t_ni for simplicity' uses the same symbols for raw fields and embeddings; using bold or different symbols would improve clarity.
  3. [Section IV-B and Figure 3] The case study example in Section IV-B says the user's 'new choice is cheap train tickets,' but Figure 3 and the surrounding text do not clearly mark the temporal ordering across merchants; adding explicit timestamps or arrows to the figure would help.
  4. [Section VI-C] The case studies are qualitative and based on two manually selected defaulters; the paper should state that these examples are illustrative and not a systematic evaluation of interpretability, and ideally report how often the described patterns occur among detected defaulters.
  5. [General] The dataset and code are not publicly available, and the paper does not include an ethics or reproducibility statement beyond the data footnote; a clear statement about anonymization and any plan for sharing features or code would strengthen the work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; LBSF is an empirical architecture paper whose results are established by held-out evaluation rather than by derivation from its own inputs.

full rationale

LBSF is an empirical architecture paper. The central claim—that merchant-level folding plus multi-field encoding improves AUC and Recall@10% over baselines—is supported by end-to-end training and evaluation on a separate test split (Table II), not by an equation-level reduction from inputs to outputs. No target quantity is defined in terms of the model's own outputs, and no fitted parameter is relabeled as a prediction. The cited prior works, including those co-authored by members of this team, are used as related work and baselines rather than as load-bearing justifications for the proposed method; there is no imported uniqueness theorem or ansatz-smuggling citation. The unstated rule for selecting M=74 merchants and the unspecified cutoff between the behavior window and the default event are potential correctness or leakage concerns, but they are not instances of a derivation reducing to its own inputs by construction. Therefore, no circular step is identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the temporal ordering of behaviors and labels, the choice of merchant as grouping key, the per-user merchant selection rule, and the time-based split independence. No new physical entities are introduced; the model is a supervised learner trained on proprietary data.

free parameters (2)
  • M (number of merchants selected per user) = 74
    Set to 74 for all users in Section V-C; the selection criterion for a user's 74 merchants is not specified, and this choice determines the folded sequence structure.
  • Transformer and optimizer hyperparameters = d_model=128, heads=4, layers=1, lr=2e-4, batch=256, epochs=10
    Chosen in Section V-C without a reported sensitivity study; central results could depend on these settings.
assumptions (3)
  • domain assumption Default labels are assigned after service access, and observed payment behaviors occur before the default event; the label is not determined by the same behavior window used for prediction.
    Sections III-A and III-B define the business setup, but the paper never states the temporal alignment between the behavior window and the default date. If behaviors after default are included, the model could learn leakage.
  • domain assumption The merchant field is a stable, meaningful grouping key for payment behaviors, and folding by merchant does not destroy order information needed for risk assessment.
    This is the core modeling premise in Section IV-B and is supported only by ablations on the proprietary dataset.
  • domain assumption Splitting users by date ranges gives independent train, validation, and test sets with no user overlap and no harmful time-dependent distribution shift.
    Section V-A gives date ranges but no overlap check; the positive rate differs between train (7.2%) and validation/test (12.2%), which is not explained.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Financial Risk Assessment via Long-term Payment Behavior Sequence Folding." pith.science (2026). https://pith.science/paper/HW4XPG66

@misc{pith2026241115056,
  author       = {Pith},
  title        = {Pith review of: Financial Risk Assessment via Long-term Payment Behavior Sequence Folding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HW4XPG66}},
  note         = {Machine review of arXiv:2411.15056}
}
read the original abstract

Online inclusive financial services encounter significant financial risks due to their expansive user base and low default costs. By real-world practice, we reveal that utilizing longer-term user payment behaviors can enhance models' ability to forecast financial risks. However, learning long behavior sequences is non-trivial for deep sequential models. Additionally, the diverse fields of payment behaviors carry rich information, requiring thorough exploitation. These factors collectively complicate the task of long-term user behavior modeling. To tackle these challenges, we propose a Long-term Payment Behavior Sequence Folding method, referred to as LBSF. In LBSF, payment behavior sequences are folded based on merchants, using the merchant field as an intrinsic grouping criterion, which enables informative parallelism without reliance on external knowledge. Meanwhile, we maximize the utility of payment details through a multi-field behavior encoding mechanism. Subsequently, behavior aggregation at the merchant level followed by relational learning across merchants facilitates comprehensive user financial representation. We evaluate LBSF on the financial risk assessment task using a large-scale real-world dataset. The results demonstrate that folding long behavior sequences based on internal behavioral cues effectively models long-term patterns and changes, thereby generating more accurate user financial profiles for practical applications.

Figures

Figures reproduced from arXiv: 2411.15056 by the authors.

Figure 1
Figure 1. The performance of Transformer and BERT4Rec on real-world [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of LBSF. A. Overview of LBSF The architecture of LBSF is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The illustration of long-term payment behavior reorganizing at merchant level on a short example sequence. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Case study of defaulter Alice, whose four top-ranked merchants are a [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Case study of defaulter Bob, whose three top-ranked merchants are [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 31 canonical work pages

  1. [1]

    Per- sonalized finance advisory through case-based recommender systems and diversification strategies,

    C. Musto, G. Semeraro, P. Lops, M. De Gemmis, and G. Lekkas, “Per- sonalized finance advisory through case-based recommender systems and diversification strategies,” Decision Support Systems , vol. 77, pp. 100–111, 2015

    work page 2015

  2. [2]

    Fintech and the future of finance,

    J. Guild, “Fintech and the future of finance,” Asian Journal of Public Affairs, pp. 17–20, 2017

    work page 2017

  3. [3]

    Fintech: Emerging trends and the future of finance,

    H. Taherdoost, “Fintech: Emerging trends and the future of finance,” Financial Technologies and DeFi: A Revisit to the Digital Finance Revolution, pp. 29–39, 2023

    work page 2023

  4. [4]

    Digital finance and corporate financial fraud,

    G. Sun, T. Li, Y . Ai, and Q. Li, “Digital finance and corporate financial fraud,” International Review of Financial Analysis , vol. 87, p. 102566, 2023

    work page 2023

  5. [5]

    China’s internet finance boom and tyrannies of inclusion,

    N. Loubere, “China’s internet finance boom and tyrannies of inclusion,” China Perspectives, vol. 2017, no. 2017/4, pp. 9–18, 2017

    work page 2017

  6. [6]

    Financial credit risk assessment: a recent review,

    N. Chen, B. Ribeiro, and A. Chen, “Financial credit risk assessment: a recent review,” Artificial Intelligence Review , vol. 45, pp. 1–23, 2016

    work page 2016

  7. [7]

    Financial ratios, discriminant analysis and the prediction of corporate bankruptcy,

    E. I. Altman, “Financial ratios, discriminant analysis and the prediction of corporate bankruptcy,” The journal of finance , vol. 23, no. 4, pp. 589–609, 1968

    work page 1968

  8. [8]

    A survey of business failures with an emphasis on prediction methods and industrial applications,

    A. I. Dimitras, S. H. Zanakis, and C. Zopounidis, “A survey of business failures with an emphasis on prediction methods and industrial applications,” European journal of operational research , vol. 90, no. 3, pp. 487–513, 1996

    work page 1996

Show all 49 references
  1. [9]

    Statistical classification methods in consumer credit scoring: a review,

    D. J. Hand and W. E. Henley, “Statistical classification methods in consumer credit scoring: a review,” Journal of the Royal Statistical Society: Series A (Statistics in Society) , vol. 160, no. 3, pp. 523–541, 1997

    1997

  2. [10]

    Benchmarking state-of-the-art classification algorithms for credit scoring,

    B. Baesens, T. Van Gestel, S. Viaene, M. Stepanova, J. Suykens, and J. Vanthienen, “Benchmarking state-of-the-art classification algorithms for credit scoring,” Journal of the operational research society , vol. 54, pp. 627–635, 2003

    2003

  3. [11]

    The evaluation of consumer loans using support vector machines,

    S.-T. Li, W. Shiue, and M.-H. Huang, “The evaluation of consumer loans using support vector machines,” Expert Systems with Applications, vol. 30, no. 4, pp. 772–782, 2006

    2006

  4. [12]

    Unsupervised quadratic surface support vector machine with application to credit risk assessment,

    J. Luo, X. Yan, and Y . Tian, “Unsupervised quadratic surface support vector machine with application to credit risk assessment,” European Journal of Operational Research, vol. 280, no. 3, pp. 1008–1017, 2020

    2020

  5. [13]

    Recent developments in consumer credit risk assessment,

    J. N. Crook, D. B. Edelman, and L. C. Thomas, “Recent developments in consumer credit risk assessment,” European Journal of Operational Research, vol. 183, no. 3, pp. 1447–1465, 2007

    2007

  6. [14]

    Learning sequential behavior repre- sentations for fraud detection,

    J. Guo, G. Liu, Y . Zuo, and J. Wu, “Learning sequential behavior repre- sentations for fraud detection,” in 2018 IEEE international conference on data mining (ICDM) . IEEE, 2018, pp. 127–136

    2018

  7. [15]

    Spatio- temporal attention-based neural network for credit card fraud detection,

    D. Cheng, S. Xiang, C. Shang, Y . Zhang, F. Yang, and L. Zhang, “Spatio- temporal attention-based neural network for credit card fraud detection,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 01, 2020, pp. 362–369

    2020

  8. [16]

    Online credit payment fraud detection via structure-aware hierarchical recurrent neural network

    W. Lin, L. Sun, Q. Zhong, C. Liu, J. Feng, X. Ao, and H. Yang, “Online credit payment fraud detection via structure-aware hierarchical recurrent neural network.” in IJCAI, 2021, pp. 3670–3676

    2021

  9. [17]

    Neural hierarchical factorization machines for user’s event sequence analysis,

    D. Xi, F. Zhuang, B. Song, Y . Zhu, S. Chen, D. Hong, T. Chen, X. Gu, and Q. He, “Neural hierarchical factorization machines for user’s event sequence analysis,” in Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval ...

    2020

  10. [18]

    Design domain specific neural network via symbolic testing,

    H. Li, X. Fu, R. Wu, J. Xu, K. Xiao, X. Chang, W. Wang, S. Chen, L. Shi, T. Xiong, and Y . Qi, “Design domain specific neural network via symbolic testing,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , ser. KDD ’22. New York, NY , U...

    2022

  11. [19]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  12. [20]

    Fraud transactions detection via behavior tree with local intention calibration,

    C. Liu, Q. Zhong, X. Ao, L. Sun, W. Lin, J. Feng, Q. He, and J. Tang, “Fraud transactions detection via behavior tree with local intention calibration,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2020, pp. 3035– 3043

    2020

  13. [21]

    Learned temporal aggregations for fraud classification on e-commerce platforms,

    X. Ling, D. Yan, B. Alsallakh, A. Pandey, M. Bakshi, and P. Bhat- tacharya, “Learned temporal aggregations for fraud classification on e-commerce platforms,” in Companion Proceedings of the ACM Web Conference 2023, 2023, pp. 1365–1372

    2023

  14. [22]

    Intention-aware heterogeneous graph attention networks for fraud transactions detection,

    C. Liu, L. Sun, X. Ao, J. Feng, Q. He, and H. Yang, “Intention-aware heterogeneous graph attention networks for fraud transactions detection,” in Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , 2021, pp. 3280–3288

    2021

  15. [23]

    Cash-out user detection based on attributed heterogeneous information network with a hierarchical attention mechanism,

    B. Hu, Z. Zhang, C. Shi, J. Zhou, X. Li, and Y . Qi, “Cash-out user detection based on attributed heterogeneous information network with a hierarchical attention mechanism,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 946–953

    2019

  16. [24]

    Detecting cash-out users via dense subgraphs,

    Y . Ji, Z. Zhang, X. Tang, J. Shen, X. Zhang, and G. Yang, “Detecting cash-out users via dense subgraphs,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022, pp. 687–697

    2022

  17. [25]

    Credit risk and limits forecasting in e-commerce consumer lending service via multi-view-aware mixture-of-experts nets,

    T. Liang, G. Zeng, Q. Zhong, J. Chi, J. Feng, X. Ao, and J. Tang, “Credit risk and limits forecasting in e-commerce consumer lending service via multi-view-aware mixture-of-experts nets,” in Proceedings of the 14th ACM international conference on web search and data mining , 2...

    2021

  18. [26]

    Financial defaulter detection on online credit payment via multi-view attributed heterogeneous information network,

    Q. Zhong, Y . Liu, X. Ao, B. Hu, J. Feng, J. Tang, and Q. He, “Financial defaulter detection on online credit payment via multi-view attributed heterogeneous information network,” in Proceedings of the web conference 2020 , 2020, pp. 785–795

    2020

  19. [27]

    Intelligent financial fraud detection practices in post-pandemic era,

    X. Zhu, X. Ao, Z. Qin, Y . Chang, Y . Liu, Q. He, and J. Li, “Intelligent financial fraud detection practices in post-pandemic era,”The Innovation, vol. 2, no. 4, 2021

    2021

  20. [28]

    A survey on user behavior modeling in recommender systems,

    Z. He, W. Liu, W. Guo, J. Qin, Y . Zhang, Y . Hu, and R. Tang, “A survey on user behavior modeling in recommender systems,” in Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, E. Elkind, Ed. International Joint Conferences o...

    2023 doi

  21. [29]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1, 2019, p. 2

    2019

  22. [30]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  23. [31]

    A survey on vision transformer,

    K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C. Xu, Y . Xuet al., “A survey on vision transformer,” IEEE transactions on pattern analysis and machine intelligence, vol. 45, no. 1, pp. 87–110, 2022

    2022

  24. [32]

    Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,

    A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma et al. , “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,” Proceedings of the National Academy of Sciences , vol. 118, no....

    2021

  25. [33]

    Efficient transformers: A survey,

    Y . Tay, M. Dehghani, D. Bahri, and D. Metzler, “Efficient transformers: A survey,” ACM Computing Surveys , vol. 55, no. 6, pp. 1–28, 2022

    2022

  26. [34]

    Reformer: The efficient trans- former,

    N. Kitaev, L. Kaiser, and A. Levskaya, “Reformer: The efficient trans- former,” in International Conference on Learning Representations, 2019

    2019

  27. [35]

    Transformer-xl: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y . Yang, J. G. Carbonell, Q. Le, and R. Salakhutdi- nov, “Transformer-xl: Attentive language models beyond a fixed-length context,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019, pp. 2978–2988

    2019

  28. [36]

    Linformer: Self-attention with linear complexity,

    S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  29. [37]

    Informer: Beyond efficient transformer for long sequence time-series forecasting,

    H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, “Informer: Beyond efficient transformer for long sequence time-series forecasting,” in Proceedings of the AAAI conference on artificial intel- ligence, vol. 35, no. 12, 2021, pp. 11 106–11 115

    2021

  30. [38]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  31. [39]

    Xgboost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining , 2016, pp. 785–794

    2016

  32. [40]

    The perceptron: a probabilistic model for information storage and organization in the brain

    F. Rosenblatt, “The perceptron: a probabilistic model for information storage and organization in the brain.” Psychological review, vol. 65, no. 6, p. 386, 1958

    1958

  33. [41]

    Bidirectional long short- term memory networks for relation classification,

    S. Zhang, D. Zheng, X. Hu, and M. Yang, “Bidirectional long short- term memory networks for relation classification,” in Proceedings of the 29th Pacific Asia conference on language, information and computation, 2015, pp. 73–78

    2015

  34. [42]

    Longformer: The long- document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,” arXiv preprint arXiv:2004.05150 , 2020

    2004 arXiv

  35. [43]

    Retentive network: A successor to transformer for large language models,

    Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei, “Retentive network: A successor to transformer for large language models,” arXiv preprint arXiv:2307.08621 , 2023

    2023 arXiv

  36. [44]

    Rwkv: Reinventing rnns for the transformer era,

    B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Bi- derman, H. Cao, X. Cheng, M. Chung, L. Derczynski et al. , “Rwkv: Reinventing rnns for the transformer era,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023, pp. 14 048–14 077

    2023

  37. [45]

    Mamba: Linear-time sequence modeling with selective state spaces,

    A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752 , 2023

    2023 arXiv

  38. [46]

    Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,

    F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang, “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” in Proceedings of the 28th ACM international confer- ence on information and knowledge management , 2019, pp. 1441–1450

    2019

  39. [47]

    S3-rec: Self-supervised learning for sequential recom- mendation with mutual information maximization,

    K. Zhou, H. Wang, W. X. Zhao, Y . Zhu, S. Wang, F. Zhang, Z. Wang, and J.-R. Wen, “S3-rec: Self-supervised learning for sequential recom- mendation with mutual information maximization,” in Proceedings of the 29th ACM international conference on information & knowledge managem...

    2020

  40. [48]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019

    2019

  41. [49]

    Fixing weight decay regularization in adam,

    I. Loshchilov, F. Hutter et al. , “Fixing weight decay regularization in adam,” arXiv preprint arXiv:1711.05101 , vol. 5, 2017

    2017 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.