Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Frozen CoLES user embeddings, injected into Mamba as an initial hidden state or prefix token, consistently improve prediction accuracy and convergence speed across three transaction-sequence benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:23 UTC pith:QHIMZ3BH

load-bearing objection Honest, modest Mamba+CoLES extension; central comparison is confounded by transductive pretraining asymmetry and missing error bars. the 3 major comments →

arxiv 2607.20228 v1 pith:QHIMZ3BH submitted 2026-07-22 cs.LG

User-Centric Modeling of Transactional Sequences with Explainable State Space Models

Ivan Palagin This is my paper
classification cs.LG
keywords State Space ModelsMambaCoLEScontrastive learningtransactional event sequencesuser embeddingsexplainabilityIntegrated Gradients
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that contrastive user representations and selective state space models are complementary: a frozen CoLES embedding, injected into Mamba as either the initial hidden state or a prepended context token, gives the model a user-level prior before it sees any transactions. Across three public datasets—banking age-group prediction, multi-label product acquisition, and e-commerce purchase prediction—both hybrid variants are reported to beat plain Mamba and CoLES-with-linear-head on every dataset, with the largest gain +3.2 percentage points on the banking age task and +2.6 points on the product-acquisition task. The hybrids also reach peak validation performance in 2–3 epochs versus 6 for plain Mamba. The authors themselves note the gains are moderate, suggesting the two encoders extract partially overlapping information from the same sequences.

Core claim

The central claim is that a frozen CoLES user embedding is a useful prior when injected into a Mamba state space model, and that two injection strategies—initializing the first Mamba layer's hidden state via h0 = W2 σ(W1 e_CoLES), and prepending a linearly projected embedding as a prefix token—both outperform standalone Mamba and CoLES with a linear classifier on all three datasets. The reported test-set numbers are 0.386 and 0.384 accuracy on Age (vs 0.354), 0.737 and 0.727 mean ROC-AUC on MBD (vs 0.711), and 0.695 and 0.699 ROC-AUC on Taobao (vs 0.692). The paper also claims 2–3x faster convergence and, through discretization-step maps and Integrated Gradients, finds selective event filter

What carries the argument

The load-bearing mechanism is the injection of a pretrained contrastive user embedding into a selective state space model. Mamba is a linear-time sequence model whose input-dependent gating (the per-position discretization step Δ_t) decides how much to update the hidden state; CoLES is a self-supervised contrastive encoder that compresses a user's event history into a fixed vector. The hybrid replaces Mamba's zero initial state with a learnable projection of the CoLES vector (h0 = W2 σ(W1 e_CoLES)), or prepends the projected vector as an extra context token. The user prior is thus available from the first step, and Mamba's selective mechanism can weight it against the incoming transaction to

Load-bearing premise

In Section 3.5, the comparison assumes plain Mamba and CoLES+linear were tuned with the same hyperparameter budget and training constraints as the hybrids; the paper details per-model automated tuning for the proposed methods but does not state that the baselines received the same 50-trial budget, 10-epoch cap, halved sequence lengths, and MBD subsample.

What would settle it

Rerun plain Mamba and CoLES+linear under the exact tuning protocol used for the hybrids—same per-pair automated search budget, same 10-epoch cap, same halved sequence lengths, same MBD subsample—and check whether the reported gains (+3.2pp, +2.6pp, +0.7pp) and the 2–3x convergence advantage persist. If they shrink to noise, the architectural contribution is not established.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Frozen CoLES embeddings are a viable user-level prior for SSM encoders, not just for linear probes.
  • The 2–3x faster convergence means hybrid training needs roughly a third of the gradient steps, a practical saving where training compute is constrained.
  • Because CoLES+linear underperforms plain Mamba on all three datasets, the sequential context provided by Mamba carries information the contrastive embedding alone does not.
  • Cold-start users (no precomputed embedding) fall back to a zero vector, which reduces the hybrid exactly to plain Mamba, so the deployment path is safe.
  • The Δ_t dips observed on Taobao suggest that on behavior-rich e-commerce logs the model learns to suppress low-intent events and amplify high-intent ones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An untested but natural extension: joint end-to-end training of CoLES and Mamba (which the paper lists as future work) would likely change how much the contrastive prior and the SSM overlap, potentially yielding larger gains than frozen injection.
  • If the equal-tuning assumption holds, the same 'frozen embedding as initial state' recipe may transfer to other SSM variants such as Mamba-2, or to other pretrained user embeddings beyond CoLES.
  • The explainability finding—selective Δ_t dips on e-commerce but not banking—could be turned into a diagnostic: a dataset's Δ_t variance may predict how much benefit a user prior provides.
  • A direct testable prediction follows from the convergence result: on longer sequences (L > 1000), the O(L) advantage plus the prior should widen the gap over Transformers, since the paper's current datasets have modest lengths.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two hybrid architectures for transactional event sequences that inject a frozen CoLES user embedding into a Mamba state-space model: hidden-state initialization (replacing h0) and prefix concatenation (prepending a projected token). On three public datasets (Age, MBD, Taobao), it reports that both hybrids consistently beat plain Mamba and a CoLES+linear baseline, converge 2–3x faster, and are amenable to explainability analysis via discretization-step maps and Integrated Gradients. The central claim is that a frozen self-supervised user prior is a useful, inexpensive inductive bias for SSM-based sequence encoders.

Significance. If the comparative claim is established, this is a simple and practically attractive way to combine contrastive user representations with a linear-time SSM encoder; the faster convergence and cold-start fallback are also useful properties. The paper is honest about the moderate size of the gains and the overlap between CoLES and Mamba information. However, the current evidence is not yet sufficient: the evaluation has no error bars, and the CoLES pretraining uses transductive access to test input sequences, which confounds the hybrid-vs-Mamba comparison.

major comments (3)
  1. [§3.3, §3.6] CoLES is pretrained on the train∪test union ('no label leakage'). This gives the hybrid models and the CoLES+linear baseline access to test input sequences during unsupervised pretraining, while the plain Mamba baseline does not have any such pretraining exposure. On the small datasets (Age: 24K clients, Taobao: 18K clients), this extra transductive data access can plausibly explain part of the reported gains. Please add an ablation with CoLES pretrained on train-only (or train+validation) data, or otherwise give plain Mamba the same unsupervised test-input access. Without this control, the comparison conflates architectural injection with data-access asymmetry.
  2. [§4.1, Table 3] All test metrics are single numbers with no multiple seeds, confidence intervals, or significance tests. The Taobao gain is +0.7pp, which is within typical run-to-run noise for deep sequence models; the Age and MBD gains may be real, but the current table does not establish 'consistently outperform all baselines'. Please report mean±std over at least 3–5 seeds and include a paired test or bootstrap confidence interval for each dataset/model. This is load-bearing for the central claim.
  3. [§3.5, §3.6] It is not clear that all baselines received exactly the same tuning and training budget. The text says 'For each (architecture, dataset) pair' Optuna is run with epochs capped at 10 and sequence lengths halved, but §3.6 later describes a training protocol with max 100 epochs and early stopping. Please state explicitly whether plain Mamba and CoLES+linear were tuned/trained with the same epoch cap, sequence-length halving, MBD subsample, and number of Optuna trials, and clarify how the 10-epoch cap relates to the 100-epoch protocol. If baselines were not treated identically, the gains in Table 3 may reflect differential tuning rather than the architectural contribution.
minor comments (5)
  1. [§3.3] The CoLES+linear baseline is underspecified: how is the linear classifier trained (loss, epochs, regularization), and does it use the same Optuna budget as the other models?
  2. [§4.4, Figure 2] The explainability analysis samples only 16 test clients per dataset, with no variance or sensitivity analysis. The Taobao 'selective filtering' dips could be idiosyncratic to this small sample. Please state this limitation explicitly or provide aggregate statistics.
  3. [§4.4, Figure 2 caption] The caption says 'Top row: normalized discretization-step (Δt) curves (Age, Taobao, MBD)' while the surrounding text discusses 'Age, MBD, Taobao'. The order should be made consistent.
  4. [References] Reference [16] has a formatting error in the author list: 'Daria Denisova Anton Klenitskiy' should have a comma between authors.
  5. [§4.2] The 2–3x faster-convergence claim is based only on peak validation epochs (2–3 vs. 6) but no learning curves or epoch tables are shown. Please include the underlying curves or a table of convergence epochs.

Circularity Check

0 steps flagged

No circular derivation; transductive CoLES pretraining is a fairness concern, not circularity.

full rationale

The paper contains no mathematical derivation whose conclusion is equivalent to its inputs. The central claims are empirical comparisons on three public benchmarks. CoLES embeddings are pretrained self-supervisely (no labels) and then injected into Mamba via a learned projection; the downstream Mamba is trained with supervision. There is no fitted parameter that is then renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via a self-citation. The references to LATTE, EAFD, FinTRACE, and related works are context citations and are not load-bearing for the paper's own experimental results. The only notable issue is that CoLES is trained on the train+test union (Section 3.3: 'CoLES is trained self-supervisedly on the full dataset (train + test union; no label leakage)'; Section 3.6: 'CoLES pretraining runs on the train∪test union for 100K gradient steps'), giving hybrid models access to test input sequences during pretraining while plain Mamba does not receive that unsupervised exposure. This is a valid benchmark-fairness or transductive-leakage concern, but it is not a circularity of derivation: the CoLES embeddings are not derived from the target labels and the claim does not reduce by construction to a fitted input. Therefore no circular step is identified.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The ledger is dominated by hyperparameters fitted to validation data and by assumptions imported from prior work (Mamba, CoLES, XAI methods). No new physical or mathematical entities are introduced. The most load-bearing assumptions are the fairness of the baseline comparison and the acceptance of train+test union pretraining as leakage-free.

free parameters (7)
  • learning rate = ~1e-4 optimum (range 1e-5 to 1e-1)
    Tuned per (architecture, dataset) with Optuna; higher values cause instability (Section 4.3).
  • number of Mamba layers = 3 optimum
    Three layers consistently outperform shallower configurations (Section 4.3).
  • state-space dimension = 32 for Age; 16–32 for short sequences
    Long sequences prefer 32; short MBD shows less sensitivity (Section 4.3).
  • timestamp encoding = diff/cat/none (selected per dataset)
    Tunable hyperparameter in preprocessing (Section 3.2).
  • categorical/numerical embedding dimensions = [15, 40] optimal within [4, 64] search
    Balance expressiveness and parameter cost (Section 4.3).
  • CoLES pretraining steps and batch size = 100K steps, batch size 32–256
    Chosen by the authors; embeddings cached after pretraining (Section 3.6).
  • training budget reduction = epochs capped at 10, sequence lengths halved, MBD 1M-sequence subsample
    Used for Optuna tuning; affects which hyperparameters are chosen and possibly comparison fairness (Section 3.5).
axioms (6)
  • standard math Mamba's selective SSM efficiently models long sequences in O(L) time
    Adopted from Gu & Dao (2023), cited as [8]; the paper does not re-derive it.
  • domain assumption CoLES contrastive embeddings capture stable user profiles from event sequences
    Adopted from Babaev et al. (2022), cited as [3]; the paper assumes this for its user prior.
  • domain assumption CoLES pretraining on the full train+test union does not cause label leakage
    Stated in Section 3.3: 'no label leakage'—but test features are observed during unsupervised pretraining, which may still leak distributional information.
  • ad hoc to paper Baseline models were compared under the same tuning/training protocol
    Implied by the comparison in Table 3 but never explicitly stated; Section 3.5 only describes tuning for the two proposed strategies.
  • domain assumption Discretization-step maps and Integrated Gradients faithfully attribute model decisions
    Adopted from Ali et al. and Sundararajan et al.; used as interpretability evidence in Section 4.4.
  • ad hoc to paper 16 test clients per dataset are representative for explainability conclusions
    The explainability analysis samples 16 clients and draws qualitative conclusions about selective event filtering (Section 4.4).

pith-pipeline@v1.3.0-alltime-deepseek · 6193 in / 9963 out tokens · 99973 ms · 2026-08-01T10:23:55.687990+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of User-Centric Modeling of Transactional Sequences with Explainable State Space Models." pith.science (2026). https://pith.science/paper/QHIMZ3BH

@misc{pith2026260720228,
  author       = {Pith},
  title        = {Pith review of: User-Centric Modeling of Transactional Sequences with Explainable State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QHIMZ3BH}},
  note         = {Machine review of arXiv:2607.20228}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We propose a hybrid approach for user-centric modeling of transactional event sequences that combines contrastive representation learning (CoLES) with State Space Models (SSMs). While contrastive methods yield high-quality compressed user representations, existing encoders -- RNNs and Transformers -- suffer from vanishing gradients or quadratic complexity, respectively. Mamba, a selective SSM, efficiently handles long-range dependencies but remains underexplored for personalized user analysis. We investigate two integration strategies: (1)~initializing the Mamba hidden state with a CoLES embedding, and (2)~prepending the projected CoLES embedding as a prefix token to the input sequence. Both approaches supply the model with an informative user prior from the first step. Experiments on three public datasets -- Age (multiclass age-group prediction), MBD (multi-label product acquisition), and Taobao (binary purchase prediction) -- demonstrate consistent improvements over standalone Mamba and CoLES with a linear classifier, with the hybrid models converging 2--3$\times$ faster than the plain SSM baseline. Explainability analysis via discretization-step maps and Integrated Gradients reveals selective event filtering on behavior-rich datasets and identifies the most informative transaction features.

Figures

Figures reproduced from arXiv: 2607.20228 by Ivan Palagin.

Figure 1
Figure 1. Figure 1: Proposed hybrid architectures. Left: hidden￾state initialization—the CoLES embedding is projected into Mamba’s initial state h0. Right: prefix concatenation—the embedding is prepended as a context token before the trans￾action sequence x𝑡 . [32, 256]; embeddings are extracted and cached for all users. Plain Mamba and both hybrid models are then trained on the training split with early stopping (patience = … view at source ↗
Figure 2
Figure 2. Figure 2: Explainability results for Mamba with hidden-state [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 1 canonical work pages

  1. [1]

    Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-Generation Hyperparameter Optimization Frame- work. InProceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining. 2623–2631

    2019

  2. [2]

    Ali, Itamar Zimerman, and Lior Wolf

    Ameen A. Ali, Itamar Zimerman, and Lior Wolf. 2025. The Hidden Attention of Mamba Models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 1516–1534

    2025

  3. [3]

    Dmitry Babaev et al. 2022. CoLES: Contrastive Learning for Event Sequences with Self-Supervision. InProceedings of the 2022 International Conference on Management of Data. 1190–1199

    2022

  4. [4]

    Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019. Behavior Se- quence Transformer for E-Commerce Recommendation in Alibaba. InProceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data

    2019

  5. [5]

    Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized Models and Efficient Algorithms through Structured State Space Duality.arXiv preprint arXiv:2405.21060(2024)

    Pith/arXiv arXiv 2024

  6. [6]

    Kireev, Andrey Savchenko, and Maksim Makarenko

    Egor Fadeev, Dzhambulat Mollaev, Aleksei Shestov, Dima Korolev, Omar Zoloev, Ivan A. Kireev, Andrey Savchenko, and Maksim Makarenko. 2025. LATTE: Learning Aligned Transactions and Textual Embeddings for Bank Clients. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2635–2647

    2025

  7. [7]

    Albert Gu et al . 2020. HiPPO: Recurrent Memory with Optimal Polynomial Projections. InAdvances in Neural Information Processing Systems, Vol. 33. 1474– 1487

    2020

  8. [8]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-Time Sequence Modeling with Selective State Spaces.arXiv preprint arXiv:2312.00752(2023)

    Pith/arXiv arXiv 2023

  9. [9]

    Albert Gu, Karan Goel, and Christopher Ré. 2022. Efficiently Modeling Long Sequences with Structured State Spaces.arXiv preprint arXiv:2111.00396(2022)

    Pith/arXiv arXiv 2022

  10. [10]

    Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk

  11. [11]

    Viktor Moskvoretskii et al. 2024. MLEM: Generative and Contrastive Learning for Event Sequences.arXiv preprint arXiv:2401.15935(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Denis Osin, Ivan Udovichenko, Evgeny Shvetsov, et al. 2025. EBES: Easy Bench- marking for Event Sequences. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Vol. 2. 5730–5741

    2025

  13. [13]

    Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev, Andrey Savchenko, and Maksim Makarenko. 2026. Embedding- Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences.arXiv preprint arXiv:2603.15713(2026)

    arXiv 2026

  14. [14]

    Artem Sakhno, Daniil Tomilov, Yuliana Shakhvalieva, Inessa Fedorova, Daria Ruzanova, Omar Zoloev, Andrey Savchenko, and Maksim Makarenko. 2026. Fin- TRACE: Financial Transaction Retrieval and Contextual Evidence for Knowledge- Grounded Reasoning.arXiv preprint arXiv:2603.15459(2026)

    arXiv 2026

  15. [15]

    Alexander Shestov et al. 2025. LLM4ES: Learning User Embeddings from Event Se- quences via Large Language Models. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 5238–5242

    2025

  16. [16]

    Aleksei Shestov, Daria Denisova Anton Klenitskiy, Andrey Savchenko Amurkhan Dzagkoev, Daniil Petrovich, and Maksim Makarenko. 2025. Topologi- cal Metric for Unsupervised Embedding Quality Evaluation. InAdvances in Infor- mation Retrieval. Springer, Cham, 1235–1260. doi:10.1007/978-3-032-21300-6_51

    work page doi:10.1007/978-3-032-21300-6_51 2025

  17. [17]

    Shriya Somvanshi et al. 2025. From S4 to Mamba: A Comprehensive Survey on Structured State Space Models.arXiv preprint arXiv:2503.18970(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  19. [19]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. InProceedings of the 34th International Conference on Machine Learning, Vol. 70. 3319–3328

    2017

  20. [20]

    Yi Tay et al. 2021. Long Range Arena: A Benchmark for Efficient Transformers. InInternational Conference on Learning Representations

    2021

  21. [21]

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, et al. 2024. An Empirical Study of Mamba-Based Language Models.arXiv preprint arXiv:2406.07887(2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Zichuan Yang and Yongzhi Wang. 2025. EVM-Fusion: An Explainable Vi- sion Mamba Architecture with Neural Algorithmic Fusion.arXiv preprint arXiv:2505.17367(2025)

    Pith/arXiv arXiv 2025

  23. [23]

    Daixuan Yuan et al. 2024. ReMamba: Equip Mamba with Effective Long-Sequence Modeling.arXiv preprint arXiv:2408.15496(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Vladislav Yugay and Alexey Zaytsev. 2025. Uniting Contrastive and Generative Learning for Event Sequences Models.arXiv preprint arXiv:2408.09995(2025)

    Pith/arXiv arXiv 2025

  25. [2016]

    In International Conference on Learning Representations

    Session-Based Recommendations with Recurrent Neural Networks. In International Conference on Learning Representations

  26. [2019]

    InProceedings of the 28th ACM International Conference on Information and Knowledge Management

    BERT4Rec: Sequential Recommendation with Bidirectional Encoder Rep- resentations from Transformer. InProceedings of the 28th ACM International Conference on Information and Knowledge Management. 1441–1450