Pith. sign in

REVIEW 3 major objections 6 minor 22 references

multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A decoder-only transformer with a class head and per-class Gaussian value head models mixed categorical and numeric sequences by maximizing the joint log-likelihood of the next token's class and value.

desk verdict A clean extension of transformer LMs to mixed-type sequences with real gains on EHR/ECG, but the timing-calibration claim needs support and the baseline set is too narrow for 'state of the art.' read the letter →

arxiv 2505.21680 v2 pith:JG5TX6XP submitted 2025-05-27 cs.LG cs.AI

Andrew J. Loza , Jun Yup Kim , Shangzheng Song , Yihang Liu , Joseph J. Y. Sung , R Andrew Taylor , Dennis L. Shung This is my paper
classification cs.LGcs.AI
keywords transformermultivariatetimeseriesmixedcategoricalandnumericdatainformativesamplingcontinuousvaluepredictionirregularelectronichealthrecordslikelihood-basedloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

multivariateGPT extends the next-token prediction task of a decoder-only transformer from predicting a token to estimating the joint distribution of the next token's class and its continuous value. The paper shows that by flattening a multivariate time series into ordered (class, value) tuples, including the elapsed time as its own class, a single architecture can model categorical data, numbers, and the timing of observations. On clinical time series from eICU it reports 40–60% lower error than trajectory flow matching for heart rate and blood pressure prediction, and it is the only model in the comparison that predicts both the value and the time of the next observation. On synthetic damped oscillators it generalizes to trajectories not present in training, while discrete-token transformers with 10 or 100 bins do not.

What carries the argument

The load-bearing mechanism is the class-value token decomposition with a twin-head output. Each token is a pair (c_i, v_i); categorical classes are expanded so each class has a single value, while numeric tokens embed as E_{c_i} + f_{c_i}(v_i) for a learnable class embedding and value map. The model's final layer feeds two heads: a class head that softmaxes over class identities and a value head that outputs the mean and softplus-standard-deviation of a normal distribution for each class. Training minimizes the sum of the cross-entropy over classes and the negative log-likelihood of the true value under the predicted Gaussian. This turns next-token prediction into a joint likelihood over class and value, gives continuous numeric representation with no binning, and produces per-observation variance estimates that the paper shows to be well calibrated.

What would settle it

Retrain the model on the eICU data with a random permutation of the measurement order within each timestamp, keeping the loss unchanged; if class accuracy or timing MSE changes materially, the imposed ordering is carrying information that the flattening assumption treats as irrelevant.

Watch

Extended reading notes

Core claim

The paper's central claim is that a single autoregressive transformer can represent numeric values continuously and provide calibrated uncertainty by decomposing the joint distribution of the next token into P(c_i, v_i) = P(c_i) P(v_i | c_i), with a standard softmax class head and a per-class Gaussian value head. The loss is the sum of the class negative log-likelihood and the conditional value negative log-likelihood. Under the flattening of each timestamp into an ordered sequence of (class, value) pairs that includes the elapsed time as its own class-value pair, the model estimates the joint distribution over which measurements occur, what values they take, and when they occur. The paper reports that this approach outperforms discrete-token transformers and TFM-ODE on eICU sepsis data, reconstructs 12-lead electrocardiograms with smaller error than discrete baselines, and predicts class and value on Physionet ICU data with the same class accuracy but far lower value error.

Load-bearing premise

The load-bearing premise is that flattening each timestamp into a fixed measurement order plus a single elapsed-time token preserves all information needed for informative sampling and for the joint distribution over which measurements occur; if that flattening discards information, the central claim about capturing informative irregular timing collapses.

Editorial extensions

If this is right

  • Numeric predictions no longer depend on bin resolution; a model with a small vocabulary can represent values continuously and interpolate between training values.
  • The same architecture can predict the elapsed time until the next observation, because time is just another class-value tuple, which no neural-ODE baseline in the comparison can do.
  • Per-token Gaussian variance estimates give calibrated predictive intervals, with eICU coverage fractions near 0.95 for heart rate and MAP.
  • The method is compatible with a pure language modeling objective: a document or a text field can be treated as a sequence of class-value tokens within the same loss.
  • Vocabularies shrink because no discretization into bins is needed, reducing the number of classes the model must learn.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper reports that arbitrary lexicographic order is used when no natural order exists; if a permutation-invariant set representation matched or beat the flattened-sequence model on the same data, the imposed ordering would not be information-preserving.
  • Inference: the conditional-Gaussian head, which the authors flag as a limitation, will miss multi-modal or bounded distributions; a mixture-of-Gaussians or per-class distribution family is a direct extension.
  • Inference: because timing is a token in the same sequence, the model can be sampled autoregressively to produce full future trajectories, not just point forecasts; the paper mentions but does not evaluate this Monte Carlo rollout use.
  • Inference: a targeted test of the informative-sampling claim would perturb a recorded value in context and check whether the model's predicted time-to-next-observation distribution shifts accordingly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes multivariateGPT, a decoder-only transformer that models sequences of mixed categorical and numeric data using an autoregressive decomposition of the joint distribution over class and value. Each timestamp is flattened into a sequence of class-value tuples, including elapsed time as a special class, and the model is trained with a joint negative log-likelihood that combines a softmax class head with a per-class Gaussian value head. The authors claim this extends next-token prediction to likelihood-based estimation of continuous values with uncertainty, supports informative irregular sampling by predicting observation timing, and outperforms discrete-token transformers and TFM-ODE on clinical time-series tasks. Experiments cover damped harmonic oscillators, eICU sepsis data, MIMIC-IV ECG lead reconstruction, and Physionet ICU data, with comparisons to discrete-token transformers, a frozen LLM, and TFM-ODE. The paper includes an anonymous code repository and detailed training configuration tables.

Significance. The central formulation is mathematically clean and the loss decomposition in Eqs. (1)-(8) is sound; if the empirical claims hold, the method offers a simple, unified way to bring continuous values and uncertainty into autoregressive transformers without discrete tokenization. The authors deserve credit for releasing code, reporting calibration on MAP and heart rate, and demonstrating a clear generalization failure of discrete tokenizers on the oscillator task. However, the significance is currently tempered by two gaps: the arbitrary ordering inside timestamps is never stress-tested, and the Gaussian likelihood for the time interval is not validated beyond mean-squared error, despite timing prediction being a headline contribution. The comparison set also omits the continuous-numeric tokenization baselines discussed in the paper itself. These issues are addressable within a revision, so the work has solid potential but is not yet fully supported.

major comments (3)
  1. [Section 2.1, Eqs. (2)-(3)] The flattening in Section 2.1 imposes a lexicographic ordering over co-occurring measurements inside a timestamp when no natural order exists, and Eq. (2) conditions each measurement on that ordering. The paper provides no sensitivity analysis over the ordering (e.g., random permutations or different sort keys) on any data set, so the claim in Section 3.2.3 that the model captures the joint distribution over which measurements occur is not yet supported; if the ordering affects predictive performance, the decomposition is not a faithful factorization of the timestamp-level joint distribution.
  2. [Section 2.3, Eq. (7); Table 2; Fig. 3] The elapsed-time token tau is modeled with a conditional Gaussian in Eq. (7), but the paper evaluates timing predictions only by MSE (Table 2) and reports calibration only for MAP and heart rate (Fig. 3, Table 3). Time intervals in EHR data are strictly positive and right-skewed, with likely point masses at scheduled measurement times, so a Gaussian likelihood is questionable; the authors themselves acknowledge this in the Limitations section. Because predicting observation timing is a headline contribution (Section 5), the paper needs to provide calibration or residual diagnostics for tau (e.g., coverage of predictive intervals or PIT histograms) to substantiate the claim of calibrated uncertainty for informative sampling.
  3. [Section 3.2.1, Table 1; Section 4] The empirical comparisons are limited to discrete-token transformers, TFM-ODE, and a frozen LLM; despite the Related Work discussion of continuous numeric embeddings such as XVal and MMD and regression-like numeric losses, no such model is included as a baseline. As a result, Table 1's improvement over discrete tokenization does not isolate the benefit of the proposed continuous value head and embedding; a continuous-numeric baseline on eICU or Physionet is needed to support the claim that the proposed representation, rather than the transformer backbone alone, drives the reported gains.
minor comments (6)
  1. [Algorithm 1] The algorithm title contains a typo: "Mutivariate" should be "Multivariate".
  2. [Section 3.2.3] The sentence "The Physionet ICU data set contains measurements collected in a sparse manner across a 36 categorical and classes" is malformed; it should read "36 categorical and numeric classes" or similar.
  3. [Table 4] There is a sentence fragment immediately after Table 4: "the discrete model erroneously predicting tokens representing extreme values." This should be integrated into a complete sentence or moved to the main text.
  4. [Section 2.3, Eq. (7)] The description of v_j as "a vector with one non-zero element equal to the correct value at the index of the correct class" is confusing; v_j is the scalar value of the correct class, not a vector of length C with a single non-zero entry (that is the role of c_j). Please clarify the notation.
  5. [Section 3.2.2] The word "autoregressivelly" is misspelled; it should be "autoregressively."
  6. [Section 5] The phrase "retrogressively modeling any database" in the conclusion is unclear; likely "retrospectively" or "generatively" was intended. Please correct the wording.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model is a forward likelihood model trained and evaluated on external data, with the only author-overlapping citation used as a baseline rather than as load-bearing support.

full rationale

The derivation chain begins with the chain-rule decomposition in Eqs. 1-4, which is an exact probabilistic identity and does not encode any of the paper's empirical conclusions. The loss in Eq. 7 defines a conditional Gaussian negative log-likelihood; the predicted means and variances are optimized parameters, not quantities derived from the targets by construction. All empirical claims are tested on held-out data: trajectory reconstruction and generalization in Section 3.1, held-out value and timing MSE in Tables 1-2, calibration in Fig. 3 and Table 3, and reconstruction/classification metrics in Tables 4-5. No reported prediction is constructed from the evaluation targets; the model is judged against them. The only author-overlapping citation is Zhang et al. (2024), used as a baseline and as the source of the eICU data split and TFM-ODE implementation. That citation is comparative, not justificatory: it does not supply a premise from which the present model's correctness follows. The paper's acknowledged limitation that the Gaussian parameterization may poorly capture time, count, or ordinal data (Limitations section) is a validation gap and a direction for future work, not a circular step. In particular, the absence of calibration diagnostics for the tau timing predictions is a legitimate correctness concern, but it does not imply that the timing predictions are fitted inputs renamed as predictions. The central method is self-contained: an autoregressive sequence model with a joint class/value likelihood, evaluated against external benchmarks. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The ledger is clean: no new particles or mediators, no explicit free parameters beyond normal model weights and dataset-specific preprocessing scalers. The load is carried by standard probability factorization and by two domain assumptions (within-timestamp ordering and Gaussian value likelihood).

assumptions (4)
  • standard math Joint distribution factorizes by the chain rule into conditional next-event probabilities (Eq. 1-4).
    Probability chain rule; used to justify the autoregressive sequence task in Section 2.1.
  • domain assumption Within each timestamp, measurements can be given an order (lexicographic when no natural order) without changing the modeled distribution.
    Section 2.1: 'A specific ordering can be imposed over the xj within X which respects the required sequence order and otherwise uses a lexicographic sort.' No sensitivity analysis is provided.
  • domain assumption Elapsed time tau, encoded as a numeric token in the flattened sequence, captures the informative sampling process.
    Section 2.1 and Algorithm 1 convert time deltas to class-value tuples; the paper motivates this by citing informative missingness, but does not prove that a scalar token representation is sufficient.
  • domain assumption Numeric values are conditionally Gaussian given class and context (Eq. 6-7).
    The value head outputs mu and sigma per class with softplus; the paper states in Limitations that this is a simplification for time, count, and ordinal data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data." pith.science (2026). https://pith.science/paper/JG5TX6XP

@misc{pith2026250521680,
  author       = {Pith},
  title        = {Pith review of: multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JG5TX6XP}},
  note         = {Machine review of arXiv:2505.21680}
}
read the original abstract

Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data.

Figures

Figures reproduced from arXiv: 2505.21680 by the authors.

Figure 1
Figure 1. Diagram of Model Architecture. Trajectory reconstruction: The first task was the reconstruction of trajectories from the training data using the seed of the first 5 points ( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Simple harmonic oscillator trajectories. Columns show results from each model type. First [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. QQ Plots. Theoretical quantiles are plotted against the sample quantiles for each model [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Additional simple harmonic oscillator experiments. Rows show task and columns show [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Example lead reconstructions of a limb lead (III) and precordial lead (V2). [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 3 canonical work pages

  1. [1]

    Interleaving Text and Number Embeddings to Solve Mathemathics Problems

    M. Alberts, G. Gabrieli, and I. E. Morales. Interleaving text and number embeddings to solve mathemathics problems. arXiv preprint arXiv:2410.19353,

    work page Pith review arXiv

  2. [5]

    A.3 eICU Model and Training Details: The following details the different model specifications and hyperpa- rameters for training models on the eICU data (Table 7)

    Figure 5: Example lead reconstructions of a limb lead (III) and precordial lead (V2). A.3 eICU Model and Training Details: The following details the different model specifications and hyperpa- rameters for training models on the eICU data (Table 7). Model V ocab Param n_embd n_head n_layer LR Batch Context Steps Discrete n=10 65 25.5M 512 8 8 1 × 10−3 819...

    work page 2024

  3. [7]

    URL https://doi.org/10.1038/s41597-022-01899-x

    doi: 10.1038/s41597-022-01899-x. URL https://doi.org/10.1038/s41597-022-01899-x . A. Karpathy. NanoGPT. https://github.com/karpathy/nanoGPT,

    doi:10.1038/s41597-022-01899-x

  4. [9]

    URL https://openreview.net/forum?id=4VIgNuQ1pY. C. Pang, X. Jiang, N. P. Pavinkurve, K. S. Kalluri, E. L. Minto, J. Patterson, L. Zhang, G. Hripcsak, G. Gürsoy, N. Elhadad, et al. Cehr-gpt: Generating electronic health records with chronological patient timelines. arXiv preprint arXiv:2402.04400,

    arXiv

  5. [11]

    S. N. Shukla and B. M. Marlin. A survey on principles, models and methods for learning from irregularly sampled time series. arXiv preprint arXiv:2012.00168,

    arXiv 2012

  6. [12]

    arXiv:2402.14903. D. Spathis and F. Kawsar. The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models. Journal of the American Medical Informatics Association, 31(9): 2151–2158, Sept

    arXiv

  7. [13]

    doi: 10.1093/jamia/ocae090. A. Stein, S. Sharpe, D. Bergman, S. Kumar, C. B. Bruss, J. Dickerson, T. Goldstein, and M. Goldblum. A simple baseline for predicting events with auto-regressive tabular transformers. arXiv preprint arXiv:2410.10648,

    arXiv

  8. [15]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

Show all 22 references
  1. [17]

    L. Yu, J. Lezama, N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y . Cheng, V . Birodkar, A. Gupta, X. Gu, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737,

    arXiv

  2. [18]

    arXiv:2411.02083. X. N. Zhang, Y . Pu, Y . Kawamura, A. Loza, Y . Bengio, D. Shung, and A. Tong. Trajectory flow matching with applications to clinical time series modelling. In Advances in Neural Information Processing Systems, volume 37, pages 107198–107224,

    arXiv

  3. [19]

    Y . Zhao, Y . Xiong, and P. Krähenbühl. Image and video tokenization with binary spherical quantiza- tion. arXiv preprint arXiv:2406.07548,

    arXiv

  4. [20]

    12 Z. Zhou, J. Wang, D. Lin, and K. Chen. Scaling behavior for large language models regarding numeral systems: An example using pythia. arXiv preprint arXiv:2409.17391,

    arXiv

  5. [21]

    arXiv:2402.01713. A Appendix: Data Set and Experimental Details All code is available in the supplementary material and in the anonymized repository here: https: //anonymous.4open.science/r/multivariateGPT_anon-4ED4/README.md. For all discrete models, bins are evenly spaced qu...

    arXiv

  6. [1997]

    Huang, K

    Z. Huang, K. Srinivas, H. Samulowitz, N. S. D’Souza, C. C. Aggarwal, P.-Y . Chen, and J. Gao. Language models are good tabular learners. Transactions on Machine Learning Research. J. Jia, J. Gao, B. Xue, J. Wang, Q. Cai, Q. Chen, X. Zhao, P. Jiang, and K. Gai. From principles ...

    arXiv

  7. [2017]

    Y . Wang, Z. Lin, Y . Teng, Y . Zhu, S. Ren, J. Feng, and X. Liu. Bridging continuous and discrete tokens for autoregressive visual generation. arXiv preprint arXiv:2503.16430,

    arXiv

  8. [2018]

    doi: 10.1038/s41598-018-24271-9. R. T. Chen, Y . Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31,

    doi

  9. [2019]

    C. W. Schmidt, V . Reddy, H. Zhang, A. Alameddine, O. Uzan, Y . Pinter, and C. Tanner. Tokenization is more than compression. arXiv preprint arXiv:2402.18376,

    arXiv

  10. [2021]

    Mentzer, D

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505,

    arXiv

  11. [2022]

    B. Gow, T. Pollard, L. A. Nathanson, A. Johnson, B. Moody, C. Fernandes, N. Greenbaum, J. W. Waks, P. Eslami, T. Carbonati, A. Chaudhari, E. Herbst, D. Moukheiber, S. Berkowitz, R. Mark, and S. Horng. Mimic-iv-ecg: Diagnostic electrocardiogram matched subset (version 1.0). htt...

    doi

  12. [2023]

    H. Han, J. Xu, M. Zhou, Y . Shao, S. Han, and D. Zhang. Luna: language understanding with number augmentations on transformers via number plugins and pre-training. arXiv preprint arXiv:2212.02691, 2022a. K. Han, Y . Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y . Tang, A. Xiao, C....

    arXiv

  13. [2024]

    org/abs/2310.02989

    URL https://arxiv. org/abs/2310.02989. arXiv preprint arXiv:2310.02989. 10 Y . Gorishniy, I. Rubachev, and A. Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991–25004,

    arXiv

  14. [2025]

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.