{"id":"c7255cde-95a9-45d2-a2c9-219dfbca71b8","arxiv_id":"2505.21680","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"multivariateGPT extends next-token prediction to jointly predict the class and continuous value of mixed categorical and numeric time series, with Gaussian uncertainty, and outperforms discrete-token baselines on clinical and physics data.","lead":"A decoder-only transformer is adapted to handle sequences that mix categorical and numeric measurements, including time gaps between records, within one model. It predicts the next measurement's type and value jointly and reports strong results on ICU vitals, electrocardiograms, and simple physics trajectories.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gaussian value head for time intervals is unvalidated; if miscalibrated, the informative-sampling claim lacks support.","rationale":"After reading the full manuscript, the central claim is that a single decoder-only transformer with a joint class/value likelihood can model mixed categorical/numeric time series, represent numeric values continuously, provide calibrated uncertainty, and exploit informative irregular sampling. The most load-bearing assumption is the per-class Gaussian value head, because it is a parametric commitment about every numeric variable, including the timing variable tau that underpins the informative-sampling contribution. I find the reader's alternative concern about the arbitrary within-timestamp ordering less compelling: the autoregressive decomposition in Eq. 2 is a valid chain-rule factorization for any fixed ordering; lexicographic sorting does not lose information, it simply fixes a factorization, and the model can represent the joint distribution over co-occurring measurements through conditioning. The reader is right to mention the Gaussian, but it deserves to be the primary concern rather than a secondary note. The missing continuous-embedding baselines (XVal, MMD) weaken the 'state of the art' framing but do not threaten the internal correctness of the method; they are an evaluation-completeness issue. The Gaussian issue, by contrast, directly affects whether the model provides calibrated uncertainty for a key novel capability (timing prediction). The paper's own Limitations section acknowledges the Gaussian may not capture time data, and no calibration evidence is provided for timing. A concrete check on predictive coverage and residual normality for tau would settle whether the timing predictions are actually well-calibrated. If they are, the concern is resolved; if not, the informative-sampling claim should be softened. This supports a CONDITIONAL verdict rather than ACCEPT or REJECT.","tokens_in":13561,"tokens_out":15005,"duration_ms":156457,"concrete_test":"On the eICU test set used in Table 2, for each seed length (3/6/9/12h), compute the empirical coverage of the model's predicted 95% predictive intervals for the time-delta tokens tau, and the standardized residuals (tau_true - mu_hat)/sigma_hat. Compare coverage to 0.95 and test residuals for normality (e.g., skewness/kurtosis or Shapiro-Wilk). If coverage deviates substantially (e.g., >2 percentage points) or residuals show systematic skewness, the Gaussian timing assumption is violated, and the informative-sampling calibration claim is not supported for timing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 (Eq. 7) parameterizes every numeric value, including the elapsed-time token tau, with a conditional Gaussian. The paper's central claim includes calibrated uncertainty and informative irregular sampling, but calibration is demonstrated only for MAP and heart rate (Fig. 3, Table 3) in one dataset. Timing predictions (Table 2) are evaluated only by MSE, which can be low even when the predictive distribution is misspecified. Time intervals in EHR data are strictly positive and typically right-skewed; a Gaussian assigns mass to negative values and cannot represent heavy tails or point masses (e.g., regularly scheduled measurements). The authors themselves list 'time, count, or ordinal data' as poorly captured by the Gaussian (Limitations). Because the model's ability to 'predict observation timing' is a headline contribution, the absence of calibration or residual diagnostics for tau is a load-bearing gap: if the Gaussian is miscalibrated for timing, the model does not provide the claimed calibrated uncertainty for informative sampling, and the low MSE may reflect only accurate conditional means, not a valid likelihood.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes multivariateGPT, a decoder-only transformer that models sequences of mixed categorical and numeric data using an autoregressive decomposition of the joint distribution over class and value. Each timestamp is flattened into a sequence of class-value tuples, including elapsed time as a special class, and the model is trained with a joint negative log-likelihood that combines a softmax class head with a per-class Gaussian value head. The authors claim this extends next-token prediction to likelihood-based estimation of continuous values with uncertainty, supports informative irregular sampling by predicting observation timing, and outperforms discrete-token transformers and TFM-ODE on clinical time-series tasks. Experiments cover damped harmonic oscillators, eICU sepsis data, MIMIC-IV ECG lead reconstruction, and Physionet ICU data, with comparisons to discrete-token transformers, a frozen LLM, and TFM-ODE. The paper includes an anonymous code repository and detailed training configuration tables.","tokens_in":13655,"tokens_out":4055,"duration_ms":43715,"significance":"The central formulation is mathematically clean and the loss decomposition in Eqs. (1)-(8) is sound; if the empirical claims hold, the method offers a simple, unified way to bring continuous values and uncertainty into autoregressive transformers without discrete tokenization. The authors deserve credit for releasing code, reporting calibration on MAP and heart rate, and demonstrating a clear generalization failure of discrete tokenizers on the oscillator task. However, the significance is currently tempered by two gaps: the arbitrary ordering inside timestamps is never stress-tested, and the Gaussian likelihood for the time interval is not validated beyond mean-squared error, despite timing prediction being a headline contribution. The comparison set also omits the continuous-numeric tokenization baselines discussed in the paper itself. These issues are addressable within a revision, so the work has solid potential but is not yet fully supported.","major_comments":[{"comment":"The flattening in Section 2.1 imposes a lexicographic ordering over co-occurring measurements inside a timestamp when no natural order exists, and Eq. (2) conditions each measurement on that ordering. The paper provides no sensitivity analysis over the ordering (e.g., random permutations or different sort keys) on any data set, so the claim in Section 3.2.3 that the model captures the joint distribution over which measurements occur is not yet supported; if the ordering affects predictive performance, the decomposition is not a faithful factorization of the timestamp-level joint distribution.","section":"Section 2.1, Eqs. (2)-(3)"},{"comment":"The elapsed-time token tau is modeled with a conditional Gaussian in Eq. (7), but the paper evaluates timing predictions only by MSE (Table 2) and reports calibration only for MAP and heart rate (Fig. 3, Table 3). Time intervals in EHR data are strictly positive and right-skewed, with likely point masses at scheduled measurement times, so a Gaussian likelihood is questionable; the authors themselves acknowledge this in the Limitations section. Because predicting observation timing is a headline contribution (Section 5), the paper needs to provide calibration or residual diagnostics for tau (e.g., coverage of predictive intervals or PIT histograms) to substantiate the claim of calibrated uncertainty for informative sampling.","section":"Section 2.3, Eq. (7); Table 2; Fig. 3"},{"comment":"The empirical comparisons are limited to discrete-token transformers, TFM-ODE, and a frozen LLM; despite the Related Work discussion of continuous numeric embeddings such as XVal and MMD and regression-like numeric losses, no such model is included as a baseline. As a result, Table 1's improvement over discrete tokenization does not isolate the benefit of the proposed continuous value head and embedding; a continuous-numeric baseline on eICU or Physionet is needed to support the claim that the proposed representation, rather than the transformer backbone alone, drives the reported gains.","section":"Section 3.2.1, Table 1; Section 4"}],"minor_comments":[{"comment":"The algorithm title contains a typo: \"Mutivariate\" should be \"Multivariate\".","section":"Algorithm 1"},{"comment":"The sentence \"The Physionet ICU data set contains measurements collected in a sparse manner across a 36 categorical and classes\" is malformed; it should read \"36 categorical and numeric classes\" or similar.","section":"Section 3.2.3"},{"comment":"There is a sentence fragment immediately after Table 4: \"the discrete model erroneously predicting tokens representing extreme values.\" This should be integrated into a complete sentence or moved to the main text.","section":"Table 4"},{"comment":"The description of v_j as \"a vector with one non-zero element equal to the correct value at the index of the correct class\" is confusing; v_j is the scalar value of the correct class, not a vector of length C with a single non-zero entry (that is the role of c_j). Please clarify the notation.","section":"Section 2.3, Eq. (7)"},{"comment":"The word \"autoregressivelly\" is misspelled; it should be \"autoregressively.\"","section":"Section 3.2.2"},{"comment":"The phrase \"retrogressively modeling any database\" in the conclusion is unclear; likely \"retrospectively\" or \"generatively\" was intended. Please correct the wording.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and presents a genuinely simple and usable architecture. The main risk is not circularity but incomplete empirical support for the timing/informative-sampling claim and the arbitrary ordering inside timestamps. These are addressable with additional experiments, so I would not reject, but the revision needs to be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: treat each measurement as a (class, value) token, flatten multivariate timestamps with an explicit time token, and train a decoder-only transformer with a joint class head and per-class Gaussian value head. The chain-rule decomposition is mathematically sound, and the implementation is straightforward. On eICU and ECG, the model clearly beats discrete-token baselines and TFM-ODE, and the oscillator experiments show a genuine generalization benefit over discrete tokenization. That is a real contribution, even if each component is known.\n\nThe soft spots are proportionate. First, the within-timestamp ordering is arbitrary (lexicographic when no natural order exists), and there is no sensitivity analysis showing the result is robust to that choice. If co-occurring measurements have correlations that depend on order, the flattening could lose information. Second, and more important, the headline claim about calibrated uncertainty and informative sampling is only validated for heart rate and MAP. Timing predictions are evaluated only by MSE, which can be low even when the predictive distribution is misspecified. Time intervals are strictly positive and right-skewed, so a Gaussian is a questionable choice; the authors themselves flag this in the Limitations section. But they still imply that the model provides calibrated uncertainty for timing, and that is unsupported by the current evidence. Third, the \"state of the art\" wording in the conclusion outruns the baseline set: no XVal, MMD, or regression-loss token models are compared, so we do not know whether the gains are specific to the joint class-value head or just to continuous numeric handling. These are addressable with additional experiments, not fatal flaws.\n\nI would send this to a serious referee. The architecture is sensible, the empirical gains are consistent, and the limitations are honestly acknowledged. The referee should ask for ordering sensitivity tests, calibration or residual diagnostics for the time intervals, and at least one continuous-embedding baseline. If those come back clean, this could be a useful reference for anyone building generative models on mixed-type clinical data. I would not cite it yet in my own work until the timing calibration is sorted, but I would bring it to the reading group and tell a student working on EHR generation to read it.","headline":"A clean extension of transformer LMs to mixed-type sequences with real gains on EHR/ECG, but the timing-calibration claim needs support and the baseline set is too narrow for 'state of the art.'","tokens_in":14247,"tokens_out":1696,"would_cite":false,"duration_ms":20025,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder-only transformer with a class head and per-class Gaussian value head models mixed categorical and numeric sequences by maximizing the joint log-likelihood of the next token's class and value.","keywords":["transformer","multivariate time series","mixed categorical and numeric data","informative sampling","continuous value prediction","irregular time series","electronic health records","likelihood-based loss"],"falsifier":"Retrain the model on the eICU data with a random permutation of the measurement order within each timestamp, keeping the loss unchanged; if class accuracy or timing MSE changes materially, the imposed ordering is carrying information that the flattening assumption treats as irrelevant.","tokens_in":13318,"feed_emoji":"📈","tokens_out":8182,"duration_ms":79820,"temperature":0.7,"pith_summary":"multivariateGPT extends the next-token prediction task of a decoder-only transformer from predicting a token to estimating the joint distribution of the next token's class and its continuous value. The paper shows that by flattening a multivariate time series into ordered (class, value) tuples, including the elapsed time as its own class, a single architecture can model categorical data, numbers, and the timing of observations. On clinical time series from eICU it reports 40–60% lower error than trajectory flow matching for heart rate and blood pressure prediction, and it is the only model in the comparison that predicts both the value and the time of the next observation. On synthetic damped oscillators it generalizes to trajectories not present in training, while discrete-token transformers with 10 or 100 bins do not.","feed_headline":"One transformer predicts numbers, categories, and timing together","feed_subtitle":"A class head plus a per-class Gaussian value head predicts continuous numbers and calibrated uncertainty without binning.","key_machinery":"The load-bearing mechanism is the class-value token decomposition with a twin-head output. Each token is a pair (c_i, v_i); categorical classes are expanded so each class has a single value, while numeric tokens embed as E_{c_i} + f_{c_i}(v_i) for a learnable class embedding and value map. The model's final layer feeds two heads: a class head that softmaxes over class identities and a value head that outputs the mean and softplus-standard-deviation of a normal distribution for each class. Training minimizes the sum of the cross-entropy over classes and the negative log-likelihood of the true value under the predicted Gaussian. This turns next-token prediction into a joint likelihood over class and value, gives continuous numeric representation with no binning, and produces per-observation variance estimates that the paper shows to be well calibrated.","core_discovery":"The paper's central claim is that a single autoregressive transformer can represent numeric values continuously and provide calibrated uncertainty by decomposing the joint distribution of the next token into P(c_i, v_i) = P(c_i) P(v_i | c_i), with a standard softmax class head and a per-class Gaussian value head. The loss is the sum of the class negative log-likelihood and the conditional value negative log-likelihood. Under the flattening of each timestamp into an ordered sequence of (class, value) pairs that includes the elapsed time as its own class-value pair, the model estimates the joint distribution over which measurements occur, what values they take, and when they occur. The paper reports that this approach outperforms discrete-token transformers and TFM-ODE on eICU sepsis data, reconstructs 12-lead electrocardiograms with smaller error than discrete baselines, and predicts class and value on Physionet ICU data with the same class accuracy but far lower value error.","pith_inferences":["Inference: the paper reports that arbitrary lexicographic order is used when no natural order exists; if a permutation-invariant set representation matched or beat the flattened-sequence model on the same data, the imposed ordering would not be information-preserving.","Inference: the conditional-Gaussian head, which the authors flag as a limitation, will miss multi-modal or bounded distributions; a mixture-of-Gaussians or per-class distribution family is a direct extension.","Inference: because timing is a token in the same sequence, the model can be sampled autoregressively to produce full future trajectories, not just point forecasts; the paper mentions but does not evaluate this Monte Carlo rollout use.","Inference: a targeted test of the informative-sampling claim would perturb a recorded value in context and check whether the model's predicted time-to-next-observation distribution shifts accordingly."],"forward_implications":["Numeric predictions no longer depend on bin resolution; a model with a small vocabulary can represent values continuously and interpolate between training values.","The same architecture can predict the elapsed time until the next observation, because time is just another class-value tuple, which no neural-ODE baseline in the comparison can do.","Per-token Gaussian variance estimates give calibrated predictive intervals, with eICU coverage fractions near 0.95 for heart rate and MAP.","The method is compatible with a pure language modeling objective: a document or a text field can be treated as a sequence of class-value tokens within the same loss.","Vocabularies shrink because no discretization into bins is needed, reducing the number of classes the model must learn."],"supporting_citations":[{"why":"Supplies the transformer architecture that the model is built on.","marker":"[Vaswani et al., 2017]"},{"why":"Provides the decoder-only implementation the model follows.","marker":"[Karpathy, 2022]"},{"why":"Defines the eICU sepsis task and the TFM-ODE baseline that multivariateGPT is compared against.","marker":"[Zhang et al., 2024]"},{"why":"Introduces the neural-ODE paradigm whose continuous representation the paper contrasts with its token-based approach.","marker":"[Chen et al., 2018]"},{"why":"Is the source of the eICU sepsis data used for heart rate and blood pressure prediction.","marker":"[Pollard et al., 2018]"},{"why":"Provides the MIMIC-IV ECG records used for lead reconstruction.","marker":"[Gow et al., 2023]"},{"why":"Provides the Physionet ICU data with 36 measurement classes used for class and value prediction.","marker":"[Moor et al., 2021]"},{"why":"Supplies evidence that observation timing carries information, the premise behind modeling time as a token.","marker":"[Getzen et al., 2023]"}],"fun_headline_variants":["Transformer predicts class and value with per-class Gaussian head","Single architecture for mixed categorical and numeric sequences","Unified model extends next-token prediction to continuous values","Autoregressive head handles irregular timing and calibrated uncertainty","One model learns what, when, and how much from mixed data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that flattening each timestamp into a fixed measurement order plus a single elapsed-time token preserves all information needed for informative sampling and for the joint distribution over which measurements occur; if that flattening discards information, the central claim about capturing informative irregular timing collapses.","fun_headline_variants_meta":{"raw":{"variants":["Transformer predicts class and value with per-class Gaussian head","Single architecture for mixed categorical and numeric sequences","Unified model extends next-token prediction to continuous values","Autoregressive head handles irregular timing and calibrated uncertainty","One model learns what, when, and how much from mixed data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1350,"prompt_tokens":880,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":496,"tokens_out":470,"duration_ms":5654,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:25:22.207222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the model on the eICU data with a random permutation of the measurement order within each timestamp, keeping the loss unchanged; if class accuracy or timing MSE changes materially, the imposed ordering is carrying information that the flattening assumption treats as irrelevant.","supporting_citations":[],"review_version":1}