Pith. sign in

REVIEW 2 major objections 1 minor 34 references

Hybrid token-based approaches with binning provide the most robust encoding of numeric values for transformers on electronic health records.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Hybrid token-based binning for numeric values in EHR transformers is more robust than explicit interaction modeling, with optimal bin count following an empirically derived power-law in dataset size.

T0 review reviewed 2026-07-03 challenge →

load-bearing objection Hybrid binning encodings come out as the practical default here, but the synthetic arithmetic tasks are a shaky proxy for real clinical numeric demands. the 2 major comments →

arxiv 2607.01391 v1 pith:36GA2OZH submitted 2026-07-01 cs.LG cs.AI

How Should Transformers Encode Numeric Values in Electronic Health Records?

Maria Elkj{\ae}r Montgomery , Christian Igel , Mikkel Odgaard , Martin Sillesen , Mads Nielsen This is my paper
classification cs.LG cs.AI
keywords transformerselectronic health recordsnumeric encodingbinninghybrid tokensclinical predictionarithmetic tasks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper compares discrete, continuous, and hybrid strategies for encoding numbers inside transformer models that process electronic health record sequences. It embeds synthetic arithmetic problems into real patient data and also runs actual clinical prediction tasks to measure trade-offs in precision, training stability, and flexibility. Hybrid methods that keep the original numeric values but bin them before projection emerge as the strongest overall choice. The best bin count follows a power-law relationship with the size of the training dataset. The work concludes that approximate, reliable numeric handling usually matters more for deployment than exact arithmetic.

Core claim

Approaches that explicitly model value-concept interactions perform best on precision-sensitive arithmetic tasks when architectural constraints permit. Hybrid token-based approaches that retain numeric values but apply binning prior to projection provide a more robust and broadly applicable alternative, with the optimal number of bins following a simple empirically derived power-law in dataset size. Across tasks, models consistently exhibit reliable good-enough numeric computation rather than exact arithmetic, while clinical gains from incorporating laboratory values are task-dependent. This suggests that robustness and deployability often outweigh maximal numeric precision in practice, moti

What carries the argument

Hybrid token-based encoding that retains numeric values but applies binning prior to projection, compared against pure discrete and continuous alternatives

Load-bearing premise

Synthetic arithmetic tasks embedded within real-world EHR data sufficiently capture the numeric demands and failure modes of real clinical prediction tasks.

What would settle it

A head-to-head experiment on a large clinical prediction task in which a continuous-value or pure-discrete encoding achieves measurably higher accuracy than the binned hybrid would falsify the robustness claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Explicit value-concept interaction modeling excels on precision-sensitive arithmetic when the architecture allows it.
  • Optimal bin count scales with dataset size according to an empirically derived power law.
  • Transformers on EHR data produce reliable approximate numeric results rather than exact arithmetic.
  • Adding laboratory numeric values improves performance only on some clinical tasks, not all.
  • Robustness and ease of deployment take priority over maximum numeric precision for most practical uses.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed power-law relation supplies a simple rule of thumb for choosing bin counts on new EHR datasets without exhaustive search.
  • If the synthetic arithmetic probes miss certain clinical numeric failure modes, the practical advantage of hybrids may be larger than measured here.
  • Task-dependent value of lab numbers implies selective feature inclusion rather than automatic addition of every numeric field.
  • The same hybrid-binning logic could be tested on sequence models outside transformers or on mixed numeric-categorical data in other domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper investigates optimal strategies for encoding numeric values in transformer models applied to electronic health record (EHR) data. It compares discrete, continuous, and hybrid encoding approaches using synthetic arithmetic tasks embedded in real EHR sequences as well as real-world clinical prediction tasks. Key findings include trade-offs in precision, stability, and flexibility; superior performance of approaches modeling value-concept interactions on precision-sensitive tasks; and that hybrid token-based binning methods are robust, with optimal bin count following a power-law relationship with dataset size. Models generally achieve 'good enough' rather than exact arithmetic, and benefits of lab values are task-dependent.

Significance. If the empirical results hold, the work provides actionable guidance for encoding numerics in EHR transformers, highlighting hybrid binning as a practical default due to robustness. The empirically derived power-law for bin count could aid in model design across dataset scales. The observation of 'good enough' computation rather than exact arithmetic has implications for expectations in clinical ML. The use of both synthetic and real tasks is a strength, but the significance is limited by the absence of reported experimental details needed to verify the central comparisons.

major comments (2)
  1. [Abstract] Abstract: the claims of hybrid token-based binning superiority and the power-law relationship for optimal bin count rest on empirical comparisons, yet no details are provided on experimental controls, error bars, data splits, statistical testing, or whether the power-law fit was pre-specified versus post-hoc. This directly affects the ability to assess whether the reported ranking of encodings is load-bearing or reproducible.
  2. [Abstract] Abstract: the assertion that hybrid approaches are 'more robust and broadly applicable' is derived from synthetic arithmetic tasks embedded in EHR data, but the abstract notes only 'good enough' (not exact) arithmetic on both synthetic and real tasks. It is unclear whether the encoding ranking and power-law would persist if the primary evaluation were on real clinical tasks (e.g., mortality or readmission) whose numeric precision requirements may be lower than the synthetic arithmetic regime.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'when architectural constraints permit' for value-concept interaction approaches is vague without elaboration on what those constraints are or how they were varied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below, committing to revisions that improve experimental transparency and clarify the scope of our claims without overstating the current results.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claims of hybrid token-based binning superiority and the power-law relationship for optimal bin count rest on empirical comparisons, yet no details are provided on experimental controls, error bars, data splits, statistical testing, or whether the power-law fit was pre-specified versus post-hoc. This directly affects the ability to assess whether the reported ranking of encodings is load-bearing or reproducible.

    Authors: We agree that the abstract and current manuscript lack sufficient detail on these elements, limiting independent verification. In the revision we will expand the Methods section with full descriptions of data splits, random seeds, error-bar computation, and the statistical tests applied to encoding comparisons. We will also state explicitly that the power-law relationship was identified post-hoc via inspection of optimal bin counts across dataset sizes. A concise summary of multi-run evaluation and error reporting will be added to the abstract. revision: yes

  2. Referee: [Abstract] Abstract: the assertion that hybrid approaches are 'more robust and broadly applicable' is derived from synthetic arithmetic tasks embedded in EHR data, but the abstract notes only 'good enough' (not exact) arithmetic on both synthetic and real tasks. It is unclear whether the encoding ranking and power-law would persist if the primary evaluation were on real clinical tasks (e.g., mortality or readmission) whose numeric precision requirements may be lower than the synthetic arithmetic regime.

    Authors: The manuscript already evaluates both synthetic and real clinical tasks and reports that hybrid binning remains stable on the latter, albeit with smaller absolute gains. However, the referee is correct that the abstract does not sufficiently distinguish how the ranking and power-law scale when precision demands are lower. We will revise the abstract and add a dedicated paragraph in the discussion to compare encoding performance on the real tasks alone and to note that the power-law observation is primarily driven by dataset-size variation rather than task type. Additional ablation tables restricted to real tasks will be included. revision: yes

Circularity Check

0 steps flagged

No circularity; claims rest on external empirical benchmarks

full rationale

The paper's central claims derive from systematic experimental comparisons of discrete, continuous, and hybrid encoding strategies evaluated on synthetic arithmetic tasks embedded in EHR sequences and on real clinical prediction tasks. The reported power-law relating optimal bin count to dataset size is obtained by fitting observed performance curves across multiple dataset sizes; it is not obtained by defining the quantity in terms of itself or by relabeling a fitted parameter as a prediction. No equations, uniqueness theorems, or ansatzes are introduced via self-citation, and no load-bearing premise reduces to a prior result authored by the same team. All performance rankings and robustness conclusions are therefore falsifiable against the held-out task metrics rather than being tautological with the input construction.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

Based solely on abstract; full methods and results unavailable. The power-law for optimal bins is described as empirically derived, implying at least one fitted relationship. No explicit axioms or invented entities stated.

free parameters (1)
  • optimal bin count
    Described as following an empirically derived power-law in dataset size, indicating a fitted functional form.

reviewed 2026-07-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of How Should Transformers Encode Numeric Values in Electronic Health Records?." pith.science (2026). https://pith.science/paper/36GA2OZH

@misc{pith2026260701391,
  author       = {Pith},
  title        = {Pith review of: How Should Transformers Encode Numeric Values in Electronic Health Records?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36GA2OZH}},
  note         = {Machine review of arXiv:2607.01391}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

How do we encode numeric values in transformer-based sequence processing, particularly in electronic health record (EHR) data? We systematically compare discrete, continuous, and hybrid value encoding strategies using synthetic arithmetic tasks embedded within real-world EHR data, as well as real-world clinical prediction tasks. Our study reveals trade-offs between numeric precision, optimisation stability, and architectural flexibility. We find that approaches that explicitly model value-concept interactions perform best on precision-sensitive arithmetic tasks when architectural constraints permit. Hybrid token-based approaches that retain numeric values but apply binning prior to projection provide a more robust and broadly applicable alternative, with the optimal number of bins following a simple empirically derived power-law in dataset size. Across tasks, models consistently exhibit reliable "good enough" numeric computation rather than exact arithmetic, while clinical gains from incorporating laboratory values are task-dependent. This suggests that robustness and deployability often outweigh maximal numeric precision in practice, motivating hybrid token-based approaches as a practical default.

Figures

Figures reproduced from arXiv: 2607.01391 by Christian Igel, Mads Nielsen, Maria Elkj{\ae}r Montgomery, Martin Sillesen, Mikkel Odgaard.

Figure 1
Figure 1. Figure 1: Illustration of a patient trajectory and the corresponding embedding layers forming the final model input. Here, E(x) denotes the embedding function for concepts, and F(x) denotes a function that maps raw timestamps to absolute positional encodings [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Numeric-integration strategies in EHR transformers following the example from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Power-law fit for the optimal bin count versus the number of unique values in the lab value distribution. Error bars indicate SEM. 4.2. Synthetic Classification Experiment [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows results for all five methods under the syn￾thetic classification setup with mean separation (µ0, µ1) = (48, 52). This setup highlights where the methods differ the most and is the most comparable to real clinical prediction scenarios as it reflects a relatively weak underlying signal. Results across all evaluated distributions are provided in Ap￾pendix E. From these results, we selected FiLM, combine… view at source ↗
Figure 6
Figure 6. Figure 6: Performance across arithmetic tasks measured in infoeff (Figures 6a–6e) and clinical prediction tasks (Figure 6f), showing AUROC change relative to the lab-name baseline. Error bars indicate standard deviation (SD) across runs. For the clinical tasks, stars indicate statistical significance relative to the lab-name baseline AUROC values (* p < 0.05, ** p < 0.01, *** p < 0.001; no star = not significant). E… view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity of performance to MSE loss weighting. The shaded band denotes the mean ± SD for w = 1, while error bars show SD for the remaining weightings. Black-edged markers indicate means within one SD of the w = 1 setting. B. Computation of theoretical AUC [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: shows the theoretical AUROC values for the baseline setting with n = 100, 000. In this setting, the synthetic lab measurement is the only feature that carries information about the assigned label, which allows us to compute the theoretical maximum achievable AUROC. These theoretical values therefore represent the upper bound on performance for any model in this setup. (a) (35, 65) (b) (45, 55) (c) (48, 52)… view at source ↗
Figure 9
Figure 9. Figure 9: Synthetic classification performance (dσ) for all five methods on all distributions. F. Results from addition tasks Results from the addition experiments are shown in Figure 10a and Figure 10b. Figure 10a reports performance as the number of labs increases, while Figure 10b shows performance under increasing noise levels. (a) Increasing number of labs. (b) Increasing noise level [PITH_FULL_IMAGE:figures/f… view at source ↗
Figure 10
Figure 10. Figure 10: Addition task performance across varying task difficulty. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Spider plot of selected arithmetic tasks including the combination method. H. FiLM evaluation under matched token budgets To evaluate whether FiLM performance was influenced by access to longer patient histories under the fixed 1024-token context window, we repeat the clinical experiments using a truncated FiLM setting with an effective token budget matched to the discretisation-based approaches [PITH_FU… view at source ↗
Figure 12
Figure 12. Figure 12: Clinical prediction performance for the original and truncated FiLM models, shown as AUROC change relative to the lab-name baseline [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Distribution of patient sequence lengths prior to the 2022 clinical-task cutoff [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Relative ranking across selected arithmetic tasks for all evaluated methods [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Clinical prediction performance for all evaluated methods, shown as AUROC change relative to the lab-name baseline. Clinical results for all methods are shown in [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages

  1. [1]

    Digital Health , volume=

    Ensemble machine learning methods in screening electronic health records: A scoping review , author=. Digital Health , volume=. 2023 , publisher=

    work page 2023

  2. [2]

    The American Journal of Gastroenterology , volume=

    Machine learning models for pancreatic cancer risk prediction using electronic health record data—a systematic review and assessment , author=. The American Journal of Gastroenterology , volume=. 2024 , OPTpublisher=

    work page 2024

  3. [3]

    JCO Clinical Cancer Informatics , volume=

    Evolution of breast cancer recurrence risk prediction: a systematic review of statistical and machine learning--based models , author=. JCO Clinical Cancer Informatics , volume=. 2023 , publisher=

    work page 2023

  4. [4]

    Critical Care , volume=

    Use of machine learning to analyse routinely collected intensive care unit data: a systematic review , author=. Critical Care , volume=. 2019 , publisher=

    work page 2019

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Attention is all you need , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

    work page

  6. [6]

    Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , pages=

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , pages=

    work page

  7. [7]

    2021 , OPTpublisher=

    Rasmy, Laila and Xiang, Yang and Xie, Ziqian and Tao, Cui and Zhi, Degui , journal=. 2021 , OPTpublisher=

    work page 2021

  8. [8]

    Scientific Reports , volume=

    Li, Yikuan and Rao, Shishir and Solares, Jos. Scientific Reports , volume=. 2020 , OPTpublisher=

    work page 2020

  9. [9]

    2021 , organization=

    Pang, Chao and Jiang, Xinzhuo and Kalluri, Krishna S and Spotnitz, Matthew and Chen, RuiJun and Perotte, Adler and Natarajan, Karthik , booktitle=. 2021 , organization=

    work page 2021

  10. [10]

    Context clues: Evaluating long context models for clinical prediction tasks on

    Wornow, Michael and Bedi, Suhana and Hernandez, Miguel Angel Fuentes and Steinberg, Ethan and Fries, Jason Alan and R. Context clues: Evaluating long context models for clinical prediction tasks on. International Conference on Learning Representations (ICLR) , year=

    work page

  11. [11]

    Machine Learning for Healthcare Conference , year=

    Odgaard, Mikkel and Klein, Kiril Vadimovic and Thysen, Sanne M. Machine Learning for Healthcare Conference , year=

    work page

  12. [12]

    CEHR-XGPT: A scalable multi-task foundation model for electronic health records.arXiv, 2509.03643, 2025

    CEHR-GPT: A Scalable Multi-Task Foundation Model for Electronic Health Records , author=. arXiv preprint arXiv:2509.03643 , year=

    work page arXiv

  13. [13]

    Ethan Steinberg and Jason Alan Fries and Yizhe Xu and Nigam Shah , booktitle=

    work page

  14. [14]

    The Lancet Digital Health , volume=

    Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling study , author=. The Lancet Digital Health , volume=. 2024 , publisher=

    work page 2024

  15. [15]

    IEEE Transactions on Industrial Electronics , volume=

    Machine health monitoring using local feature-based gated recurrent unit networks , author=. IEEE Transactions on Industrial Electronics , volume=. 2017 , publisher=

    work page 2017

  16. [16]

    arXiv preprint arXiv:1907.09600 , year=

    Evaluation of embeddings of laboratory test codes for patients at a cancer center , author=. arXiv preprint arXiv:1907.09600 , year=

    work page arXiv 1907

  17. [17]

    npj Digital Medicine , year =

    Zero shot health trajectory prediction using transformer , author =. npj Digital Medicine , year =

    work page

  18. [18]

    Multimodal

    Mbaye, Nd. Multimodal. Frontiers in Oncology , volume =

    work page

  19. [19]

    Machine Learning for Health Symposium , publisher=

    Labrador: Exploring the limits of masked language modeling for laboratory data , author=. Machine Learning for Health Symposium , publisher=

    work page

  20. [20]

    Tokenization Tradeoffs in Structured

    Guo, Lin Lawrence and Arciniegas, Santiago Eduardo and Lee, Joseph Jihyung and Yan, Adam Paul and Tomlinson, George and Fries, Jason and Sung, Lillian , journal=. Tokenization Tradeoffs in Structured

    work page

  21. [21]

    Communications Medicine , volume=

    A self-supervised framework for laboratory data imputation in electronic health records , author=. Communications Medicine , volume=. 2025 , publisher=

    work page 2025

  22. [22]

    NeurIPS 2023 AI for Science Workshop , year=

    xVal: A Continuous Number Encoding for Large Language Models , author=. NeurIPS 2023 AI for Science Workshop , year=

    work page 2023

  23. [23]

    Huang, Xin and Khetan, Ashish and Cvitkovic, Milan and Karnin, Zohar , journal=

    work page

  24. [24]

    Advances in Neural Information Processing Systems (NeurIPS) , volume=

    Revisiting deep learning models for tabular data , author=. Advances in Neural Information Processing Systems (NeurIPS) , volume=

    work page

  25. [25]

    Somepalli, Gowthami and Goldblum, Micah and Schwarzschild, Avi and Bruss, C Bayan and Goldstein, Tom , year=

    work page

  26. [26]

    2025 , booktitle=

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference , author=. 2025 , booktitle=

    work page 2025

  27. [27]

    AAAI Conference on Artificial Intelligence (AAAI) , volume=

    Film: Visual reasoning with a general conditioning layer , author=. AAAI Conference on Artificial Intelligence (AAAI) , volume=

    work page

  28. [28]

    , title =

    Knuth, Donald E. , title =. 1997 , isbn =

    work page 1997

  29. [29]

    International Conference on Learning Representations (ICLR) , year=

    Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count , author=. International Conference on Learning Representations (ICLR) , year=

    work page

  30. [30]

    arXiv preprint arXiv:2102.13019 , year=

    Investigating the limitations of transformers with simple arithmetic tasks , author=. arXiv preprint arXiv:2102.13019 , year=

    work page arXiv

  31. [31]

    The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

    Transformers Can Do Arithmetic with the Right Embeddings , author=. The 4th Workshop on Mathematical Reasoning and AI at NeurIPS'24 , year=

    work page

  32. [32]

    GitHub repository , howpublished =

    yandexdataschool , title =. GitHub repository , howpublished =. 2021 , publisher =

    work page 2021

  33. [33]

    Journal of the American Statistical Association , volume=

    Adaptive huber regression , author=. Journal of the American Statistical Association , volume=. 2020 , publisher=

    work page 2020

  34. [34]

    and Lin, Yancong and Dijkstra, Jouke and van Gemert, Jan C

    Pintea, Silvia L. and Lin, Yancong and Dijkstra, Jouke and van Gemert, Jan C. , booktitle=. A step towards understanding why classification helps regression , year=

    work page

This paper was first reviewed by grok-4.3 on July 3, 2026.