ORIGAMI: A generative transformer architecture for predictions from semi-structured data

Alana Huang; Robin Vujanic; Thomas R\"uckstie\ss

REVIEW 2 major objections 7 minor 51 references

ORIGAMI: A generative transformer architecture for predictions from semi-structured data

T0 review · 2 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A generative transformer that consumes JSON directly as key/value token sequences with stack-based position encodings matches or beats tabular and specialized baselines, and reformulates classification as next-token prediction.

desk verdict Well-engineered transformer for JSON with a solid ablation story, but the CodeNet comparison needs a split statement before the headline claim is supported. read the letter →

arxiv 2412.17348 v1 pith:P3ALIRHX submitted 2024-12-23 cs.LG

Thomas Rückstiess , Alana Huang , Robin Vujanic This is my paper · ORCID

classification cs.LG MSC 68T0768Q45

keywords semi-structureddataJSONgenerativetransformerkey/valuepositionencodingpushdownautomatonconstraineddecodingnext-tokenpredictionmulti-labelclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ORIGAMI is a decoder-only transformer that learns directly from JSON objects by turning each object into a token sequence and training the model to predict the next token; classification becomes a generation problem, so single-label and multi-label predictions need no architectural change. The paper's claim is that this works only if the architecture respects the data's structure, which it does through three components: a reversible tokenizer that keeps keys and values atomic, a position encoding computed from the stack of a pushdown automaton that parses the token stream, and grammar-based masking that forbids invalid tokens during both training and inference. On tabular benchmarks converted to JSON, ORIGAMI matches gradient-boosted trees on average; on the DDXPlus medical-diagnosis dataset it beats multi-output baselines; and on CodeNet Java250 code classification it reports 94.7% accuracy, above an MLP, CNN, and the best GNN, though below a much larger pre-trained CodeBERT. A synthetic 'Dungeons' task, where the correct label requires looking up a key path in a shuffled array, is solved to 100% test accuracy by ORIGAMI while flattened baselines stay near chance. The upshot is that flattening semi-structured data may be unnecessary: a structural inductive bias plus grammar constraints can let a small generative model outperform specialized architectures.

What carries the argument

The load-bearing mechanism is the key/value position encoding (KVPE). For each input token, a deterministic pushdown automaton -- the same automaton that recognizes the language of valid JSON token sequences -- records its stack state, and KVPE encodes that state as the sum of the embeddings of the stack symbols; the sum of the root marker, the current key, and the current array position represents the token's full key path independently of where it appears in the linear sequence. This is what makes sibling order irrelevant, enables sampling random permutations of key/value pairs during training, and lets the model answer queries about any key at inference time. The second mechanism is grammar-constrained decoding: the same automaton's transition masks set invalid next-token logits to $-\infty$, so the model never wastes capacity on learning grammar and convergence on the Dungeons task is reached in 303 training steps on average versus 495 without guardrails.

What would settle it

Re-run CodeNet Java250 using the exact official train/test split and preprocessing of the published baselines, applying the same UNKNOWN relabeling of discarded long instances to every method; if ORIGAMI's accuracy no longer exceeds the best GNN, the headline advantage is an evaluation artifact. In a second check, enumerate all key paths of a dataset and test whether any two distinct pushdown-automaton stack states produce identical KVPE embedding sums; a collision would falsify the paper's 'unique' encoding claim.

Watch

Extended reading notes

Core claim

Stated on its own terms, the paper's discovery is a recipe for end-to-end supervised learning on JSON: represent every object as a depth-first token sequence of key, value, and structural tokens; add to each position a key/value position embedding formed by summing the embeddings of the pushdown-automaton stack symbols that are active while parsing that token; and use the automaton to mask out next tokens that would violate JSON grammar, during training as well as inference. Because the position encoding depends only on the stack, not on absolute token order, sibling key/value pairs can be shuffled without changing the model's input representation, which the paper exploits as a regularization and data-upscaling technique. The reformulation of classification as next-token prediction then lets a single model output a single label, an array of labels, or even a nested object, and the paper's experiments claim this beats the json2vec baselines on 7 of 8 JSON-ified benchmark datasets, outperforms multi-label baselines on DDXPlus, and outperforms an MLP, CNN, and GNN on the CodeNet Java250 classification task while trailing only the far larger pre-trained CodeBERT.

Load-bearing premise

The headline comparison against convolutional and graph networks depends on baseline accuracy numbers taken from the CodeNet paper that may come from a different test split, and only ORIGAMI's dropped long examples are counted as errors, so the lead over the best graph network could come from evaluation mismatch.

Editorial extensions

If this is right

If ORIGAMI's results hold, nothing about a prediction task requires flattening semi-structured data first: deeply nested objects such as ASTs can be fed to a transformer in native form, avoiding feature matrices with over a million columns.
Multi-label and variable-size outputs become free: an $\text{Array}(n)$ token tells the model to keep sampling labels, so one architecture handles single-label, multi-label, and structured outputs.
Order-invariant position encoding plus key permutation upscaling is a data-efficient regularizer: on the small contraceptive dataset mean accuracy rises from 50.85% to 54.0%, and the gains are largest where data is scarce.
Grammar masking during training accelerates convergence by about 39% on the synthetic task, suggesting the same trick will speed up any structured-output model where outputs must obey a formal grammar.
Because predictions for any key of an object are available from one trained model, the same framework can auto-complete partial documents or predict missing fields, applications the paper explicitly flags as next steps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the summed-stack-state encoding is not shown to be collision-free: two different key paths could in principle yield the same KVPE vector, which would break the 'unique position embedding' claim; counting collisions on real datasets would settle this.
Beyond the paper, ORIGAMI's recipe is portable: KVPE could be attached to any transformer decoder as a drop-in positional encoding for structured tokens, potentially improving models that currently flatten JSON before embedding.
Beyond the paper, the CodeNet gap to CodeBERT (94.7% vs 97.4%) is attributed by the authors to scale and pre-training, which implies that a pre-trained or larger ORIGAMI-style model -- or one fine-tuned from a language model checkpoint -- could close that gap while retaining native JSON structure.
Beyond the paper, the same generative setup could support unsupervised tasks the paper only names, such as cardinality estimation for document databases or synthetic JSON generation, since the model already approximates the full joint distribution over object tokens.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know two things about this paper. First, the architecture is a credible piece of engineering: a tokenizer that keeps JSON keys and values atomic, a position encoding (KVPE) derived from the stack of a pushdown automaton, and grammar masking applied during training and inference. The ablations are careful and mostly support the component choices. Second, the headline claim that ORIGAMI beats convolutional and graph neural networks on CodeNet Java250 is not yet supported, because the test split and preprocessing protocol are unspecified. The 0.6-point margin over the GNN could easily be split- or preprocessing-induced noise.

What is genuinely new is the combination: KVPE is a sensible way to encode hierarchical key paths and array positions in a permutation-invariant manner, and using the PDA both for position encoding and for masking invalid next tokens during training is a clean idea. The order-permutation upscaling is a reasonable adaptation of XLNet-style factorization to JSON, and the Dungeons synthetic probe is a good way to isolate the value of the position encoding. The DDXPlus multi-label results are strong and appear well controlled, with multiple seeds and a clear advantage for the object-structured representation.

Now the soft spots, in proportion. The CodeNet issue is real: Section 4.3 does not state which split of the 75,000 submissions was used, how it relates to Puri et al.'s baselines, or how many test instances were evaluated. The paper also truncates sequences at 4000 tokens and discards about 1% of instances, relabeling them as UNKNOWN during evaluation; if the baselines were not subject to the same policy, the comparison is not apples-to-apples. That should be fixed before the claim is taken at face value. Second, Table 2 shows zero-variance F1 for all models while precision and recall have non-zero variance; that inconsistency needs an explanation or a correction. Third, the footnote in Section 3.4 admitting that the last array element has no array symbol on the stack is waved away with 'does not affect model performance' but no evidence is given. Fourth, there is no code release, which limits reproducibility. Finally, the Dungeons dataset is a designed probe, not a real-world benchmark; the 'only ORIGAMI generalizes' framing is fine as an ablation result, but should not be read as evidence of general superiority.

The central argument holds up: the architecture does what it claims, and the ablations support the importance of the components. This deserves a serious referee. I would send it to review and ask the authors to clarify the CodeNet split, report variance or seeds on that comparison, and fix the Table 2 inconsistency.

Referee Report

2 major / 7 minor

Summary. ORIGAMI is a decoder-only transformer designed for end-to-end supervised learning on semi-structured JSON data. It makes three contributions: (1) a tokenizer that treats keys and primitive values as atomic tokens and adds grammatical tokens for objects and arrays; (2) a Key/Value Position Encoding (KVPE) computed as the sum of embeddings of the stack symbols of a pushdown automaton parsing the token stream, making the encoding invariant to the order of key/value pairs and compositional over nested key paths; and (3) PDA-based guardrails that mask invalid next tokens both during training and inference. Classification is cast as next-token prediction, so the same architecture handles single-label and multi-label targets and can generate array-valued outputs. Experiments cover eight UCI tabular benchmarks converted to JSON, the DDXPlus multi-label medical diagnosis dataset, code classification on CodeNet Java250, and ablations on a synthetic Dungeons key-lookup task. The authors report that ORIGAMI is competitive with GBDTs on the tabular benchmarks, outperforms multi-output baselines on DDXPlus, and, on CodeNet, exceeds MLP, CNN and GNN baselines (94.7% vs. 94.1% for the best GNN) while remaining below CodeBERT.

Significance. Assuming the empirical claims hold, ORIGAMI is a useful and timely contribution: it is an end-to-end generative model for semi-structured data, with a principled position encoding (KVPE) grounded in a pushdown automaton and a clean mechanism for constrained training and inference. The evaluation is generally careful: 5-fold CV with hyperparameter search on the UCI benchmarks, 5-seed runs on an independent DDXPlus test set, and well-designed ablations (PE variants, guardrails, permutation upscaling) that provide falsifiable, machine-checkable evidence for each contribution. The main deficits are in the CodeNet experiment, where the test split and preprocessing protocol are unspecified and the 0.6-point margin over the best GNN is within the range of split and preprocessing noise, and in the Dungeons ablation, which is deliberately constructed to favor KVPE. These issues do not invalidate the architecture but currently prevent the paper from supporting its strongest headline claim.

major comments (2)

[4.3, Table 3] The claim that ORIGAMI 'outperform[s] ... specialized models such as convolutional and graph neural networks on a code classification task' (Abstract) rests on Table 3, but the evaluation protocol is not specified. The text does not state how the 75,000 CodeNet Java250 submissions are partitioned into train/test, how many test instances ORIGAMI is evaluated on, or whether the split coincides with the one underlying the baseline numbers taken from Puri et al. [29]. In addition, ORIGAMI truncates sequences and vocabulary at 4000 tokens, discards roughly 1% of instances, and, per the text, classifies discarded instances as UNKNOWN, whereas the baselines' preprocessing is not described. Given a 0.6 percentage-point margin over the best GNN with no variance, no seed count, and no test-set size, the comparison is not yet apples-to-apples. Please report the split, the number of test instances, the standard deviation over several seeds, and either rerun the baselines under identical preprocessing or demonstrate that Puri et al.'s published split and preprocessing match.
[4.4.2, Figures 5 and 7] The Dungeons experiment is constructed in a way that strongly favors ORIGAMI: the target depends on a key-path lookup (door_no then key_color), while the flattened representation given to tabular baselines uses positional column names such as corridor.4.blue_key, which are not invariant to the shuffling of door objects. Hence the tabular baselines cannot solve the task regardless of their capacity; the 100%-vs-32-41% result is a mechanism check for KVPE, not evidence that ORIGAMI generally outperforms tabular models on semi-structured data. I recommend stating this explicitly and, if feasible, adding a variant where tabular baselines are given the correctly aligned key paths, so the comparison isolates the effect of structure preservation.

minor comments (7)

[Table 2] Five independent runs with different seeds are reported with 0.0% standard deviation for F1 on every model, while precision and recall show 0.1-0.2% standard deviations; please clarify whether F1 (and the other metrics) are computed on pooled predictions per run or whether the 0.0% values are rounding artifacts. Also state whether F1 is micro- or macro-averaged.
[3.4, footnote 3] The KVPE encoding drops the array-position symbol for the last element of an array: for an array of length n, the stack contains Array(n) for the first element, Array(n-1) for the second, ..., and no array symbol for the last element, so the last element is encoded identically to a scalar value under the same key path. The paper acknowledges this in a footnote but does not explain why this boundary case does not affect the benchmarks; please either fix the encoding (e.g., with a dedicated LAST_ARRAY_ITEM symbol) or provide evidence that this quirk does not impact the reported results.
[4.2] The decoding procedure for multi-label outputs is underspecified: when the model emits an Array(n) token, the paper says it 'greedily extracts n additional tokens', but it should state how n is determined and how invalid or repeated labels are handled. Also, the MOC baselines are fitted on a binary matrix while ORIGAMI directly predicts the label list; this is a fair difference, but the paper should explicitly note that the baselines do not receive the label co-occurrence structure.
[4.3, Table 6] The CodeNet hyperparameters are listed with a dagger indicating they were excluded from the hyperparameter search, but no justification or selection procedure is given; please state how these values were chosen (e.g., from prior experiments or a small validation sweep).
[3.1, Abstract] The abstract and Section 3.1 describe the tokenization as 'reversible', but reversibility holds only for sequences that represent valid objects, as the footnote in Section 3.3 notes. Please qualify the claim, for instance, as 'reversible on the language of valid token sequences'.
[Figures 4, 6-8] The figures would be easier to interpret with error bars or shaded confidence bands around the averaged curves, particularly Figure 4 and Figure 8, and with larger axis labels. As printed, the 'O' markers for ordered sequences and the many thin individual-run lines are hard to distinguish.
[Section 1] The novelty statement 'to the best of our knowledge this work is the first to utilize the same approach during training to accelerate convergence' should be softened or supported by a broader comparison with the guided-decoding literature, since training-time masking with grammars has been explored in other settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all central claims are benchmarked against external datasets and published baselines, with ablations as controlled experiments rather than fitted predictions.

full rationale

The paper's central claims are empirical: ORIGAMI is evaluated on JSONified UCI tabular benchmarks (Section 4.1), the DDXPlus multi-label diagnosis task (Section 4.2), CodeNet Java250 (Section 4.3), and synthetic Dungeons ablations (Section 4.4). None of these results is derived from a fitted parameter that is then renamed as a prediction. The KVPE position encoding is defined by construction (Section 3.4, Eq. 2) as a sum of stack-symbol embeddings, and its claimed order-invariance follows from the invariance of the stack state under sibling permutation; this is a stated design property, not a prediction about held-out data. The Dungeons experiments test whether models can learn to follow key-path clues; the task is generated independently of the model, and the finding that only KVPE generalizes is a controlled empirical outcome, not an analytic consequence of the encoding definition. The paper contains no load-bearing self-citations: the authors cite external work for baselines and related methods (e.g., json2vec [45], CodeNet [29], XLNet [48]), and no uniqueness theorem or prior result by the same authors is invoked to force a conclusion. The CodeNet Java250 comparison has a possible validity limitation, because Section 4.3 does not fully specify the train/test split or confirm that truncation and UNKNOWN relabeling match Puri et al.'s protocol; however, that is a benchmarking/protocol concern, not circularity, since the ORIGAMI accuracy is measured on held-out data rather than being equivalent to its own inputs. No equation is equal by construction to another equation it is claimed to predict, and no fitted value is presented as an independent forecast. Therefore the appropriate finding is no significant circularity, score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

The central claims rest on a hand-crafted PDA whose stack states define position embeddings, atomic string tokenization with an [UNKNOWN] fallback, and an order-permutation augmentation scheme; hyperparameters such as the upscaling factor are fit per dataset. No formal proof of KVPE's uniqueness properties is provided.

free parameters (3)

upscaling factor uf = automobile 400, bank 4, car 4, contraceptive 1000, mushroom 100, nursery 40, seismic 1000, student 1000, DDXPlus 2
Chosen by hyperparameter search; controls how many order permutations are added to the training set and materially changes accuracy (Table 4: automobile 77.6% without vs 83.4% with).
CodeNet sequence-length truncation = 4000 tokens
Hand-chosen memory limit; discards approx. 1% of instances and replaces rare tokens with [UNKNOWN], affecting the evaluated accuracy.
CodeNet vocabulary truncation = 4000 entries
Hand-chosen; least frequent tokens beyond this limit are replaced with [UNKNOWN], which can degrade representation of rare AST node types.

assumptions (5)

standard math Transformer architecture computes next-token probabilities via self-attention and softmax with cross-entropy training.
Section 3.2 describes f as stacked decoder-only transformer blocks; no proof needed, background result.
domain assumption JSON objects are unordered collections of key/value pairs (RFC 7159), so all key permutations are equally valid inputs.
Invoked in Sections 2.4 and 3.4 to justify permutation-based training and order-invariant KVPE.
ad hoc to paper The PDA in Figure 2 correctly recognizes exactly the language of token sequences produced by the tokenizer.
Section 3.3: guardrails and KVPE both depend on the stack states and transition validity; an incorrect PDA would corrupt position embeddings and masks.
domain assumption Treating string values as atomic vocabulary tokens and mapping unseen test values to [UNKNOWN] preserves task-relevant information.
Section 3.1 defines Vvalue and the [UNKNOWN] token; for classification, target values are typically in the training vocabulary, but evidence values and code AST names may not be.
ad hoc to paper The synthetic Dungeons dataset is a valid proxy for real-world nested key-lookup tasks.
Section 4.4.2 uses it to claim ORIGAMI generalization and to compare position encodings; the dataset is generated by the authors, so the conclusion inherits its design assumptions.

invented entities (3)

Key/Value Position Encoding (KVPE)
purpose: Sum of stack-symbol embeddings encodes each token's key path and array position, making position independent of sibling order.
Validated only on the paper's synthetic Dungeons task and internal ablations (Figures 6-7); no external benchmark or formal uniqueness proof.
PDA-based training guardrails
purpose: Mask logits of grammatically invalid next tokens during training and inference to accelerate convergence.
Ablation on dungeons-easy/hard (Section 4.4.3) only; no external replication.
Order-permutation upscaling
purpose: Augment training data by sampling shuffled key/value permutations, enabled by KVPE's order invariance.
Evaluated only on the paper's datasets (Section 4.4.1); external utility not demonstrated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ORIGAMI: A generative transformer architecture for predictions from semi-structured data." pith.science (2026). https://pith.science/paper/P3ALIRHX

@misc{pith2026241217348,
  author       = {Pith},
  title        = {Pith review of: ORIGAMI: A generative transformer architecture for predictions from semi-structured data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P3ALIRHX}},
  note         = {Machine review of arXiv:2412.17348}
}

read the original abstract

Despite the popularity and widespread use of semi-structured data formats such as JSON, end-to-end supervised learning applied directly to such data remains underexplored. We present ORIGAMI (Object RepresentatIon via Generative Autoregressive ModellIng), a transformer-based architecture that directly processes nested key/value pairs while preserving their hierarchical semantics. Our key technical contributions include: (1) a structure-preserving tokenizer, (2) a novel key/value position encoding scheme, and (3) a grammar-constrained training and inference framework that ensures valid outputs and accelerates training convergence. These enhancements enable efficient end-to-end modeling of semi-structured data. By reformulating classification as next-token prediction, ORIGAMI naturally handles both single-label and multi-label tasks without architectural modifications. Empirical evaluation across diverse domains demonstrates ORIGAMI's effectiveness: On standard tabular benchmarks converted to JSON, ORIGAMI remains competitive with classical and state-of-the-art approaches. On native JSON datasets, we outperform baselines on multi-label classification and specialized models such as convolutional and graph neural networks on a code classification task. Through extensive ablation studies, we validate the impact of each architectural component and establish ORIGAMI as a robust framework for end-to-end learning on semi-structured data.

Figures

Figures reproduced from arXiv: 2412.17348 by the authors.

**Figure 1.** Overall architecture of ORIGAMI consisting of preprocessing of documents into integer sequences (a) and model architecture for training and inference (b). and generating a sequence of key, value and grammatical tokens. Each token is taken from a global vocabulary t (j) i ∈ V, and n is chosen large enough to fit all resulting sequences from the dataset O. Shorter sequences are right-padded with a special [PAD] token.… view at source ↗

**Figure 2.** PDA transition diagram, where edges are labeled with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Evolution of stack states when parsing the token sequence of the example [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Varying levels of data upscaling on the contraceptive dataset. We observe that low upscaling factors lead to overfitting on the training data, with typical increase of test loss after initial drop. With increasing upscaling factors beyond 5x, this phenomenon is mitigat…

**Figure 5.** Figure 5: The Dungeons synthetic dataset. The corridor array contains between 4 and 8 objects, each has a door_no key, 3 color-coded keys (red_key, green_key, blue_key), and between 0 and 2 monsters, randomly selected. The monsters, if present, add further randomness to the posi…

**Figure 6.** Figure 6: Loss and accuracy for ORIGAMI models with 4 different position encoding strategies. Only KVPE generalizes on the test data [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Train and test accuracy on dungeonshard dataset for ORIGAMI and baselines. Only ORIGAMI generalizes to held-out data [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 30 canonical work pages

[29]

Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks, August 2021. ...

work page 2021
[1]

Alcorn and Anh Nguyen

Michael A. Alcorn and Anh Nguyen. The DEformer: An Order-Agnostic Distribution Estimating Trans- former, July 2021. Preprint. URL https://arxiv.org/abs/2106.06989

arXiv 2021
[2]

Tabnet: Attentive interpretable tabular learning

Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679–6687, 2021

work page 2021
[3]

Transformers for tabular data representation: A tutorial on models and applications

Gilbert Badaro and Paolo Papotti. Transformers for tabular data representation: A tutorial on models and applications. Proceedings of the VLDB Endowment, 15(12):3746–3749, August 2022. ISSN 2150-8097. doi: 10.14778/3554821.3554890

arXiv 2022
[4]

The JavaScript Object Notation (JSON) Data Interchange Format

Tim Bray. The JavaScript Object Notation (JSON) Data Interchange Format. Request for Comments RFC 7159, Internet Engineering Task Force, March 2014

work page 2014
[5]

XGBoost: A Scalable Tree Boosting System

Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, August 2016. doi: 10.1145/2939672.2939785

arXiv 2016
[6]

TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data

Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 322–331. Association for Computing Machinery, August 2021

work page 2021
[7]

Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey

Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun (Jane) Qi, Scott Nickleach, Diego Socolinsky, "SHS" Srinivasan Sengamedu, and Christos Faloutsos. Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey. Transactions on Machine Learning Research, 2024

work page 2024

Show all 51 references

[8]

DDXPlus: A new dataset for automatic medical diagnosis

Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn. DDXPlus: A new dataset for automatic medical diagnosis. Advances in Neural Information Processing Systems, 35:31306–31318, 2022

2022
[9]

CodeBERT: A Pre-Trained Model for Programming and Natural Languages, September 2020

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A Pre-Trained Model for Programming and Natural Languages, September 2020. Preprint. URL https://arxiv.org/abs/2002.08155

2020 arXiv
[10]

A new algorithm for data compression

Philip Gage. A new algorithm for data compression. C Users J., 12(2):23–38, February 1994. ISSN 0898-9788

1994
[11]

Convolutional Sequence to Sequence Learning

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional Sequence to Sequence Learning. In International Conference on Machine Learning, pages 1243–1252. PMLR, 2017. 16

2017
[12]

Goller and A

C. Goller and A. Kuchler. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of International Conference on Neural Networks (ICNN’96), volume 1, pages 347–352 vol.1, June 1996. doi: 10.1109/ICNN.1996.548916

1996
[13]

Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 507–520, Red Hook, NY , USA, 2022...

2022
[14]

Long Short-Term Memory

Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735– 1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735

1997 doi
[15]

TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second, September 2023

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second, September 2023. Preprint. URL https: //arxiv.org/abs/2207.01848

2023 arXiv
[16]

Multilayer feedforward networks are uni- versal approximators

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks , 2(5):359–366, January 1989. ISSN 0893-6080. doi: 10.1016/0893-6080(89)90020-8

1989 doi
[17]

TabTransformer: Tabular Data Modeling Using Contextual Embeddings

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. https://arxiv.org/abs/2012.06678v1, December 2020. Preprint. URL https://arxiv.org/abs/2012.06678

2012 arXiv
[18]

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

2017
[19]

Adam: A Method for Stochastic Optimization

Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations, December 2014

2014
[20]

Automata-based constraints for language model decoding

Terry Koo, Frederick Liu, and Luheng He. Automata-based constraints for language model decoding. In First Conference on Language Modeling, 2024

2024
[21]

TabDDPM: Modelling Tabular Data with Diffusion Models

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. TabDDPM: Modelling Tabular Data with Diffusion Models. In Proceedings of the 40th International Conference on Machine Learning, pages 17564–17579. PMLR, July 2023

2023
[22]

The Neural Autoregressive Distribution Estimator

Hugo Larochelle and Iain Murray. The Neural Autoregressive Distribution Estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 29–37. JMLR Workshop and Conference Proceedings, June 2011

2011
[23]

TreeRNN: Topology-preserving deep graph embedding and learning

Yecheng Lyu, Ming Li, Xinming Huang, Ulkuhan Guler, Patrick Schaumont, and Ziming Zhang. TreeRNN: Topology-preserving deep graph embedding and learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7493–7499. IEEE, 2021

2020
[24]

JsonGrinder.jl: Automated differentiable neural architecture for embedding arbitrary JSON data

Šimon Mandlík, Matˇej Raˇcinský, Viliam Lisý, and Tomáš Pevný. JsonGrinder.jl: Automated differentiable neural architecture for embedding arbitrary JSON data. Journal of Machine Learning Research, 23(298): 1–5, 2022. ISSN 1533-7928

2022
[25]

The UCI Machine Learning Repository

Kelly Markelle, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Repository. Website. URL https://archive.ics.uci.edu/
[26]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learni...

2011
[27]

Approximation capability of neural networks on spaces of probability measures and tree-structured domains, June 2019

Tomas Pevny and V ojtech Kovarik. Approximation capability of neural networks on spaces of probability measures and tree-structured domains, June 2019. Preprint. URL https://arxiv.org/abs/1906. 00764

2019
[28]

CatBoost: Unbiased boosting with categorical features

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr V orobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: Unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 2018

2018
[30]

Improving Language Understanding by Generative Pre-Training,

Alec Radford and Karthik Narasimhan. Improving Language Understanding by Generative Pre-Training,
[31]

The graph neural network model

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61–80, January 2009. ISSN 1941-0093. doi: 10.1109/TNN.2008.2005605

2009
[32]

Novel positional encodings to enable tree-based transformers

Vighnesh Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

2019
[33]

Tabular data: Deep learning is not all you need

Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, 2022

2022
[34]

Introduction to the Theory of Computation

Michael Sipser. Introduction to the Theory of Computation. International Thomson Publishing, 1st edition, November 1996. ISBN 978-0-534-94728-6

1996
[35]

Bayan Bruss, and Tom Goldstein

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training, June 2021. Preprint. URL https://arxiv.org/abs/2106.01342

2021 arXiv
[36]

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15 (56):1929–1958, 2014. ISSN 1533-7928

1929
[37]

Roformer: Enhanced transformer with rotary position embedding

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

2024
[38]

Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. In Chengqing Zong and Michael Strube, editors, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguis...

2015 doi
[39]

A deep and tractable density estimator

Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pages 467–475. PMLR, 2014

2014
[40]

Gomez, Łukasz Kaiser, and Illia Polosukhin

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, 2017

2017
[41]

Statistical learning theory: Models, concepts, and results

Ulrike V on Luxburg and Bernhard Schölkopf. Statistical learning theory: Models, concepts, and results. In Handbook of the History of Logic, volume 10, pages 651–706. Elsevier, 2011

2011
[42]

A Survey on Self-supervised Learning for Non-sequential Tabular Data

Wei-Yao Wang, Wei-Wei Du, Derek Xu, Wei Wang, and Wen-Chih Peng. A Survey on Self-supervised Learning for Non-sequential Tabular Data. 2024. Preprint. URLhttps://arxiv.org/abs/2402.01204

2024 arXiv
[43]

Willard and Rémi Louf

Brandon T. Willard and Rémi Louf. Efficient Guided Generation for Large Language Models, August
[44]

Deep Learning on Semi-Structured Data and Its Applications to Video-game AI

William Woof. Deep Learning on Semi-Structured Data and Its Applications to Video-game AI. Doctoral Thesis, University of Manchester, 2020

2020
[45]

A Framework for End-to-End Learning on Semantic Tree-Structured Data, February 2020

William Woof and Ke Chen. A Framework for End-to-End Learning on Semantic Tree-Structured Data, February 2020. Preprint. URL https://arxiv.org/abs/2002.05707

2020 arXiv
[46]

Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, October 2016

Yonghui Wu et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, October 2016. Preprint. URL https://arxiv.org/abs/1609.08144

2016 arXiv
[47]

Modeling tabular data using conditional GAN

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional GAN. Advances in neural information processing systems, 32, 2019

2019
[48]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V . Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding, January 2020. Preprint. URL https://arxiv.org/abs/1906.08237

2020 arXiv
[49]

AGE": 12,

Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M. Hellerstein, Sanjay Krishnan, and Ion Stoica. Deep Unsupervised Cardinality Estimation. Proceedings of the VLDB Endowment, 13(3):279–292, November 2019. ISSN 2150-8097. doi: 10....

2019
[2018]

URL https://cdn.openai.com/research-covers/language-unsupervised/ language_understanding_paper.pdf

Preprint. URL https://cdn.openai.com/research-covers/language-unsupervised/ language_understanding_paper.pdf
[2023]

https://arxiv.org/abs/2307.09702

Preprint. https://arxiv.org/abs/2307.09702

arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [29]

Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks, August 2021. ...

work page 2021

[2] [1]

Alcorn and Anh Nguyen

Michael A. Alcorn and Anh Nguyen. The DEformer: An Order-Agnostic Distribution Estimating Trans- former, July 2021. Preprint. URL https://arxiv.org/abs/2106.06989

arXiv 2021

[3] [2]

Tabnet: Attentive interpretable tabular learning

Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679–6687, 2021

work page 2021

[4] [3]

Transformers for tabular data representation: A tutorial on models and applications

Gilbert Badaro and Paolo Papotti. Transformers for tabular data representation: A tutorial on models and applications. Proceedings of the VLDB Endowment, 15(12):3746–3749, August 2022. ISSN 2150-8097. doi: 10.14778/3554821.3554890

arXiv 2022

[5] [4]

The JavaScript Object Notation (JSON) Data Interchange Format

Tim Bray. The JavaScript Object Notation (JSON) Data Interchange Format. Request for Comments RFC 7159, Internet Engineering Task Force, March 2014

work page 2014

[6] [5]

XGBoost: A Scalable Tree Boosting System

Tianqi Chen and Carlos Guestrin. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, August 2016. doi: 10.1145/2939672.2939785

arXiv 2016

[7] [6]

TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data

Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 322–331. Association for Computing Machinery, August 2021

work page 2021

[8] [7]

Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey

Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun (Jane) Qi, Scott Nickleach, Diego Socolinsky, "SHS" Srinivasan Sengamedu, and Christos Faloutsos. Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey. Transactions on Machine Learning Research, 2024

work page 2024