TACTICL: Task-Aware Compression of Tabular ICL Models

Katharina Eggensperger; Matthias Feurer; Mykhailo Koshil

REVIEW 4 major objections 4 minor 31 references

TACTICL: Task-Aware Compression of Tabular ICL Models

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TACTICL replaces up to 85% of transformer layers with lightweight adapters while keeping tabular accuracy and cutting inference cost.

desk verdict Useful, honestly-reported compression method for tabular ICL models — solid core experiments, but the ICL-preservation claim is only tested before fine-tuning, and the authors admit the gap. read the letter →

arxiv 2608.10837 v1 pith:7SN2QPAV submitted 2026-08-11 cs.LG cs.AI

Mykhailo Koshil , Matthias Feurer , Katharina Eggensperger This is my paper · ORCID

classification cs.LGcs.AI

keywords tabularfoundationmodelsin-contextlearningmodelcompressionlayerpruningadaptermodulestask-awarein-weightinferencespeed-up

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TACTICL is a task-aware compression method for tabular in-context-learning models such as TabPFNv2.5. The paper tries to show that, instead of distilling the model into a task-specific network and losing in-context adaptability, one can replace up to 85% of transformer layers with lightweight adapter networks trained on the downstream task and keep most of the accuracy. On 47 TabArena datasets, the resulting models run substantially faster, with median speedups of 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression for classification, and up to 5.4x on larger synthetic data. The paper also argues that the compressed model remains robust to feature-order permutation, injected irrelevant features, and label flips, and that freezing the transformer layers during optional fine-tuning keeps the in-context learning circuitry intact. If true, this gives practitioners a cheaper deployment route for tabular foundation models without forfeiting the ability to adapt to new data at inference time.

What carries the argument

The engine of the method is layer substitution with a lightweight two-layer MLP adapter (hidden width 128, GELU activation, skip connection) that is trained, before the search, to approximate the input–output mapping of the transformer layer it will replace. A greedy search then iteratively removes the layer whose substitution hurts validation AUC (or RMSE) least, relying on the locality assumption that each layer decision can be made independently of the others. Optionally, a final fine-tuning stage updates only the adapter weights with a few epochs of AdamW on a mixture of downstream data and synthetic data from the model's pretraining prior, deliberately keeping all transformer weights frozen so that learning is confined to the in-weight pathway. The paper's evidence that cheap structural proxies such as cosine similarity, CKA similarity, residual contributions, or adapter reconstruction loss fail to predict which layers can be removed is what motivates using the task's own validation metric as the guiding signal.

What would settle it

Take a TACTICL-compressed and fine-tuned model at 85% compression, then evaluate it as a few-shot learner on a held-out tabular dataset from a different domain than the one used for compression, under the same context perturbations used in the paper (feature-order permutation, injected random features, label flips). If its performance on that new task is substantially worse than the full uncompressed model's performance under the same conditions, and the gap grows with compression depth, the claim that in-context ability remains intact would be falsified. A cheaper calculation would be to compare the fine-tuned compressed model's accuracy-versus-number-of-in-context-examples slope on an unseen task against the full model's slope; a flattened slope indicates the in-context circuitry was damaged rather than preserved.

Watch

Extended reading notes

Core claim

The central claim is that transformer depth in tabular in-context-learning models is highly redundant for any given task, and that the redundancy can be exploited task-by-task without collapsing the model into a pure in-weight learner. TACTICL does this by searching over which layers to drop or substitute, replacing removed layers with small MLP adapters trained to reconstruct the removed layer's hidden-state mapping, and then optionally fine-tuning only those adapters while every original transformer weight stays frozen. The authors report that on TabPFNv2.5, substituting up to 15 of 25 layers produces negligible performance degradation, and even at 85% compression classification AUC drops by only about 4% in the median, with inference speed-ups of 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression. They interpret the robustness of compressed models to perturbations and their negative cross-dataset transfer as evidence that the model has specialized to the downstream task while retaining in-context learning: the retained transformer layers can still read new contexts, and the adapters supply task-specific in-weight knowledge.

Load-bearing premise

The load-bearing premise is that a model's ability to learn from examples shown at inference time (in-context learning) and its ability to store task knowledge in its weights (in-weight learning) use separate internal circuits, so freezing all transformer layers and training only the small adapters can specialize the model to a task without damaging its in-context flexibility.

Editorial extensions

If this is right

Tabular ICL models can be deployed at a fraction of their original inference cost: classification speed-ups are 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression, with median AUC effectively unchanged at 25–50% and roughly 4% lower at 85%.
Because the transformer backbone is frozen, the compressed model is not a one-shot distillation artifact: a new task can be handled by training new adapters and rerunning the greedy search, keeping the model's in-context capability available for later tasks.
Fine-tuning at low compression rates can recover performance lost during pruning and sometimes improve on the full model, so the same machinery doubles as a domain-adaptation mechanism rather than only a compression tool.
No universal layer-removal order exists across datasets, so task-aware selection is necessary; the greedy search recovers more than 90% of the oracle configuration's AUC for up to eight substituted layers on TabPFNv2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Our inference: if the in-context/in-weight independence assumption transfers, the same layer-substitution recipe could be applied to other tabular in-context-learning models, and possibly to ICL-capable language models, with the adapter architecture and search metric adapted to each backbone.
Our inference: because the paper offers no principled rule for choosing a compression budget, one could predict the accuracy-versus-depth tradeoff from dataset meta-features (sample size, feature count, class balance) and stop the greedy search at a target cost.
Our inference: the authors note that they never directly measure the fine-tuned model's few-shot ICL performance; running that test before and after fine-tuning, with and without prior-data injection, would settle whether freezing the backbone truly protects in-context ability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: TACTICL is a useful compression method for tabular in-context-learning models, and the core empirical work is solid. But the abstract's ICL claim is broader than what is actually tested, and the authors admit the missing experiment in Appendix H.

The novel piece is the task-aware greedy search over which transformer layers to replace with adapters, guided by validation AUC, for tabular PFNs. Layer-dropping-plus-healing is established for LLMs, and early exit exists for tabular models, but this combination — searched substitution with adapter healing, explicitly aimed at keeping in-context ability — is genuinely new. The paper also earns credit for a useful negative result: cheap structural proxies (cosine similarity, CKA, residual norms) do not predict which layers matter in these models. The exhaustive oracle on TabPFNv2 (all 4096 configurations) is the right way to validate a greedy search, and greedy recovering over 90% of oracle AUC at up to eight substituted layers is concrete evidence that the locality assumption holds. Held-out search splits, three folds, and the perturbation tests are proper practice. Nothing here is circular; the search criterion is a standard validation procedure, and no equation is fitted to the headline result.

The soft spots, in proportion. The main one: the "leaves its in-context ability intact" claim is tested only on the compressed model before fine-tuning. The perturbation and cross-dataset evidence in Section 4.3 never touches the fine-tuned model, which is the final output of the full pipeline, and Appendix H says so explicitly. Given that retained ICL is the paper's whole differentiator from distillation, and given Singh et al.'s result that ICL can erode sharply during fine-tuning, this is a real gap — either the experiment or a qualified abstract is needed. Second, the headline "85% without substantial drop" is a median over datasets with wide spread and no significance tests (the authors state this in the checklist), and it is essentially a classification result: regression degrades far more sharply (RMSE +61% at 85%) without that qualification. Third, the cross-dataset negative transfer is read as benign task specialization, but it is also consistent with partial damage to the ICL circuitry; the evidence there is indirect. Minor: raw results are not released, and the prior-injection ablation is inconclusive — both disclosed.

Net: the compression method holds up; the ICL-preservation claim does not yet. This is a revise-and-resubmit, not a reject. Worth a serious referee, and worth your time if you work on tabular foundation models or structured compression.

Referee Report

4 major / 4 minor

Summary. The paper introduces TACTICL, a framework for task-aware compression of tabular in-context-learning (ICL) transformers such as TabPFNv2.5. The method trains lightweight MLP adapters to approximate removed transformer layers, uses a greedy search guided by a held-out validation split to select which layers to substitute, and optionally fine-tunes only the adapter weights while freezing the backbone, with periodic injection of synthetic prior-distribution data. Experiments on 47 TabArena datasets (34 classification, 13 regression) are used to support claims of up to 85% layer substitution with modest median performance loss, inference speed-ups of about 1.25x/1.63x/2.83x at 25%/50%/85% compression, and retained robustness to input perturbations and cross-dataset transfer. The paper also reports that common structural proxies fail to predict layer importance and that the best layers to delete differ from the best layers to substitute.

Significance. If the central claims hold, TACTICL is practically valuable: it offers a way to reduce inference cost of tabular foundation models while retaining some in-context adaptability, with a structured search that avoids exhaustive enumeration. The paper has notable strengths: exhaustive oracle evaluation on TabPFNv2, a held-out search split for configuration selection, three folds for TabPFNv2.5, comparison against a distillation baseline and several selection strategies, and ablations of fine-tuning and prior-data injection. The main differentiator from standard distillation, however, is the claim that ICL ability remains intact, and that claim is not empirically established for the full pipeline. The quantitative headline about 85% compression also rests on median ratios with wide dataset spread and no significance testing, so the strength of the evidence is currently below the level of the abstract's claims.

major comments (4)

[Section 3 F, Section 4.3, Section 4.4, Appendix H] The abstract's claim that TACTICL 'leaves its in-context ability intact' is not tested for the full method. The perturbation and cross-dataset evaluations in Section 4.3 (Figure 6) are performed on the compressed model selected by the greedy search before the fine-tuning stage F. Section 4.4 evaluates fine-tuned models only on in-distribution AUC/RMSE (Figure 7), and Appendix H explicitly states that assessing the ICL performance of the fine-tuned model is left to future work. Since F is presented as part of TACTICL (Figure 1, step 4; Section 3 F), and since the paper itself cites Singh et al. (2023) to note that ICL-to-IWL transitions can be sharp during fine-tuning, the current evidence does not support the claim that the full pipeline preserves ICL. Please add an ICL evaluation of the fine-tuned model (for example, re-running the perturbation and cross-dataset tests after F) or restrict the claim to the unfine-tuned compressed model.
[Section 4.2, Figure 5, Figure 8, submission checklist] The headline result of 'up to 85% of layers without substantial performance drop' is supported only by median AUC ratios with wide dataset-to-dataset spread and no statistical significance testing. In Figure 5b the 0.05-0.95 quantile band widens considerably as the number of substituted layers grows, and Figure 8a shows median AUC/AUCfull near 0.96 at 85% compression while the spread is not quantified beyond quantiles. The submission checklist states that statistical significance was not reported. Because 'substantial' is a quantitative threshold and the claim is central, please report per-dataset changes, paired confidence intervals or effect sizes, and a pre-specified definition of 'substantial'. The statement in Section 4.2 that TACTICL can 'replace up to 15 layers with negligible performance degradation' should also be qualified by the same uncertainty.
[Section 4.3, Figure 6b] The cross-dataset experiment is interpreted as evidence that TACTICL 'specializes the model' for the source task, but negative transfer is also exactly what would be observed if compression damaged the in-context circuitry while leaving the source-task in-weight pathway intact. The perturbation tests in Figure 6a use feature permutations, added random features, and label flips on the same underlying task; these are useful sanity checks but do not disambiguate task specialization from ICL damage. Since the paper's novelty claim is that ICL remains intact rather than merely that the model is robust to those perturbations, please report the fine-tuned model in the cross-dataset test, or measure ICL on held-out tasks sampled from the pretraining prior, or otherwise provide evidence that the negative transfer is not a symptom of broken ICL.
[Section 4.2, Figures 4 and 5] The claim that greedy search finds 'near-optimal configurations in our small-scale setting as well as on state-of-the-art models' is not fully supported on TabPFNv2.5. The oracle comparison in Figure 4 is performed only on TabPFNv2, where exhaustive enumeration is feasible; for TabPFNv2.5 the paper compares against the full model and baselines but not against any oracle or lower bound. The word 'near-optimal' should therefore be restricted to the TabPFNv2 setting, or the v2.5 section should clearly state that optimality is not evaluated there.

minor comments (4)

[Section 4.1, Figure 2] The caption 'Minimal, maximal, and Median of maximal relative performance change' is confusing, and the text says both 'maximum change' and 'median, maximum, and minimum of this metric across all datasets.' Please clarify the aggregation direction: is the maximum taken over configurations per dataset and then summarized across datasets, or is the maximum taken across datasets as well?
[Appendix H, Figure 10] The caption of Figure 10 says 'Change compared to the full model in AUC' but the figure reports RMSE for regression tasks. Please correct the caption.
[Section 4.5] The sentence 'the overhead for substitution is minimal compared to deletion' in the Figure 15 caption is not obvious from the plotted panels; please specify which overhead is being compared (e.g., adapter execution vs. pure deletion) and quantify it in the text.
[Section 1] The introduction cites 'TabPFNv3 (Grinsztajn et al., 2026)' but the reference list entry is titled 'Tabpfn-3: Technical report'; please make the citation and reference title consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TACTICL's compression results come from held-out search and independent test evaluation; the main caveat is an empirical coverage gap in the ICL claim, not a definitional or self-citational reduction.

full rationale

The core derivation chain is self-contained. TACTICL chooses layer configurations by greedy search on a held-out search split using AUC/RMSE, then reports performance on held-out test data; the search metric is not the test metric, so the main 'substitute up to 85%' claim is an empirical result rather than a quantity fitted to its own target. Adapters are trained to reconstruct hidden states of dropped layers, and the fine-tuning stage updates only adapter weights on a downstream-data/prior-data mixture; none of these steps defines the reported accuracy or speedup in terms of the search criterion. The self-citations to Balef et al. (2025, 2026) are used as motivation for layer redundancy, but Section 4.1 independently exhaustively evaluates all configurations on TabPFNv2, so the redundancy premise is re-established within the paper rather than imported as an unverified load-bearing theorem. The cited ICL/IWL independence result (Nguyen and Nallamala, 2025) is external and is treated as motivation, while the paper's own perturbation and cross-dataset experiments provide direct evidence at the compression stage. The one genuine weakness is that Appendix H admits: 'We have realized that an experiment that assesses the ICL performance of the fine-tuned model might shine more light on the benefits of the prior injection, and leave this to the future work.' This means the abstract's ICL-preservation claim is not fully verified for the final fine-tuned pipeline, and the perturbation evidence covers only the pre-fine-tuning compressed model. However, this is an empirical gap or over-claim, not circularity: no equation, fitted parameter, or self-citation chain makes the ICL claim true by construction. Under the hard rules requiring a specific reduction, no circular step is identified, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The ledger shows that TACTICL is an empirical method rather than a derivation. Its results rest on hand-chosen hyperparameters (adapter architecture, split ratios, prior-injection cadence) and on domain assumptions about layer locality, ICL/IWL independence, and adapter expressivity. No new theoretical entities are introduced.

free parameters (4)

Adapter architecture C3_wide(128) = hidden width 128, depth 2, GELU, skip connection
Selected as best of 10 configurations on 5 held-out TALENT datasets (Appendix E). This choice affects the substitution performance that underlies all compression results.
Search split ratio 8:1:1 = Dtrain2:Dval:Dsearch = 8:1:1
Chosen by hand in Section 4.2. The search budget and adapter training set depend on this split; a different split could change selected configurations.
Prior injection cadence = every 10 epochs, 2 batches per prior epoch
Hand-chosen in Appendix H; the ablation comparing no-prior vs with-prior injection is reported as inconclusive, so this parameter is not strongly validated.
Adapter pretraining hyperparameters = lr 4e-3, weight decay 1.5e-5, max 2000 epochs, patience 150
Fixed in Appendix D without per-dataset tuning. The quality of the adapter approximation depends on these choices.

assumptions (5)

domain assumption Layer decisions are local: the benefit of removing or replacing one layer does not depend on which other layers are removed or replaced.
Stated in Section 3 S as the locality assumption underlying greedy search. Empirically supported on TabPFNv2 (greedy recovers more than 90% of oracle AUC up to 8 substituted layers), but not proven for TabPFNv2.5.
domain assumption ICL and IWL sub-circuits are largely independent, so freezing transformer weights and training only adapters preserves ICL.
Invoked in Section 3 F, citing Nguyen and Nallamala (2025). The perturbation experiments provide indirect support, but cross-dataset experiments show negative transfer, indicating task specialization.
domain assumption A two-layer MLP adapter trained on cached hidden states can approximate the input-output mapping of a dropped transformer layer well enough to heal the model.
Central to the substitution mechanism; the reconstruction loss is MSE on hidden states (Appendix D), and no guarantee is given that this preserves the layer's role in in-context computation.
domain assumption AUC/RMSE on a held-out search set is a reliable proxy for test performance when selecting layers.
Used as the greedy selection criterion (Section 3 S). The paper shows structural proxies fail, and validation AUC is the only reliable one on the tested data, but the transfer to unseen datasets is not guaranteed.
domain assumption The 47 TabArena datasets that fit TabPFNv2.5 pretraining limits are representative enough to support general claims about tabular ICL compression.
Dataset selection is described in Appendix C.1; the authors acknowledge the evaluation is limited to tasks within TabPFNv2.5 pretraining constraints.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TACTICL: Task-Aware Compression of Tabular ICL Models." pith.science (2026). https://pith.science/paper/7SN2QPAV

@misc{pith2026260810837,
  author       = {Pith},
  title        = {Pith review of: TACTICL: Task-Aware Compression of Tabular ICL Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7SN2QPAV}},
  note         = {Machine review of arXiv:2608.10837}
}

read the original abstract

The strong performance of foundation models for tabular tasks comes at substantial inference costs. Distilling models into task-specific architectures reduces model size and computational demands but also sacrifices in-context adaptability. Here we introduce TACTICL, an automated task-aware compression framework for tabular in-context learning models that jointly prunes transformer layers and replaces them with lightweight adapters trained on downstream tasks, thus blending in-context with in-weight learning. We study TACTICL on 47 benchmark datasets and show that we can substitute up to 85% of layers without substantial performance drop on a given downstream task. We further show that TACTICL maintains robustness to data shifts, leaving its in-context ability intact. Overall, TACTICL provides a robust framework for exploiting the depth-wise redundancy of tabular foundation models by combining task-specific adaptation and structured compression. We provide the code at: https://github.com/Hebog/tfm_compression

Figures

Figures reproduced from arXiv: 2608.10837 by the authors.

**Figure 1.** Overview of TACTICL, which jointly compresses and adapts TabPFN models. All TabPFNv2 (Hollmann et al., 2025), TabPFNv2.5 (Grinsztajn et al., 2025), and TabPFNv3 (Grinsztajn et al., 2026) introduce dedicated distillation engines that produce, e.g., a compact multi-layer perceptron (MLP) for a given dataset with orders-of-magnitude lower latency. While effective, this approach explicitly converts the model to the in-… view at source ↗

**Figure 2.** Minimal, maximal, and Median of maximal relative performance change over 34 classification and 13 regression datasets from TabArena as a function of the number of layers dropped/substituted with adapters compared to the full model (black dotted line). This plot can be interpreted as worst/best/median performance of an oracle search algorithm. sets: permuting the feature order, injecting uninformative features, and … view at source ↗

**Figure 3.** Correlation between proxy scores and AUC for deleting (left) and substituting (right) 𝑛 layers of TabPFNv2 (across all configurations in 𝐶𝑛; see Eq.(1)), calculated from exhaustive evaluation in Section 4.1. None of the proxies reliably predict the performance impact, consistent with findings in Sajjad et al. (2023). Note that the best layers for dropping are not indicative of the best layers for substitution and vi… view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Greedy compression configuration search on TabPFNv2. We report the ratio of AUC𝑓 𝑜𝑢𝑛𝑑 to the best possible AUC on the test set as well as overall performance degradation (blue). For regression results, see [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: TACTICL substitution applied to TabPFNv2.5. We report the ratio of performances in AUC [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Evaluations on perturbed datasets (OOD) (6a) and cross-dataset generalization (evaluated on 3, 11, and 20 layers substituted) (6b) confirm that the compressed model retains its ICL capabilities beyond the training distribution. We show results for regression in [PITH_…

**Figure 7.** Figure 7: Change compared to the full model in AUC after fine-tuning the adapter weights at three compression levels. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Ratios of TACTICL speed-up vs task performance across three compression ratios on all evaluated datasets. (25%, 50%, 85% of transformer blocks substituted with a lightweight MLP) [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

**Figure 9.** Figure 9: Shapley across layer deletions/substitution based on the exhaustive evaluation of the TabPFNv2 over the 47 TabArena datasets. The lower the value, the more harmful the change applied to the layer. Individual point – dataset; bar – mean over all datasets. As can be seen…

**Figure 10.** Figure 10: Change compared to the full model in AUC after fine-tuning the adapter weights at three compression levels (25%, 50%, 85% of layers replaced). No Prior Injection With Prior Injection 0.90 0.92 0.94 0.96 0.98 1.00 1.02 A U C/A U Cful ↑l Compression depth: 0.25 No Prior…

**Figure 11.** Figure 11: Change compared to the full model in AUC after fine-tuning the adapter weights at three compression levels (25%, 50%, 85% of layers replaced). No Prior Injection With Prior Injection 1.000 1.025 1.050 1.075 1.100 1.125 R M S E/R M S Eful ↓l Compression depth: 0.25 No …

**Figure 12.** Figure 12: Change compared to the full model in AUC after fine-tuning the adapter weights at three compression levels (25%, 50%, 85% of layers replaced). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

**Figure 13.** Figure 13: Correlation between proxy scores and performance change for deleting (left) and substituting (right) 𝑛 layers for regression. None of the proxies reliably predict the performance impact. 2 4 6 8 10 12 Layers 1.0 1.1 1.2 1.3 1.4 R M S Efound/R M S E best Median 0.95 q…

**Figure 14.** Figure 14: Greedy search on TabPFN v2: ratio of RMSE found by the greedy configuration over the oracle RMSE on the test set and overall performance (blue). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

**Figure 15.** Figure 15: Speedup of TACTICL for 0.25, 0.5 and 0.85 compression ratios measured on the random synthetic datasets (classification task) of varying sample and feature sizes. The top row displays compression with deletion, bottom – substitution. As evident from the plots, the over…

**Figure 16.** Figure 16: TACTICL deletion applied to TabPFNv2.5. We report the ratio in performance drop in AUC aggregated across 34 datasets for dropping layers across 3 strategies. 0 3 6 9 12 15 18 Layers 1 1.2 1.5 2 3 4 5 6 7 8 10 12 R M S E / R M S Efull TACTICL q05-q95 TACTICL q50 Distil…

**Figure 17.** Figure 17: TACTICL substitution applied to TabPFNv2.5. We report the ratio in performance drop in RMSE aggregated across 13 datasets for substituting layers across 3 strategies. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]

**Figure 18.** Figure 18: TACTICL deletion applied to TabPFNv2.5. We report the relative difference in RMSE aggregated across 13 datasets for dropping layers across 3 strategies. 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Layers 0.50 0.75 1.00 1.25 1.50 1.75 2.00 R M S E perturbed/R M S Eunperturbed Feat…

**Figure 19.** Figure 19: Evaluations on perturbed datasets (OOD) (19a) and cross-dataset generalization (19b) confirm that the compressed model retains its ICL capabilities beyond the training distribution. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 27 canonical work pages

[1]

R., Koshil, M., and Eggensperger, K

Balef, A. R., Koshil, M., and Eggensperger, K. (2025). Towards understanding layer contributions in tabular in-context learning models. In EurIPS 2025 Workshop: AI for Tabular Data

work page 2025
[2]

R., Koshil, M., and Eggensperger, K

Balef, A. R., Koshil, M., and Eggensperger, K. (2026). Is one layer enough? understanding inference dynamics in tabular foundation models. In ICML 2026

work page 2026
[3]

B \'e thune, L., Grangier, D., Busbridge, D., Gualdoni, E., Cuturi, M., and Ablin, P. (2025). Scaling laws for forgetting during finetuning with pretraining data injection. In ICML 2025

work page 2025
[4]

M., Rodol \`a , E., Rieck, B., and Vogt, J

Cannistraci, I., Antonelli, S., Palumbo, E., Sutter, T. M., Rodol \`a , E., Rieck, B., and Vogt, J. E. (2026). TOAST : Transformer optimization using adaptive and simple transformations. Transactions on Machine Learning Research

work page 2026
[5]

Chen, X., Hu, Y., Zhang, J., Wang, Y., Li, C., and Chen, H. (2025). Streamlining redundant layers to compress large language models. In ICLR 2025

work page 2025
[6]

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. (2021). A mathematical framework for transfor...

work page 2021
[7]

M., Salinas, D., and Hutter, F

Erickson, N., Purucker, L., Tschalzev, A., Holzm \"u ller, D., Desai, P. M., Salinas, D., and Hutter, F. (2026). Tabarena: A living benchmark for machine learning on tabular data. In NeurIPS 2025 Datasets and Benchmarks Track

work page 2026
[8]

and Bansal, A

Gangwani, I. and Bansal, A. (2025). Light-weight benchmarks reveal the hidden hardware cost of zero-shot tabular foundation models. arXiv preprint arXiv:2512.00888

work page arXiv 2025

Show all 31 references

[9]

Garg, A., Ali, M., Hollmann, N., Purucker, L., M \"u ller, S., and Hutter, F. (2025). Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data. In 1st ICML Workshop on Foundation Models for Structured Data

2025
[10]

o ge, K., Key, O., Birkel, F., Jund, P., Roof, B., Manium, M., Hoo, S. B., B \

Grinsztajn, L., Fl \"o ge, K., Key, O., Birkel, F., Jund, P., Roof, B., Manium, M., Hoo, S. B., B \"u hler, M., Garg, A., et al. (2026). Tabpfn-3: Technical report. arXiv preprint arXiv:2605.13986

2026 arXiv
[11]

B., Garg, A., Robertson, J., Bühler, M., Moroshan, V., Purucker, L., Cornu, C., Wehrhahn, L

Grinsztajn, L., Flöge, K., Key, O., Birkel, F., Jund, P., Roof, B., Jäger, B., Safaric, D., Alessi, S., Hayler, A., Manium, M., Yu, R., Jablonski, F., Hoo, S. B., Garg, A., Robertson, J., Bühler, M., Moroshan, V., Purucker, L., Cornu, C., Wehrhahn, L. C., Bonetto, A., Schölkop...

2025 arXiv
[12]

Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. (2025). The unreasonable ineffectiveness of the deeper layers. In ICLR 2025

2025
[13]

Guo, D., Rush, A., and Kim, Y. (2021). Parameter-efficient transfer learning with diff pruning. In ACL-IJCNLP 2021

2021
[14]

Hollmann, N., M \"u ller, S., Eggensperger, K., and Hutter, F. (2023). Tabpfn: A transformer that solves small tabular classification problems in a second. In ICLR 2023

2023
[15]

u ller, S., Purucker, L., Krishnakumar, A., K \

Hollmann, N., M \"u ller, S., Purucker, L., Krishnakumar, A., K \"o rfer, M., Hoo, S. B., Schirrmeister, R. T., and Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature , 637(8045):319--326

2025
[16]

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP . In ICML 2019

2019
[17]

Kolberg, C., Kreuer, J., Huurdeman, J., Ouaari, S., Eggensperger, K., and Pfeifer, N. (2026). Tabpfn-wide: Continued pre-training for extreme feature counts. arXiv preprint arXiv:2510.06162v2

2026
[18]

K \"u ken, J., Purucker, L., and Hutter, F. (2025). Early stopping tabular in-context learning. In 1st ICML Workshop on Foundation Models for Structured Data

2025
[19]

H., Gurnee, W., and Tegmark, M

Lad, V., Lee, J. H., Gurnee, W., and Tegmark, M. (2025). Remarkable robustness of LLM s: Stages of inference? In NeurIPS 2025

2025
[20]

Li, Y., Yang, S., Tan, S., Mishra, M., Panda, R., Zhou, J., and Kim, Y. (2026). Distilling to hybrid attention models via KL -guided layer selection. In ICLR 2026

2026
[21]

Liu, S.-Y., Cai, H.-R., Zhou, Q.-L., Yin, H.-H., Zhou, T., Jiang, J.-P., and Ye, H.-J. (2025). Talent: A tabular analytics and learning toolbox. Journal of Machine Learning Research , 26(226)

2025
[22]

and Ye, H.-J

Liu, S.-Y. and Ye, H.-J. (2026). Tabswift: An efficient tabular foundation model with row-wise attention. In ICML 2026

2026
[23]

Men, X., Xu, M., Zhang, Q., Yuan, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. (2025). S hort GPT : Layers in large language models are more redundant than you expect. In ACL 2025

2025
[24]

Muralidharan, S., Turuvekere Sreenivas, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. (2024). Compact language models via pruning and knowledge distillation. In NeurIPS 2024

2024
[25]

and Nallamala, G

Nguyen, A. and Nallamala, G. R. (2025). Differential learning kinetics govern the transition from memorization to generalization during in-context learning. In ICLR 2025

2025
[26]

Qu, J., Holzm \"u ller, D., Varoquaux, G., and Le Morvan, M. (2026). TabICLv2 : A better, faster, scalable, and open tabular foundation model. In ICML 2026

2026
[27]

Rubachev, I., Kotelnikov, A., Kartashev, N., and Babenko, A. (2025). On finetuning tabular foundation models. arXiv preprint arXiv:2506.08982

2025 arXiv
[28]

Sajjad, H., Dalvi, F., Durrani, N., and Nakov, P. (2023). On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language , 77

2023
[29]

Shopkhoev, D., Ali, A., Zhussip, M., Malykh, V., Lefkimmiatis, S., Komodakis, N., and Zagoruyko, S. (2025). Replaceme: Network simplification via depth pruning and transformer block linearization. In NeurIPS 2025

2025
[30]

Singh, A., Chan, S., Moskovitz, T., Grant, E., Saxe, A., and Hill, F. (2023). The transient nature of emergent in-context learning in transformers. In NeurIPS 2023

2023
[31]

Tanna, A., Seth, P., Bouadi, M., and Sankarapu, V. K. (2026). Exploring fine-tuning for tabular foundation models. In ACM Web Conference 2026

2026

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

R., Koshil, M., and Eggensperger, K

Balef, A. R., Koshil, M., and Eggensperger, K. (2025). Towards understanding layer contributions in tabular in-context learning models. In EurIPS 2025 Workshop: AI for Tabular Data

work page 2025

[2] [2]

R., Koshil, M., and Eggensperger, K

Balef, A. R., Koshil, M., and Eggensperger, K. (2026). Is one layer enough? understanding inference dynamics in tabular foundation models. In ICML 2026

work page 2026

[3] [3]

B \'e thune, L., Grangier, D., Busbridge, D., Gualdoni, E., Cuturi, M., and Ablin, P. (2025). Scaling laws for forgetting during finetuning with pretraining data injection. In ICML 2025

work page 2025

[4] [4]

M., Rodol \`a , E., Rieck, B., and Vogt, J

Cannistraci, I., Antonelli, S., Palumbo, E., Sutter, T. M., Rodol \`a , E., Rieck, B., and Vogt, J. E. (2026). TOAST : Transformer optimization using adaptive and simple transformations. Transactions on Machine Learning Research

work page 2026

[5] [5]

Chen, X., Hu, Y., Zhang, J., Wang, Y., Li, C., and Chen, H. (2025). Streamlining redundant layers to compress large language models. In ICLR 2025

work page 2025

[6] [6]

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. (2021). A mathematical framework for transfor...

work page 2021

[7] [7]

M., Salinas, D., and Hutter, F

Erickson, N., Purucker, L., Tschalzev, A., Holzm \"u ller, D., Desai, P. M., Salinas, D., and Hutter, F. (2026). Tabarena: A living benchmark for machine learning on tabular data. In NeurIPS 2025 Datasets and Benchmarks Track

work page 2026

[8] [8]

and Bansal, A

Gangwani, I. and Bansal, A. (2025). Light-weight benchmarks reveal the hidden hardware cost of zero-shot tabular foundation models. arXiv preprint arXiv:2512.00888

work page arXiv 2025