{"id":"745a03cb-2846-4757-8ff8-c622db7fdec2","arxiv_id":"2608.10837","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"TACTICL replaces up to 85% of transformer layers in TabPFN models with lightweight adapters chosen per task, keeping most accuracy and in-context robustness while cutting inference cost.","lead":"This paper introduces TACTICL, a method that compresses tabular foundation models by removing transformer layers and replacing them with small task-trained adapters. It reports up to 85% layer substitution with modest median accuracy loss and up to 5.4x faster inference, while claiming to preserve in-context learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ICL-preservation claim is only tested on the compressed model before fine-tuning; the full TACTICL pipeline's fine-tuning step is never evaluated for ICL, and Appendix H admits this.","rationale":"The reader's CONDITIONAL verdict is well-founded. My stress-test converges on the same load-bearing area—the preservation of in-context learning—but sharpens it to a specific unexecuted experiment. The abstract promises that TACTICL 'leaves its in-context ability intact,' and the method as presented includes the fine-tuning stage (Section 3 F, Figure 1 step 4). All direct ICL evidence, however, comes from Section 4.3, which evaluates the greedy-searched compressed model before fine-tuning. Section 4.4 and Figures 7/10 measure only in-distribution AUC/RMSE after fine-tuning; no perturbation or cross-dataset test is applied to the fine-tuned model. Appendix H candidly states that an experiment assessing the ICL performance of the fine-tuned model is left to future work. Because fine-tuning updates task-specific adapter weights, it is precisely the stage at which ICL could be overwritten by IWL (cf. Singh et al., 2023, on sharp ICL-to-IWL transitions). The locality/independence assumption cited from Nguyen and Nallamala 2025 is theoretical; the perturbation tests cannot validate it for the fine-tuned pipeline since they were not run on it. If fine-tuning degrades perturbation robustness or cross-dataset transfer, the central claim fails for the delivered artifact, even if the pre-fine-tuning compressed model is robust. The concrete check—re-running Section 4.3's perturbations and cross-dataset transfer on the fine-tuned models—would settle this. Thus the verdict remains CONDITIONAL, with the additional condition that the fine-tuned model must pass the ICL tests.","tokens_in":19655,"tokens_out":9583,"duration_ms":99429,"concrete_test":"Run the Section 4.3 perturbation protocol (feature-order permutation, +20% random features, label flip) and the cross-dataset transfer experiment on the fine-tuned TACTICL models at compression depths 0.25, 0.5, and 0.85 across the same 47 TabArena datasets. Compare the resulting perturbed/unperturbed AUC ratios and AUC_source/AUC_target ratios against (i) the non-fine-tuned compressed models and (ii) the full TabPFNv2.5 model. If the fine-tuned models' perturbation ratios stay within the 0.05-0.95 quantile band of Figure 6a and cross-dataset transfer does not degrade substantially relative to the non-fine-tuned model, the ICL claim for the delivered pipeline is supported. If the ratios drop materially (e.g., label-flip ratio falls below 0.9), fine-tuning damages ICL and the claim must be revised.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"TACTICL's central claim is that it 'leaves its in-context ability intact' (Abstract). The full pipeline includes an optional fine-tuning stage (Section 3 F; Section 4.4) that updates adapter weights on the downstream task, and the method is presented with this stage as part of TACTICL (Figure 1, step 4; Figure 7). Yet the only evidence for ICL preservation—the perturbation and cross-dataset experiments of Section 4.3 (Figure 6)—is obtained on the compressed model selected by the greedy search, before fine-tuning. Section 4.4 evaluates fine-tuned models exclusively on in-distribution AUC/RMSE. Appendix H explicitly states: 'We have realized that an experiment that assesses the ICL performance of the fine-tuned model might shine more light on the benefits of the prior injection, and leave this to the future work.' Thus the load-bearing premise that freezing the backbone while updating adapters preserves ICL (Section 3 F, citing Nguyen and Nallamala 2025) is empirically unverified at the final stage of the method. Prior work (Singh et al., 2023) shows ICL-to-IWL transitions can be sharp and irreversible during fine-tuning; the perturbation tests cannot rule this out because they were not administered to the fine-tuned model. If fine-tuning erodes the remaining in-context circuitry, the abstract's claim fails even though the compressed-before-fine-tuning model is robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TACTICL, a framework for task-aware compression of tabular in-context-learning (ICL) transformers such as TabPFNv2.5. The method trains lightweight MLP adapters to approximate removed transformer layers, uses a greedy search guided by a held-out validation split to select which layers to substitute, and optionally fine-tunes only the adapter weights while freezing the backbone, with periodic injection of synthetic prior-distribution data. Experiments on 47 TabArena datasets (34 classification, 13 regression) are used to support claims of up to 85% layer substitution with modest median performance loss, inference speed-ups of about 1.25x/1.63x/2.83x at 25%/50%/85% compression, and retained robustness to input perturbations and cross-dataset transfer. The paper also reports that common structural proxies fail to predict layer importance and that the best layers to delete differ from the best layers to substitute.","tokens_in":19898,"tokens_out":4893,"duration_ms":50241,"significance":"If the central claims hold, TACTICL is practically valuable: it offers a way to reduce inference cost of tabular foundation models while retaining some in-context adaptability, with a structured search that avoids exhaustive enumeration. The paper has notable strengths: exhaustive oracle evaluation on TabPFNv2, a held-out search split for configuration selection, three folds for TabPFNv2.5, comparison against a distillation baseline and several selection strategies, and ablations of fine-tuning and prior-data injection. The main differentiator from standard distillation, however, is the claim that ICL ability remains intact, and that claim is not empirically established for the full pipeline. The quantitative headline about 85% compression also rests on median ratios with wide dataset spread and no significance testing, so the strength of the evidence is currently below the level of the abstract's claims.","major_comments":[{"comment":"The abstract's claim that TACTICL 'leaves its in-context ability intact' is not tested for the full method. The perturbation and cross-dataset evaluations in Section 4.3 (Figure 6) are performed on the compressed model selected by the greedy search before the fine-tuning stage F. Section 4.4 evaluates fine-tuned models only on in-distribution AUC/RMSE (Figure 7), and Appendix H explicitly states that assessing the ICL performance of the fine-tuned model is left to future work. Since F is presented as part of TACTICL (Figure 1, step 4; Section 3 F), and since the paper itself cites Singh et al. (2023) to note that ICL-to-IWL transitions can be sharp during fine-tuning, the current evidence does not support the claim that the full pipeline preserves ICL. Please add an ICL evaluation of the fine-tuned model (for example, re-running the perturbation and cross-dataset tests after F) or restrict the claim to the unfine-tuned compressed model.","section":"Section 3 F, Section 4.3, Section 4.4, Appendix H"},{"comment":"The headline result of 'up to 85% of layers without substantial performance drop' is supported only by median AUC ratios with wide dataset-to-dataset spread and no statistical significance testing. In Figure 5b the 0.05-0.95 quantile band widens considerably as the number of substituted layers grows, and Figure 8a shows median AUC/AUCfull near 0.96 at 85% compression while the spread is not quantified beyond quantiles. The submission checklist states that statistical significance was not reported. Because 'substantial' is a quantitative threshold and the claim is central, please report per-dataset changes, paired confidence intervals or effect sizes, and a pre-specified definition of 'substantial'. The statement in Section 4.2 that TACTICL can 'replace up to 15 layers with negligible performance degradation' should also be qualified by the same uncertainty.","section":"Section 4.2, Figure 5, Figure 8, submission checklist"},{"comment":"The cross-dataset experiment is interpreted as evidence that TACTICL 'specializes the model' for the source task, but negative transfer is also exactly what would be observed if compression damaged the in-context circuitry while leaving the source-task in-weight pathway intact. The perturbation tests in Figure 6a use feature permutations, added random features, and label flips on the same underlying task; these are useful sanity checks but do not disambiguate task specialization from ICL damage. Since the paper's novelty claim is that ICL remains intact rather than merely that the model is robust to those perturbations, please report the fine-tuned model in the cross-dataset test, or measure ICL on held-out tasks sampled from the pretraining prior, or otherwise provide evidence that the negative transfer is not a symptom of broken ICL.","section":"Section 4.3, Figure 6b"},{"comment":"The claim that greedy search finds 'near-optimal configurations in our small-scale setting as well as on state-of-the-art models' is not fully supported on TabPFNv2.5. The oracle comparison in Figure 4 is performed only on TabPFNv2, where exhaustive enumeration is feasible; for TabPFNv2.5 the paper compares against the full model and baselines but not against any oracle or lower bound. The word 'near-optimal' should therefore be restricted to the TabPFNv2 setting, or the v2.5 section should clearly state that optimality is not evaluated there.","section":"Section 4.2, Figures 4 and 5"}],"minor_comments":[{"comment":"The caption 'Minimal, maximal, and Median of maximal relative performance change' is confusing, and the text says both 'maximum change' and 'median, maximum, and minimum of this metric across all datasets.' Please clarify the aggregation direction: is the maximum taken over configurations per dataset and then summarized across datasets, or is the maximum taken across datasets as well?","section":"Section 4.1, Figure 2"},{"comment":"The caption of Figure 10 says 'Change compared to the full model in AUC' but the figure reports RMSE for regression tasks. Please correct the caption.","section":"Appendix H, Figure 10"},{"comment":"The sentence 'the overhead for substitution is minimal compared to deletion' in the Figure 15 caption is not obvious from the plotted panels; please specify which overhead is being compared (e.g., adapter execution vs. pure deletion) and quantify it in the text.","section":"Section 4.5"},{"comment":"The introduction cites 'TabPFNv3 (Grinsztajn et al., 2026)' but the reference list entry is titled 'Tabpfn-3: Technical report'; please make the citation and reference title consistent.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the compression results are interesting, but the central ICL-preservation claim is currently missing its decisive experiment: the fine-tuned model, which is part of TACTICL, is never evaluated for ICL. The authors themselves acknowledge this in Appendix H. I would recommend asking for that experiment or for a substantial reframing of the claim. The statistical support for the 85% headline is also thin; given the journal context, the editor may want to require per-dataset breakdowns and a clearer definition of 'substantial'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: TACTICL is a useful compression method for tabular in-context-learning models, and the core empirical work is solid. But the abstract's ICL claim is broader than what is actually tested, and the authors admit the missing experiment in Appendix H.\n\nThe novel piece is the task-aware greedy search over which transformer layers to replace with adapters, guided by validation AUC, for tabular PFNs. Layer-dropping-plus-healing is established for LLMs, and early exit exists for tabular models, but this combination — searched substitution with adapter healing, explicitly aimed at keeping in-context ability — is genuinely new. The paper also earns credit for a useful negative result: cheap structural proxies (cosine similarity, CKA, residual norms) do not predict which layers matter in these models. The exhaustive oracle on TabPFNv2 (all 4096 configurations) is the right way to validate a greedy search, and greedy recovering over 90% of oracle AUC at up to eight substituted layers is concrete evidence that the locality assumption holds. Held-out search splits, three folds, and the perturbation tests are proper practice. Nothing here is circular; the search criterion is a standard validation procedure, and no equation is fitted to the headline result.\n\nThe soft spots, in proportion. The main one: the \"leaves its in-context ability intact\" claim is tested only on the compressed model before fine-tuning. The perturbation and cross-dataset evidence in Section 4.3 never touches the fine-tuned model, which is the final output of the full pipeline, and Appendix H says so explicitly. Given that retained ICL is the paper's whole differentiator from distillation, and given Singh et al.'s result that ICL can erode sharply during fine-tuning, this is a real gap — either the experiment or a qualified abstract is needed. Second, the headline \"85% without substantial drop\" is a median over datasets with wide spread and no significance tests (the authors state this in the checklist), and it is essentially a classification result: regression degrades far more sharply (RMSE +61% at 85%) without that qualification. Third, the cross-dataset negative transfer is read as benign task specialization, but it is also consistent with partial damage to the ICL circuitry; the evidence there is indirect. Minor: raw results are not released, and the prior-injection ablation is inconclusive — both disclosed.\n\nNet: the compression method holds up; the ICL-preservation claim does not yet. This is a revise-and-resubmit, not a reject. Worth a serious referee, and worth your time if you work on tabular foundation models or structured compression.","headline":"Useful, honestly-reported compression method for tabular ICL models — solid core experiments, but the ICL-preservation claim is only tested before fine-tuning, and the authors admit the gap.","tokens_in":20525,"tokens_out":5765,"would_cite":true,"duration_ms":55147,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TACTICL replaces up to 85% of transformer layers with lightweight adapters while keeping tabular accuracy and cutting inference cost.","keywords":["tabular foundation models","in-context learning","model compression","layer pruning","adapter modules","task-aware compression","in-weight learning","inference speed-up"],"falsifier":"Take a TACTICL-compressed and fine-tuned model at 85% compression, then evaluate it as a few-shot learner on a held-out tabular dataset from a different domain than the one used for compression, under the same context perturbations used in the paper (feature-order permutation, injected random features, label flips). If its performance on that new task is substantially worse than the full uncompressed model's performance under the same conditions, and the gap grows with compression depth, the claim that in-context ability remains intact would be falsified. A cheaper calculation would be to compare the fine-tuned compressed model's accuracy-versus-number-of-in-context-examples slope on an unseen task against the full model's slope; a flattened slope indicates the in-context circuitry was damaged rather than preserved.","tokens_in":19401,"feed_emoji":"⚡","tokens_out":10464,"duration_ms":99449,"temperature":0.7,"pith_summary":"TACTICL is a task-aware compression method for tabular in-context-learning models such as TabPFNv2.5. The paper tries to show that, instead of distilling the model into a task-specific network and losing in-context adaptability, one can replace up to 85% of transformer layers with lightweight adapter networks trained on the downstream task and keep most of the accuracy. On 47 TabArena datasets, the resulting models run substantially faster, with median speedups of 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression for classification, and up to 5.4x on larger synthetic data. The paper also argues that the compressed model remains robust to feature-order permutation, injected irrelevant features, and label flips, and that freezing the transformer layers during optional fine-tuning keeps the in-context learning circuitry intact. If true, this gives practitioners a cheaper deployment route for tabular foundation models without forfeiting the ability to adapt to new data at inference time.","feed_headline":"Swap 85% of layers, keep tabular model accuracy","feed_subtitle":"TACTICL cuts inference cost up to 2.8x (5.4x on large data) while preserving in-context learning.","key_machinery":"The engine of the method is layer substitution with a lightweight two-layer MLP adapter (hidden width 128, GELU activation, skip connection) that is trained, before the search, to approximate the input–output mapping of the transformer layer it will replace. A greedy search then iteratively removes the layer whose substitution hurts validation AUC (or RMSE) least, relying on the locality assumption that each layer decision can be made independently of the others. Optionally, a final fine-tuning stage updates only the adapter weights with a few epochs of AdamW on a mixture of downstream data and synthetic data from the model's pretraining prior, deliberately keeping all transformer weights frozen so that learning is confined to the in-weight pathway. The paper's evidence that cheap structural proxies such as cosine similarity, CKA similarity, residual contributions, or adapter reconstruction loss fail to predict which layers can be removed is what motivates using the task's own validation metric as the guiding signal.","core_discovery":"The central claim is that transformer depth in tabular in-context-learning models is highly redundant for any given task, and that the redundancy can be exploited task-by-task without collapsing the model into a pure in-weight learner. TACTICL does this by searching over which layers to drop or substitute, replacing removed layers with small MLP adapters trained to reconstruct the removed layer's hidden-state mapping, and then optionally fine-tuning only those adapters while every original transformer weight stays frozen. The authors report that on TabPFNv2.5, substituting up to 15 of 25 layers produces negligible performance degradation, and even at 85% compression classification AUC drops by only about 4% in the median, with inference speed-ups of 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression. They interpret the robustness of compressed models to perturbations and their negative cross-dataset transfer as evidence that the model has specialized to the downstream task while retaining in-context learning: the retained transformer layers can still read new contexts, and the adapters supply task-specific in-weight knowledge.","pith_inferences":["Our inference: if the in-context/in-weight independence assumption transfers, the same layer-substitution recipe could be applied to other tabular in-context-learning models, and possibly to ICL-capable language models, with the adapter architecture and search metric adapted to each backbone.","Our inference: because the paper offers no principled rule for choosing a compression budget, one could predict the accuracy-versus-depth tradeoff from dataset meta-features (sample size, feature count, class balance) and stop the greedy search at a target cost.","Our inference: the authors note that they never directly measure the fine-tuned model's few-shot ICL performance; running that test before and after fine-tuning, with and without prior-data injection, would settle whether freezing the backbone truly protects in-context ability."],"forward_implications":["Tabular ICL models can be deployed at a fraction of their original inference cost: classification speed-ups are 1.25x, 1.63x, and 2.83x at 25%, 50%, and 85% compression, with median AUC effectively unchanged at 25–50% and roughly 4% lower at 85%.","Because the transformer backbone is frozen, the compressed model is not a one-shot distillation artifact: a new task can be handled by training new adapters and rerunning the greedy search, keeping the model's in-context capability available for later tasks.","Fine-tuning at low compression rates can recover performance lost during pruning and sometimes improve on the full model, so the same machinery doubles as a domain-adaptation mechanism rather than only a compression tool.","No universal layer-removal order exists across datasets, so task-aware selection is necessary; the greedy search recovers more than 90% of the oracle configuration's AUC for up to eight substituted layers on TabPFNv2."],"supporting_citations":[{"why":"Establishes that tabular ICL layers contribute unequally and that layer sensitivity differs from LLMs, motivating task-aware layer selection.","marker":"Balef et al., 2026"},{"why":"Earlier findings on layer contributions in tabular ICL models that the paper extends.","marker":"Balef et al., 2025"},{"why":"Introduces TabPFNv2.5, the main model compressed and evaluated, whose pretraining limits define the 47-dataset benchmark.","marker":"Grinsztajn et al., 2025"},{"why":"Introduces TabPFNv1, used for the exhaustive proof-of-concept study of layer deletion and substitution.","marker":"Hollmann et al., 2023"},{"why":"Supplies the theoretical premise that in-context and in-weight learning sub-circuits are largely independent, motivating the frozen-backbone design.","marker":"Nguyen and Nallamala, 2025"},{"why":"Provides scaling laws for forgetting during fine-tuning that motivate injecting pretraining-style data into adapter fine-tuning.","marker":"Béthune et al., 2025"},{"why":"Offers the stability metric adapted in the paper and the idea of replacing redundant blocks with lightweight adapters.","marker":"Chen et al., 2025"},{"why":"Supplies the TabArena benchmark containing the 47 datasets and the evaluation protocol used throughout the experiments.","marker":"Erickson et al., 2026"},{"why":"Reference for the embedding-similarity proxy that the paper tests and finds unreliable for predicting layer-removal impact.","marker":"Sajjad et al., 2023"},{"why":"The distillation baseline implementation is adapted from this work and serves as the main comparison for compression quality.","marker":"Tanna et al., 2026"}],"fun_headline_variants":["85% of transformer layers redundant per task","Trim 85% of tabular model layers, keep the learning","Compress tabular AI 85% with near-zero loss","Tabular ICL: swap deep layers for light adapters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a model's ability to learn from examples shown at inference time (in-context learning) and its ability to store task knowledge in its weights (in-weight learning) use separate internal circuits, so freezing all transformer layers and training only the small adapters can specialize the model to a task without damaging its in-context flexibility.","fun_headline_variants_meta":{"raw":{"variants":["85% of transformer layers redundant per task","Trim 85% of tabular model layers, keep the learning","Compress tabular AI 85% with near-zero loss","Tabular ICL: swap deep layers for light adapters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001157,"raw_usage":{"total_tokens":4795,"prompt_tokens":948,"completion_tokens":3847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3779}},"tokens_in":564,"tokens_out":3847,"duration_ms":29040,"temperature":1.0,"reasoning_tokens":3779,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:22:18.644810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a TACTICL-compressed and fine-tuned model at 85% compression, then evaluate it as a few-shot learner on a held-out tabular dataset from a different domain than the one used for compression, under the same context perturbations used in the paper (feature-order permutation, injected random features, label flips). If its performance on that new task is substantially worse than the full uncompressed model's performance under the same conditions, and the gap grows with compression depth, the claim that in-context ability remains intact would be falsified. A cheaper calculation would be to compare the fine-tuned compressed model's accuracy-versus-number-of-in-context-examples slope on an unseen task against the full model's slope; a flattened slope indicates the in-context circuitry was damaged rather than preserved.","supporting_citations":[{"cited_title":"R., Koshil, M., and Eggensperger, K","cited_arxiv_id":null,"evidence_quote":"Establishes that tabular ICL layers contribute unequally and that layer sensitivity differs from LLMs, motivating task-aware layer selection."},{"cited_title":"R., Koshil, M., and Eggensperger, K","cited_arxiv_id":null,"evidence_quote":"Earlier findings on layer contributions in tabular ICL models that the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces TabPFNv1, used for the exhaustive proof-of-concept study of layer deletion and substitution."},{"cited_title":"and Nallamala, G","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical premise that in-context and in-weight learning sub-circuits are largely independent, motivating the frozen-backbone design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers the stability metric adapted in the paper and the idea of replacing redundant blocks with lightweight adapters."},{"cited_title":"M., Salinas, D., and Hutter, F","cited_arxiv_id":null,"evidence_quote":"Supplies the TabArena benchmark containing the 47 datasets and the evaluation protocol used throughout the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reference for the embedding-similarity proxy that the paper tests and finds unreliable for predicting layer-removal impact."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The distillation baseline implementation is adapted from this work and serves as the main comparison for compression quality."}],"review_version":1}