{"id":"a6f1756c-7526-4532-a5cd-b51326a2d386","arxiv_id":"2502.03147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A retrieval module that selects the most relevant training rows for each test row lets LLMs do tabular in-context learning on large training sets, with a fitted power-law improvement.","lead":"This paper combines retrieval-augmented generation with large language models so that tabular in-context learning works with large training sets, not just a few examples. Across 69 public datasets, the approach improves accuracy and scales with data, while still falling short of the best tuned numeric models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling evidence conflates retrieval with LLM effects: Figure 1 reports L(D) only for RAG+Phi3-GTL, with no RAG+kNN or RAG+LR control across D, so the reported power law may be a property of the retrieval module rather than of LLM-based TabICL.","rationale":"Reading in good faith: the paper makes a concrete, falsifiable claim—median held-out error follows a power law in training-set size under RAG+Phi3-GTL. My concern is not about honesty; it is a missing experimental control in the chain from observation to attribution. Section 5.2's Figure 1 establishes only that TabRAG beats Random for Phi3-GTL and that the RAG curve decreases with D. It does not establish that the LLM is the consumer responsible for the decrease. Since the LLM's weights are fixed and its context is bounded, the retrieval module alone could account for the trend. This is the most load-bearing point because the paper's framing—scalable TabICL as a language-interface property—depends on it. The reader's weakest assumptions were the local-support premise and dataset leakage. I agree leakage is serious, but it is an auditability issue fixable by releasing the dataset list; the missing control requires new computation and directly targets the quantitative central claim. The paper's decision-boundary analyses and case studies (Section 5.6, Appendix E) provide some independent evidence that the LLM does something different from kNN, so the control might well confirm a real effect; the point is that this evidence is qualitative and does not quantify scaling. My recommendation is UNCHANGED: the reader's CONDITIONAL verdict already captures that artifacts and reframing are needed. I would add one explicit condition: demonstrate that the fitted alpha for RAG+Phi3-GTL is significantly larger than for RAG+KNN/RAG+LR on the same retrieved contexts, or soften the claim that the LLM interface is what scales.","tokens_in":28850,"tokens_out":9210,"duration_ms":83446,"concrete_test":"Run the Figure 1 scaling sweep for RAG+KNN and RAG+LR on the same 69 held-out datasets, with D in {400, 2000, 10000, 50000}, using the same TabRAG retrieval, the same N_C <= 128, and the same median-error aggregation. Fit L(D) = (Dc/D)^alpha separately for each consumer and compute bootstrap confidence intervals on alpha. If the RAG+KNN or RAG+LR exponents overlap with the reported alpha ~ 0.102 / 0.053, the power law is a retrieval artifact and the claim of scalable LLM-based TabICL must be weakened; if the LLM's exponent is steeper with non-overlapping CIs, the specific LLM contribution is supported. Also report per-dataset fits to check whether the median hides a few large-D outliers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is the power-law scaling L(D)=(Dc/D)^alpha in Section 5.2/Figure 1, with alpha ~ 0.102 (classification) and ~ 0.053 (regression), presented as evidence that LLM-based TabICL 'effectively leverages larger datasets.' The inference-time LLM is frozen and the context size is capped at 128 instances (Appendix D.5), so when D grows from 400 to 50,000 the only change is the composition of the TabRAG-retrieved contexts. Figure 1 compares Random versus TabRAG for Phi3-GTL; it never shows the same D-scaling sweep for a non-parametric consumer of those identical contexts. RAG+KNN appears only in Table 4, at a single full training size. Because kNN-style error naturally decreases as the candidate pool grows (neighbors become closer to the query), the observed declines in median L(D) could be entirely a geometric property of the retrieval module, not of the LLM's in-context reasoning. The fitted exponents are small, consistent with high-dimensional nearest-neighbor convergence, and no confidence intervals or per-dataset fits are reported. This gap matters because the abstract and Section 4 attribute scalable TabICL to the LLM-based interface; if RAG+KNN reproduces the same scaling, the LLM component is not necessary for the headline result. This is distinct from, and logically prior to, the Appendix B.1 leakage auditability problem: even under a perfectly clean split, the attribution to the text-interface model is unsupported without the control.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes retrieval-augmented LLMs for tabular in-context learning (TabICL), decoupling context selection (TabRAG) from LLM prediction, and aligning the LLM with retrieval patterns via retrieval-guided instruction tuning. The authors post-train Phi-3 with a generative tabular learning objective and evaluate on 29 classification and 40 regression datasets. They report that RAG+Phi3-GTL outperforms random-context Phi3-GTL and RAG+KNN on nearly all datasets, that median error follows a power law L(D) = (Dc/D)^alpha with alpha ~ 0.102 (classification) and ~0.053 (regression), and that LLM-based TabICL adds ensemble diversity and excels on certain datasets, while still lagging behind well-tuned numeric models overall.","tokens_in":29156,"tokens_out":3513,"duration_ms":32602,"significance":"If the scaling claim is established, the paper would show that LLM-based TabICL can leverage full-sized training sets through retrieval, a meaningful step beyond the few-shot limitation of prior LLM-based TabICL. The benchmark breadth (69 datasets), the inclusion of tuned GBDT and neural baselines, the RAG+X controls, and the detailed per-dataset case studies are strengths. The retrieval-guided post-training and the decision-boundary analysis are useful contributions. However, the headline scaling-law claim is not yet supported by the evidence as presented, because the D-sweep lacks a non-LLM control that consumes the identical retrieved contexts, and the power-law parameters are fitted to the same data they are used to describe. The data-leakage auditability issue also needs resolution before the held-out generalization claims can be fully credited.","major_comments":[{"comment":"The power-law L(D) is reported only for RAG+Phi3-GTL, with no RAG+KNN or RAG+LR control evaluated across the same D sweep. Since the LLM context is capped at 128 instances (Appendix D.5) and the frozen LLM receives the same context size at every D, the observed decline in median error as D grows from 400 to 50,000 could be entirely a property of the retrieval pool (nearest neighbors become closer as the candidate pool grows), not of LLM-based TabICL. This is load-bearing for the abstract and Section 4's attribution of scalability to the LLM-based interface. Please add RAG+KNN (and ideally RAG+LR) scaling curves to Figure 1, or otherwise explicitly separate the retrieval-pool effect from the LLM's in-context reasoning effect.","section":"Section 5.2, Figure 1"},{"comment":"The power-law claims rest on four median data points (D=400, 2000, 10000, 50000) from which both alpha and Dc are estimated, with no confidence intervals, no per-dataset fits, and no out-of-sample predictive check. As presented, L(D) is a fitted curve rather than a scaling law, and the statement that 'the expected prediction error approaches zero' extrapolates beyond the fitted range without uncertainty quantification. Please report uncertainties, show per-dataset fits, and test the law by predicting a held-out D value; also temper the extrapolative claim or justify it with a theoretical argument.","section":"Section 5.2, Eq. L(D) = (Dc/D)^alpha"},{"comment":"The claim that the 319 post-training datasets and the 69 held-out datasets are disjoint relies on a manual filtering process, but no list of post-training dataset identifiers or the overlap-filtering procedure is provided. Because several held-out datasets come from the GTL benchmark (Wen et al., 2024), which itself used a 350-dataset pool, the leakage risk is non-trivial and not auditable. Please release the full list of post-training dataset identifiers and the exact filtering steps, or otherwise make the split independently verifiable.","section":"Appendix B.1 / Section 5.1"},{"comment":"The method is built on the stated assumption that a limited per-test-instance support set can suffice for accurate prediction. As the paper's own case study R-25 (Appendix E) shows, tasks that require global structure or feature combinations (e.g., day-of-week and year interactions) are not recoverable by the default local retrieval policy. This scope limitation directly bounds the 'any data size' claim in the abstract and should be stated prominently in the abstract or introduction, not only in the limitations/case-study sections.","section":"Section 4, Eq. (3)"}],"minor_comments":[{"comment":"The abstract says 'any data size,' but the experiments cap training sets at 100,000 samples (Appendix B.2) and the LLM context at 128 instances; please clarify the intended scope.","section":"Abstract / Appendix B.2"},{"comment":"The dashed lines are said to represent a power-law relationship, but the fitted parameters and the metric for L are not given in the caption; please report the fitted equation in the caption or in the text of Section 5.2.","section":"Figure 1"},{"comment":"The text uses 'over 300 public datasets,' '319 public datasets,' and '146 classification and 173 regression datasets' in close proximity; please align the terminology and state explicitly that 146+173 equals the 319 total.","section":"Appendix A.2 / Appendix C"},{"comment":"The sentence about adopting Phi-3 and extending the effective context length from 4K to 128K conflates the base model's architecture with the post-training procedure; please specify which component (base LLM, post-training sequence length, or retrieval setup) determines the effective context.","section":"Section 5.1"},{"comment":"The citation 'Dong et al.' lacks a year and full venue information; please add the complete reference.","section":"Related Work, ICL paragraph"},{"comment":"Several numerical entries in Table 4 show an unusual spacing between the integer and fractional parts (e.g., '0 .960'); please reformat the table for clarity.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core retrieval-augmented TabICL framework and its broad empirical comparison are valuable, and the RAG+Phi3-GTL versus RAG+KNN comparison at full data size supports the claim that retrieval plus LLM reasoning adds value. The main risk is the scaling-law attribution: without a non-LLM control across D, the headline claim may overstate what is currently demonstrated. Also, the data-leakage auditability issue should be addressed before publication, especially since the post-training set comes from a prior benchmark with overlapping provenance. I would support acceptance after the scaling analysis is reworked and the dataset split is made auditable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about LLMs on tabular data. The core idea is clean: decouple a retrieval module from the LLM, so the LLM only ever sees a fixed-size context of retrieved neighbors, and post-train the LLM to work with that retrieval pattern. That is a sensible move, and the experiments mostly back it up. Table 4 shows RAG+Phi3-GTL beating both the random-context Phi3-GTL and RAG+KNN on nearly every dataset, and the ablations in Figure 6 show each retrieval component matters. The paper is also honest that the whole approach still trails well-tuned numeric models in aggregate, which I appreciate.\n\nThe load-bearing soft spot is the scaling claim. Figure 1 sweeps training-set size D for Random versus RAG, but only for Phi3-GTL. Because the LLM context is capped at 128 instances, the only thing that changes with D is the composition of the retrieved candidates. Error would drop for any model consuming those candidates—kNN error naturally falls as the candidate pool grows. Without a control like RAG+KNN or even RAG+linear regression swept over the same D values, the reported power law L(D)=(Dc/D)^alpha cannot be attributed to the LLM's in-context reasoning. It might be a property of the retrieval module alone. That gap is logically prior to the leakage question, and it undermines the 'scalable TabICL' framing.\n\nSecond, the power law is a fitted curve on the same data it describes, with no confidence intervals, no per-dataset exponents, and no prediction for held-out D values. Calling it 'scaling behavior' is fine; calling it a law overstates the evidence. Third, the manual filtering of the 319 post-training datasets is not auditable because no list is given. Fourth, there is no code, model, or data release, so the numbers cannot be checked as stated.\n\nAll that said, the paper does not deserve a desk rejection. The central claim—that retrieval substantially improves LLM-based TabICL—is supported, and the retrieval-guided instruction tuning is a useful contribution even without the scaling law. A serious referee should ask for the missing RAG+KNN control across D, a reframing of the power-law as a fitted trend, and the artifact release. I would send it to review, conditionally.","headline":"Retrieval helps—the main comparison holds—but the scaling-law claim is overreaching without a non-LLM control swept over D.","tokens_in":29738,"tokens_out":2107,"would_cite":true,"duration_ms":22147,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval-augmented LLMs make tabular in-context learning scalable to any training-set size, with median error falling as a power law.","keywords":["tabular in-context learning","retrieval-augmented generation","large language models","tabular data","scaling behavior","power-law scaling","any-shot learning","retrieval-guided instruction tuning"],"falsifier":"Find a tabular task whose label depends on global structure rather than local neighbors—for example, predicting a row's rank in the full training distribution—and show that increasing training set size $D$ no longer lowers the median error along the reported power law $L(D)=(D_c/D)^\\alpha$. A second, simpler check is to audit the overlap between the post-training and held-out datasets: if any held-out dataset appears in the training list, the power-law gains could be partly memorization rather than scalable learning.","tokens_in":28617,"feed_emoji":"📊","tokens_out":6590,"duration_ms":55699,"temperature":0.7,"pith_summary":"The paper tries to remove the few-shot ceiling on LLM-based in-context learning for tabular data. Because each table row burns many tokens when written as text, earlier LLM approaches could only put a handful of examples in context; this work claims that a retrieval module that selects the most relevant rows for each test instance, combined with instruction tuning that mimics that selection, lets the same text interface use training sets of any size. On 29 classification and 40 regression datasets the median error falls as a power law of training-set size, with exponent near 0.102 for classification and 0.053 for regression. If true, a single chat-style model could serve as a universal, accessible predictor for tabular data without per-dataset training, while still improving as more data arrives.","feed_headline":"Retrieval gives LLMs a power-law scaling law on tabular data","feed_subtitle":"Median error drops as training data grows, so text-based tabular models can escape the few-shot token limit.","key_machinery":"The load-bearing object is the non-parametric retrieval policy TabRAG, which scores feature-wise distances between the test row and every training row after quantile normalization, weights each feature by linear (Pearson correlation) and non-linear (single-feature tree score) importance, and aggregates these into a sample distance with a weighted L2 norm. It then feeds a fixed quota of nearest neighbors, up to 128 rows, to an LLM whose post-training objective was changed to predict from contexts produced by that same policy rather than from randomly selected rows. This alignment of retrieval and instruction tuning is what lets the text-based model convert more training data into better predictions instead of being overwhelmed by token limits.","core_discovery":"The central claim is that scalable tabular in-context learning can be decoupled into a retrieval step that picks a fixed-size support set for each test row and an LLM step that predicts from that support set alone. The paper formalizes this as $C_{x_{\\mathrm{test}}}=\\mathrm{TabRAG}(x_{\\mathrm{test}}, D_{\\mathrm{train}})$ and shows that when a Phi-3-based LLM is post-trained with contexts generated by this retrieval policy, its median error on held-out datasets follows $L(D)=(D_c/D)^\\alpha$, with $\\alpha\\approx0.102$ for AUROC-based classification error and $\\alpha\\approx0.053$ for regression NMAE. It further reports that the retrieval-augmented model beats its random-context and KNN ablated variants, outperforms the leading numeric TabICL baseline on roughly a fifth of the datasets, contributes measurable diversity to ensembles, and still trails well-tuned numeric models on average.","pith_inferences":["A corollary the paper leaves implicit is that 'retrieval engineering'—choosing feature weights, normalization, and similarity constraints per dataset—becomes a practical skill analogous to prompt engineering, and the case studies give a direct recipe for testing this on new datasets.","The power-law exponents imply strongly diminishing returns: the classification exponent of about 0.102 means multiplying training data by 1000 cuts median error by roughly half, so the practical value sits in the small-to-mid data range rather than in massive datasets.","A testable extension is to post-train the LLM on synthetic data with diverse feature distributions and feature-label interactions, mirroring TabPFN's recipe, and measure whether the held-out error gap closes.","One could also test whether retrieval-plus-LLM serves as a drop-in component for conversational data analysis by checking whether the same retrieved support set improves answer faithfulness on natural-language questions about a table."],"forward_implications":["LLM-based TabICL can escape the few-shot regime: a fixed-size retrieved context gives a forward pass per test row even when the training set has 100,000 rows.","More training data will keep helping the text interface, with median error improving roughly as a power law, until dataset-specific limits are reached.","Retrieval policy quality becomes the main lever: simple changes such as swapping normalization or adding a domain-informed constraint cut regression error by 14 percent to 59 percent on the case-study datasets.","Because the model's decision boundaries differ from numeric models, adding it to an ensemble of TabPFN-v2 and gradient-boosted trees improves aggregate robustness.","Text representation also captures discrete structure: the model achieves zero NMAE on an integer regression task where numeric models emit float outputs."],"supporting_citations":[{"why":"Supplies the generative tabular learning post-training objective and dataset collection that this paper extends to retrieval-guided training.","marker":"Wen et al., 2024"},{"why":"Provides TabPFN-v1 and TabPFN-v2, the numeric TabICL baselines the paper must beat and whose synthetic-prior training it contrasts with real-data post-training.","marker":"Hollmann et al., 2023; 2025"},{"why":"Provides TabR and the tuned neural/tree baseline protocol and benchmark splits reused in evaluation.","marker":"Gorishniy et al., 2024"},{"why":"Supplies the tree-versus-deep benchmark, including cat and num series, that forms much of the held-out evaluation.","marker":"Grinsztajn et al., 2022"},{"why":"Supplies the retrieval-augmented generation idea that the paper adapts to tabular data.","marker":"Lewis et al., 2020"},{"why":"Supplies the k-nearest-neighbor assumption that a limited support set can govern an accurate prediction.","marker":"Fix & Hodges, 1951; Cover & Hart, 1967"},{"why":"Supplies Phi-3, the base LLM whose long context window enables the longer-context TabICL comparisons.","marker":"Abdin et al., 2024"},{"why":"Supplies Tabula-8B, the alternative LLM-based TabICL baseline compared in the appendix.","marker":"Gardner et al., 2024"}],"fun_headline_variants":["Retrieval frees tabular LLMs from the few-shot token ceiling","Scaling law: retrieval-augmented LLMs improve on tabular data","Retrieval lets tabular LLMs scale beyond the few-shot limit","Power-law gains: retrieval makes tabular LLMs scalable","Retrieval rescues tabular ICL from the few-shot token limit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small set of retrieved neighbors can carry enough information to predict any given test row, and the evaluation assumes the manual filtering fully removed overlap between the post-training dataset and the held-out datasets.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval frees tabular LLMs from the few-shot token ceiling","Scaling law: retrieval-augmented LLMs improve on tabular data","Retrieval lets tabular LLMs scale beyond the few-shot limit","Power-law gains: retrieval makes tabular LLMs scalable","Retrieval rescues tabular ICL from the few-shot token limit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3752,"prompt_tokens":964,"completion_tokens":2788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2692}},"tokens_in":580,"tokens_out":2788,"duration_ms":17953,"temperature":1.0,"reasoning_tokens":2692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:43:36.516160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a tabular task whose label depends on global structure rather than local neighbors—for example, predicting a row's rank in the full training distribution—and show that increasing training set size $D$ no longer lowers the median error along the reported power law $L(D)=(D_c/D)^\\alpha$. A second, simpler check is to audit the overlap between the post-training and held-out datasets: if any held-out dataset appears in the training list, the power-law gains could be partly memorization rather than scalable learning.","supporting_citations":[{"cited_title":"FTT is a widely recognized neural model for tabular learning","cited_arxiv_id":null,"evidence_quote":"Provides TabR and the tuned neural/tree baseline protocol and benchmark splits reused in evaluation."}],"review_version":1}