{"id":"f62316ad-55d5-4b73-9f86-af23faa30082","arxiv_id":"2501.14693","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A systematic hyperparameter study of table instruction tuning finds that small learning rates and 2,600 examples suffice, yielding TAMA, an 8B model competitive with GPT-3.5/GPT-4 on table benchmarks while keeping general abilities.","lead":"The paper shows that instruction-tuning LLMs for table tasks with a small learning rate and only a few hundred examples per dataset preserves general abilities while still boosting table skills, and releases a tuned 8B model called TAMA. A generalist reader might care because it suggests cheaper, faster development of specialized table models without wrecking their general knowledge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GPT-3.5/GPT-4 comparisons in Table 7 are mostly cited scores from other papers, not controlled evaluations; TAMA's 'surpassing GPT' claim rests on heterogeneous baselines that should be re-run identically.","rationale":"The reader's conditional verdict already flags 'partially incomparable GPT baselines' as a reason not to accept outright, so my concern is not entirely new. However, the reader's weakest_assumption focuses on the randomness and representativeness of the 200-example-per-dataset training subset. That is a legitimate statistical concern: no seeds or variance are reported, and the recipe could behave differently under another draw. But the paper's Figure 2 and Appendix D.4/D.5 show the diminishing-returns trend consistently across multiple learning rates, multiple base models, and full-parameter, LoRA, and QLoRA setups, which gives me moderate confidence that the qualitative 'fewer instances' claim is robust. The GPT comparison, by contrast, is the direct evidence for the abstract's flashiest claim, and the cited-score issue is visible in the paper's own footnotes. A controlled re-evaluation of GPT models is the single check most likely to change the headline conclusion. The paper's other strengths are real: TAMA is released, the hyperparameter sweeps are extensive, and the failure analysis of existing table LLMs is useful. For those reasons, I do not think the verdict should move from CONDITIONAL; the paper should require the GPT baselines to be re-run under identical conditions before the superiority claim is accepted.","tokens_in":32766,"tokens_out":11065,"duration_ms":94014,"concrete_test":"Re-run GPT-3.5-turbo and GPT-4-turbo on the exact test instances used for TAMA in Table 7 (or on the same 500-instance subsample described in footnote 2, if budget-constrained), using the same prompts, table serialization, and decoding parameters as TAMA (temperature 0, top_p 0.95 per Appendix B.2). Then recompute per-dataset scores and the win/loss counts in Section 4.2. If the re-evaluated GPT scores differ by more than the current TAMA-vs-GPT margins on any dataset, the abstract's 'surpassing GPT' claim is not supported by controlled evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline claim that TAMA is 'on par with, or surpassing GPT-3.5 and GPT-4 on table tasks' rests on Table 7, but that table's GPT columns are not a controlled comparison. The dagger footnote attributes 11 of the 13 GPT-3.5 and GPT-4 scores to Gou et al. (2023), Srivastava et al. (2024), and Zhang et al. (2024a); only AIT-QA and InfoTabs appear to be the authors' own GPT runs. These cited scores come from different papers using different prompts, decoding settings, and possibly different test splits. Footnote 2 further states that budget limits led the authors to uniformly sample 500 data points from the original test set for GPT prompting, while the paper does not state that TAMA and GPT are evaluated on the same instances or that TAMA's reported numbers use the same 500-point subsample. Section 4.2's conclusion that TAMA surpasses GPT-3.5 on almost every table task except KVRET and WikiTQ therefore compares TAMA against heterogeneous, non-reproduced baselines. If those baselines were re-run under identical evaluation conditions, the win/loss counts supporting the abstract could change materially. This is a load-bearing concern because the 'surpassing GPT' comparison is a central advertised outcome, even though the paper's data-efficiency findings may still hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies instruction tuning of LLMs for table understanding, focusing on hyperparameter choices rather than data scale. It first evaluates three open-source table LLMs (TableLLaMA, TableLLM, TableBenchLLM) on out-of-domain table understanding (Table-Syn) and general benchmarks (IFEval, MMLU, MMLU Pro, AI2ARC, GPQA), finding large drops relative to their base models. It then performs a systematic exploration on LLaMA 3.1 8B Instruct over learning rate, number of training examples, epochs, multi-task training, and additional setups (LoRA/QLoRA, other base models), concluding that a small learning rate (around 1e-6), two epochs, and 200 random examples per dataset (2,600 total) yield strong table understanding while preserving general capabilities. The resulting model, TAMA, is reported to match or exceed GPT-3.5 and GPT-4 on several table benchmarks while retaining MMLU, IFEval, and AI2ARC performance. The paper also releases code, models, and data.","tokens_in":33090,"tokens_out":7664,"duration_ms":64868,"significance":"If the central recipe is robust, the paper makes a practically valuable contribution: it suggests that table instruction tuning can be far more data-efficient than the multi-million-example corpora used by prior systems (TableLLaMA, TableBenchLLM), and that careful hyperparameter selection can avoid the general-capability degradation reported for existing table LLMs. The exploration is unusually broad for this area, spanning full-parameter tuning across five model families (Table 14), LoRA/QLoRA (Tables 16-17), varying training sizes, and epoch counts, with consistent diminishing-returns trends across settings. The open release of code, model weights, and the 2.6K instruction dataset increases the reproducibility value of the work. However, the two headline claims—that TAMA surpasses GPT-3.5/GPT-4 and that the 200-example recipe is reliably optimal—are weakened by uncontrolled baseline comparisons and by selection on test benchmarks, as detailed in the major comments.","major_comments":[{"comment":"The abstract's claim that TAMA is 'on par with, or surpassing GPT-3.5 and GPT-4 on table tasks' is supported only by Table 7, whose GPT columns are a mixture of cited scores: the dagger footnote assigns 11 of the 13 GPT-3.5 and GPT-4 values to Gou et al. (2023), Srivastava et al. (2024), and Zhang et al. (2024a). These scores come from different papers with different prompts, decoding settings, and possibly different test splits. Footnote 2 states that 'Due to budget limit for prompting GPT models, we uniformly sample 500 data points from the original test set as our test set,' but the paper does not state that TAMA's numbers are computed on the same 500-instance subset, and for the cited datasets no matched subsampling is described. The win/loss summary in Section 4.2 is therefore not a controlled comparison. To support the advertised comparison to GPT-3.5/GPT-4, the authors should re-run the proprietary baselines under identical evaluation conditions (same instances, prompts, decoding, and scoring) for all datasets, or explicitly restrict the claim to the two datasets (AIT-QA and InfoTabs) for which they appear to have run GPT themselves.","section":"Section 4.2, Table 7"},{"comment":"The 'hindsight analysis' validates the selected hyperparameters by training with learning rates from 1.0e-7 to 1.0e-5 and one to six epochs, and then aggregating performance on 'all the datasets listed in Table 6' (Section 4.3, Figure 3a). These are the same test and development sets whose scores are later reported as TAMA's final results in Table 7. Selecting the configuration that maximizes performance on these benchmarks means the reported TAMA numbers are not clean held-out estimates; the 'hindsight' framing does not remove the selection. Additionally, Figure 3a sums raw scores across tasks with incompatible metrics (accuracy, BLEU, ROUGE-L, F1), which is not a meaningful aggregate. The paper should reserve a held-out set of table benchmarks for final evaluation, or present Section 4.3 as an exploratory diagnostic and avoid using it as evidence that the selected hyperparameters are optimal.","section":"Section 4.3, Figure 3"},{"comment":"The data-efficiency claim rests on 200 randomly selected instruction pairs per dataset, but no random seeds or repeated draws are reported. Section 3.2's learning curves (Figure 2) and the final TAMA results (Section 4.1, Table 7) are based on single realizations of the random subset, so the large gains on some test sets (e.g., HybridQA 60.86 vs. base 32.83) could partly reflect a favorable draw rather than a robust property of the recipe. Since the paper's central recommendation is that 200 examples per dataset suffice, the authors should report mean and variance over at least three seeds, or provide a bootstrap over the training pool, to establish that the recipe transfers to new random subsets and new datasets.","section":"Section 3.2, Section 4.1 (Table 6)"}],"minor_comments":[{"comment":"The sentence 'We post our updates in Table 1.' is out of place in an abstract; it should be removed or moved to the appendix.","section":"Abstract"},{"comment":"The text states 'TAMA achieves 52.81' for WikiTQ, but Table 7 reports 52.88; the numbers should be made consistent.","section":"Section 4.2"},{"comment":"The phrase 'the learning rate is around 1.0e-6 and 5.0e-7' should read 'around 1.0e-6 or 5.0e-7' to avoid ambiguity about whether both values are recommended simultaneously.","section":"Section 3.2"},{"comment":"The learning-curve plots do not show numeric values or error bars, which makes it difficult to verify the claim that performance improvement beyond 200 examples is 'marginal'; a companion table with the plotted values would strengthen the exposition.","section":"Figure 2"},{"comment":"The explanation that table training improves STEM performance in MMLU because table tasks involve math reasoning is plausible but speculative; it should be clearly framed as a hypothesis rather than a demonstrated conclusion.","section":"Section 4.3, Table 10"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a CL journal, but the advertised comparison to GPT-3.5/GPT-4 is not backed by controlled evaluation and should either be redone under matched conditions or substantially softened. The hindsight analysis should be repositioned as exploratory, and variance estimates are needed for the 200-example recipe. If these load-bearing issues are addressed, the paper's practical message about data-efficient table instruction tuning could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core empirical finding is real: with a learning rate around 1e-6, two epochs, and two hundred randomly chosen examples per dataset, you can instruction-tune an 8B model for table tasks without nuking its general knowledge. The paper's most useful contribution is the systematic sweep over learning rate, data size, epochs, multi-task synergy, and training regimes (full-parameter, LoRA, QLoRA) across five model families. Those trends show diminishing returns consistently, and the released TAMA weights and data are open. That is genuine, citable evidence for data-efficient table tuning.\n\nThe soft spots are mostly around the advertised 'surpasses GPT' claim. Table 7's GPT-3.5/4 numbers are almost all cited from other papers with different prompts and evaluation settings; only AIT-QA and InfoTabs appear to be the authors' own runs. Footnote 2 says GPT prompting used a uniform 500-point subsample of each test set due to budget, but the paper never states that TAMA's reported numbers use the same 500 points or that TAMA and GPT see identical instances. So the win/loss counts comparing TAMA to GPT are not controlled, and the abstract overstates what is actually shown. That is load-bearing for the headline, but it does not undermine the hyperparameter findings.\n\nTwo smaller issues. First, the random selection of 200 examples per dataset is run once with no seeds or variance estimates. The HybridQA jump from 32.83 to 60.86 could partly be a lucky draw; the paper acknowledges random sampling as a limitation, but the recipe's transferability would be much more convincing with a few seeds. Second, Section 4.3 calls itself a 'hindsight analysis' and validates the chosen hyperparameters on aggregate test performance from the same benchmarks reported in Table 7. That makes the final TAMA scores a selection on the test set, not a clean out-of-sample result.\n\nWho is this for? People building table-specialized models or doing data-efficient instruction tuning, and anyone thinking about how to evaluate specialized models against API baselines. It deserves a serious referee. The right revision would add seeds, restrict the GPT comparison to identically-evaluated runs, and pick hyperparameters on a genuinely held-out set.","headline":"The data-efficiency recipe is credible and useful; the 'surpasses GPT' headline rests on uncontrolled cited baselines and a 500-point subsample ambiguity.","tokens_in":33591,"tokens_out":3047,"would_cite":true,"duration_ms":25383,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Smaller learning rate and 2,600 examples produce a table LLM that beats GPT-3.5 and often GPT-4.","keywords":["table understanding","instruction tuning","learning rate","data efficiency","LLM fine-tuning","out-of-domain generalization","general capabilities","TAMA"],"falsifier":"Train TAMA ten times, each with a different random seed for the 200-example draw, and compare the spread on HybridQA (reported 60.86 versus 32.83 for the base model). If some draws fall well below that figure or the variance exceeds roughly five accuracy points, the data-efficiency claim rests on the particular sample rather than the recipe.","tokens_in":32576,"feed_emoji":"📊","tokens_out":9336,"duration_ms":70833,"temperature":0.7,"pith_summary":"The paper argues that table instruction tuning has been tuned with the wrong dials: prior open table LLMs train on hundreds of thousands to millions of examples at a large learning rate, and as a result they lose out-of-domain table skill and general ability. The paper claims that a much smaller learning rate near $10^{-6}$, two epochs, and only 200 randomly sampled instruction pairs per dataset (2,600 pairs total) produce stronger table understanding while preserving the base model's general capabilities. Applied to LLaMA 3.1 8B Instruct, that recipe yields TAMA, which the paper reports matches or beats GPT-3.5 on most table benchmarks and exceeds GPT-4 on several, while keeping MMLU and IFEval near base levels. The payoff the authors aim at is lower annotation cost and more efficient model development through careful hyperparameter selection.","feed_headline":"Training on 2,600 examples, TAMA beats GPT-3.5 on table tasks","feed_subtitle":"A learning rate of 1e-6 and two epochs keep general skills intact while sharpening table reasoning.","key_machinery":"The central object is a training recipe rather than a new architecture: full-parameter instruction tuning of LLaMA 3.1 8B Instruct at learning rate $10^{-6}$ for two epochs on 200 randomly sampled instruction pairs from each of thirteen table datasets, 2,600 pairs in total. The machinery that justifies it is a systematic sweep over learning rate ($10^{-7}$ to $10^{-5}$), number of examples (30 to 1,500), and epochs (1 to 6), evaluated on table tasks and on general benchmarks, plus a hindsight grid over the final setup confirming that the chosen hyperparameters sit on the best-performing ridge. The paper's interpretation is that the base model already holds most of the needed table reasoning, so a few hundred examples at a small learning rate sharpen that ability without erasing general knowledge.","core_discovery":"Existing open table LLMs—TableLLaMA, TableLLM, and TableBenchLLM—all use a learning rate of $2 \\times 10^{-5}$ and training sets of 20K to 2M examples, and the paper measures that every one of them degrades on held-out table tasks such as Table-Syn and on general benchmarks such as MMLU, AI2ARC, and IFEval compared with its base model. Through full-parameter tuning experiments on LLaMA 3.1 8B Instruct with data from FeTaQA, HiTab, and TabFact, the paper finds a learning-rate sweet spot near $10^{-6}$ to $5 \\times 10^{-7}$: larger rates near $10^{-5}$ damage general capabilities, while smaller rates near $10^{-7}$ leave table gains on the table. Performance rises sharply over the first 200 examples and then plateaus, and extra epochs add little. Combining these findings, the paper trains TAMA for two epochs at learning rate $10^{-6}$ on 200 randomly selected instruction pairs from each of thirteen datasets, and reports that this 8B model surpasses GPT-3.5 on nearly every table benchmark, beats GPT-4 on WikiSQL, HybridQA, InfoTabs, and FEVEROUS, and keeps MMLU, AI2ARC, and most of IFEval at or above base-model levels.","pith_inferences":["This inference is mine: the 200-example-per-dataset recipe should transfer to adjacent structured-data tasks such as text-to-SQL and document question answering, but only if the base model already has strong instruction following; for weaker base models the required sample size is likely larger.","This inference is mine: because the paper reports a single random draw of 200 examples per dataset without seeds or variance estimates, the headline numbers should be read as point estimates; repeating the draw across seeds is the direct test of whether the recipe or the particular sample carries the result.","This inference is mine: the learning-rate sweet spot probably shifts with model scale and with LoRA/QLoRA, so a practical recipe should re-sweep the learning rate whenever the tuning method changes.","This inference is mine: the paper's observation that MMLU STEM scores improve after table tuning suggests table instruction data may double as targeted math-reasoning training; a testable extension is to measure STEM gains on a separate math benchmark."],"forward_implications":["Table instruction tuning can be done with roughly 2,600 instruction pairs rather than 20K to 2M, cutting annotation cost by orders of magnitude.","A learning rate near $10^{-6}$, not $2 \\times 10^{-5}$, should become the default starting point for full-parameter table tuning of instruct-tuned 7B/8B models; the same qualitative pattern holds across Llama 2, Llama 3.1, Qwen 2.5, Mistral, and Phi in the paper's appendix.","Specializing a model on tables need not trade away general knowledge: TAMA keeps MMLU and AI2ARC at or above base level and retains most IFEval instruction-following, so table-specialized assistants remain usable for general requests.","Out-of-domain table generalization is achievable with this recipe, as TAMA outperforms GPT-3.5 on two synthesized table benchmarks whose distributions differ from all training data.","Existing table LLMs' reported drops on general benchmarks are largely attributable to an overly large learning rate combined with long training, not to table tuning itself."],"supporting_citations":[{"why":"Supplies TableLLaMA, the main baseline whose 2M examples, $2 \\times 10^{-5}$ learning rate, and six epochs define the recipe that TAMA overturns.","marker":"(Zhang et al., 2024a)"},{"why":"Supplies TableLLM, a second baseline with 309K examples and the same large learning rate that the paper evaluates.","marker":"(Zhang et al., 2024b)"},{"why":"Supplies TableBenchLLM (20K examples, $2 \\times 10^{-5}$) as a baseline and the synthesized out-of-domain benchmark used for TAMA.","marker":"(Wu et al., 2024)"},{"why":"Defines LLaMA 3.1 8B Instruct, the base model TAMA starts from and whose general capabilities the small-learning-rate recipe preserves.","marker":"(Dubey et al., 2024)"},{"why":"FeTaQA is one of the three datasets in the hyperparameter sweep and a training source for TAMA.","marker":"(Nan et al., 2022)"},{"why":"HiTab is one of the three datasets in the hyperparameter sweep and a training source for TAMA.","marker":"(Cheng et al., 2022)"},{"why":"TabFact is one of the three datasets in the hyperparameter sweep and a training source for TAMA.","marker":"(Chen et al., 2019)"},{"why":"Supplies Table-Syn, the out-of-domain table benchmark where existing table LLMs drop and TAMA surpasses GPT-3.5.","marker":"(Li et al., 2023)"},{"why":"MMLU is the main general-knowledge benchmark used to measure whether tuning preserves general capabilities.","marker":"(Hendrycks et al., 2021)"},{"why":"IFEval is the instruction-following benchmark that quantifies whether table tuning preserves the model's ability to follow format constraints.","marker":"(Zhou et al., 2023)"}],"fun_headline_variants":["TAMA: 2,600 examples and a tiny LR beat GPT-3.5","Low LR, fewer examples: TAMA outdoes GPT-3.5 on tables","TAMA's edge: 1e-6 LR and 2,600 examples trump GPT-3.5","Forget big data: TAMA's low LR keeps smarts and beats GPT-3.5","Small LR, small data: TAMA tops GPT-3.5 without losing skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recipe assumes that 200 randomly drawn instruction pairs from each dataset are representative enough that the large reported gains are not a favorable draw; the paper reports no repeated sampling, seeds, or variance estimates.","fun_headline_variants_meta":{"raw":{"variants":["TAMA: 2,600 examples and a tiny LR beat GPT-3.5","Low LR, fewer examples: TAMA outdoes GPT-3.5 on tables","TAMA's edge: 1e-6 LR and 2,600 examples trump GPT-3.5","Forget big data: TAMA's low LR keeps smarts and beats GPT-3.5","Small LR, small data: TAMA tops GPT-3.5 without losing skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3756,"prompt_tokens":1048,"completion_tokens":2708,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":2587}},"tokens_in":664,"tokens_out":2708,"duration_ms":19290,"temperature":1.0,"reasoning_tokens":2587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:53:51.252072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train TAMA ten times, each with a different random seed for the 200-example draw, and compare the spread on HybridQA (reported 60.86 versus 32.83 for the base model). If some draws fall well below that figure or the variance exceeds roughly five accuracy points, the data-efficiency claim rests on the particular sample rather than the recipe.","supporting_citations":[],"review_version":1}