{"id":"311dbfe2-065a-443b-9a70-770c3e67c1f8","arxiv_id":"2412.15594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A template-plus-LLM pipeline generates 23,000 tabular math word problems with illustrative solutions, and fine-tuning on them raises the accuracy of 7B-8B LLMs on TabMWP by roughly four percentage points.","lead":"This paper presents TeLL, a pipeline that generates math word problems with tables by combining question templates with LLM paraphrasing. The resulting dataset, TabMWP-TeLL, improves the accuracy of small LLMs on tabular math reasoning tasks by about four points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may rest on a modified test set: the methodology says test questions were removed when BLEU exceeded δ=0.95, but Table 2 does not disclose which test questions were excluded or recompute baselines on the same subset.","rationale":"The reader identifies paraphrasing semantic drift as the weakest assumption, but the paper's own methodology contains a more direct threat to the central claim: the evaluation set may have been altered by the BLEU-based decontamination step. If test questions were removed only for the TeLL-trained models, the headline improvements are not comparable to the baselines, and the central claim is not established. This concern is concrete, textually grounded, and testable from the released artifacts. The paper's human verification rate of 97.5% on 1,000 sampled generated examples is useful positive evidence, and the controlled ablation in Table 3 does suggest that the specific TeLL pipeline matters more than template-only or LLM-only generation, but none of this addresses whether the reported test accuracy is computed on the same test set for all rows. I would keep the verdict CONDITIONAL rather than REJECT because the issue could be resolved by a code/data check and a full-test-set rerun; however, the condition must be that the evaluation is performed on the identical, unmodified TabMWP test set and that the number of any excluded questions is reported. My disagreement with the reader is therefore about which assumption is most load-bearing, not about the overall verdict.","tokens_in":14675,"tokens_out":5272,"duration_ms":50580,"concrete_test":"Inspect the released code and data to determine the actual target of the BLEU filter: are test-set entries being removed from the evaluation, or generated entries being removed from TabMWP-TeLL? Then rerun the three fine-tuned models and the corresponding 'Trained with TabMWP' baselines on the full original TabMWP test set (7,686 questions) without any BLEU-based exclusion, and report per-model accuracy and the number of excluded questions. If the gains persist on the identical full test set, the concern is resolved; if they shrink or vanish, the reported improvement is an artifact of test-set pruning. Also verify that template extraction used only the training split of TabMWP.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing issue is not the semantic drift of paraphrases but the integrity of the evaluation set. In the Problem Paraphrasing subsection, the paper states: 'We also remove questions in the test set where at least one sample has a BLEU score greater than δ to prevent potential data leakage issues.' As written, this removes TabMWP test questions rather than removing generated training examples. If any test questions were deleted before evaluating the TeLL-trained models, while the baseline rows in Table 2 were evaluated on the full 7,686-question test set, then the reported gains of 3.96%, 3.70%, and 3.78% are not like-for-like comparisons. The paper does not report how many questions were removed, whether the same subset was used for all models, or whether any baseline was recomputed on the reduced set. A second, related risk is template leakage: the template database is built from 'existing real samples' without stating that only the TabMWP training split was used. If test-set samples contributed templates, the BLEU filter at δ=0.95 would only catch near-verbatim duplicates, not structurally similar but lexically different generated instances. Both issues place the central empirical claim—consistent large gains from TabMWP-TeLL—on unverified evaluation foundations.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TeLL, a two-stage framework for generating tabular math word problems (TMWPs): first, abstract templates are extracted from existing real samples and augmented by an LLM; second, template-instantiated problems are paraphrased by an LLM into diverse, contextualized questions while preserving the template's arithmetic logic. The authors construct a new dataset TabMWP-TeLL and fine-tune Mistral-7B, Qwen 2-7B, and Llama-3-8B on TabMWP plus TabMWP-TeLL. They report consistent gains over fine-tuning on TabMWP alone: 3.96%, 3.70%, and 3.78% absolute accuracy improvements on the TabMWP test set, with Llama 3 reaching 98.07% overall accuracy. They also report a human-verification correctness rate of 97.5% on 1,000 generated samples and ablations showing that template-based, LLM-only, and question-only generation methods underperform the full TeLL pipeline.","tokens_in":14911,"tokens_out":4595,"duration_ms":42483,"significance":"If the reported gains are verified, the paper makes a useful contribution to data augmentation for tabular math reasoning: it combines a correctness-preserving template stage with an LLM paraphrase stage, releases code and data, performs controlled ablations that fix the generated-data size at 23K, and includes human verification of generated samples. The three-LLM evaluation and the emphasis on illustrative step-by-step solutions are also strengths. However, the central empirical claim depends on the integrity of the test-set construction and on the absence of a like-for-like control for added data volume; both need to be clarified before the measured improvements can be taken at face value.","major_comments":[{"comment":"The paper states: \"We also remove questions in the test set where at least one sample has a BLEU score greater than δ to prevent potential data leakage issues.\" This means that the TabMWP test set used in Table 2 is a filtered subset, not the full 7,686-question test set, yet the table does not report how many questions were removed, whether all baseline and fine-tuned rows were evaluated on exactly the same subset, or whether any baseline accuracy was recomputed on the reduced set. Because BLEU-based filtering is likely to remove test questions that resemble the generated data, the reported gains of 3.96%, 3.70%, and 3.78% could be inflated by evaluating on an easier subset. The authors must report the number and identities of removed test questions, recompute all rows of Table 2 on the identical subset, and either evaluate on the full test set or justify the filtering as a necessary leakage-prevention step.","section":"Methodology, Problem Paraphrasing"},{"comment":"The template database is described as being built from \"existing real samples\" without specifying that only the TabMWP training split was used. If any validation or test examples contributed templates, then generated training samples could share structural, lexical, or tabular patterns with test examples, and the BLEU filter at δ=0.95 would only catch near-verbatim duplicates rather than structurally similar rewrites. The paper should explicitly state which TabMWP split was used for template extraction and, ideally, rerun the main experiment with templates derived solely from the training split.","section":"Methodology, Template Abstraction"},{"comment":"The main comparison in Table 2 contrasts fine-tuning on TabMWP alone with fine-tuning on TabMWP plus TabMWP-TeLL. Since the latter adds roughly 23K extra training instances, the observed gains could be due to increased data quantity rather than to the quality or design of TabMWP-TeLL. Table 3 controls for the generation method by fixing the generated-data size at 23K, but it does not include a control that adds an equal volume of original TabMWP training examples (or a similarly human-written dataset) to the TabMWP training set. An equal-volume augmentation control is needed to attribute the improvements to the framework rather than to the larger training set.","section":"Main Results and Ablation Study"},{"comment":"The claim that \"Since our LLM-based paraphrasing does not change the mathematical logic, the correctness can be ensured\" is stronger than the evidence supports. The only automated consistency check is whether the LLM-produced solution's final answer matches the template's answer; this cannot detect paraphrases that change the question semantics while coincidentally yielding the same numerical answer. The human verification reports a 97.5% correctness rate and categorizes errors, but it does not report a separate judgment on whether the paraphrased question is semantically equivalent to the template question. The authors should either strengthen the automated verification (e.g., checking the paraphrase against the template's logical form) or provide a human-verified measure of semantic equivalence on a sample of the generated data.","section":"Methodology, Problem Paraphrasing and Human Verification"},{"comment":"All fine-tuning results in Tables 2–4 are reported from what appears to be a single run per configuration, with no standard deviations, no repeated seeds, and no significance tests. Given that the headline gains are around 3.7–4.0 percentage points and that QLoRA fine-tuning is stochastic, the authors should report means and variances over at least three runs, or otherwise justify that the differences are stable. This is particularly important for the subtable comparisons in Table 5, where some cells differ by only a fraction of a point.","section":"Experimental Setup"}],"minor_comments":[{"comment":"The column header \"TableMWP\" appears to be a typo for \"TabMWP.\"","section":"Table 5"},{"comment":"The abstract states \"we find the reasoning annotation is important for solving TMWPs,\" but the paper does not provide a comparison with alternative reasoning-annotation formats (e.g., compact formulas or numeric-only solutions); the ablation in Table 4 compares free-form versus illustrative solutions only.","section":"Abstract and Methodology"},{"comment":"The main text never states the total size of the released TabMWP-TeLL dataset. Table 3 mentions that the ablation methods generate 23K problems, but the final dataset size should be stated explicitly in the experiments section.","section":"Dataset Description"},{"comment":"Equation (2) writes the paraphrasing step as producing (q,t,a,s), but in the pipeline the answer a is supposed to remain unchanged from the template while only the question, table wording, and solution are paraphrased. The notation should clarify that the answer is carried through the paraphrase step rather than regenerated.","section":"Problem Definition and Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The most serious risk is the test-set filtering described in the Problem Paraphrasing subsection. If the authors can show that all baselines in Table 2 were evaluated on the same filtered subset and that the number of removed questions is small, the central claim could survive. If they cannot, the reported gains would not be a like-for-like comparison. The template-split question and the missing equal-volume augmentation control are also important but are fixable with additional experiments or disclosure. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent empirical paper with a plausible central claim, but the main result may rest on an unfair evaluation. The pipeline is a genuine combination—template extraction, LLM augmentation/paraphrasing, and illustrative solutions—and the authors release a new dataset plus a human-verified quality check. The controlled ablation at fixed 23K generated problems is the right kind of comparison, and the consistent ~4-point gains across Mistral, Qwen2, and Llama3 are striking. However, the methodology text says they removed questions from the test set when a generated sample had BLEU > 0.95, and Table 2 never discloses how many questions were dropped, which ones, or whether the baseline rows were recomputed on the same subset. If baselines were run on the full 7,686-question test set and the TeLL models on a reduced set, the comparison is not like-for-like. This is not a manufactured quibble; it is the central result.\n\nSecond, the gain could be partly a data-volume effect. The baseline is TabMWP only; adding 23K generated problems increases total data. There is no equal-volume control that adds 23K original TabMWP samples (or another augmentation) while holding quantity fixed. The scaling curve in Figure 6 varies the mix but not the total. So the specific contribution of TeLL quality is not cleanly separated from the simple benefit of more training data.\n\nSingle-run results without error bars are a minor issue here because the deltas are large, but they should still be reported. The semantic-preservation assumption in paraphrasing is also a mild worry: answer-consistency filtering cannot catch cases where the paraphrased question asks something different but still yields the same answer. The 97.5% human correctness on 1,000 generated samples is reassuring but concerns the generated data, not the evaluation set.\n\nWho should read this: anyone working on synthetic data for math reasoning, especially tabular problems. It deserves a serious referee, but the authors must disclose the test-set modification and recompute baselines on the same subset, ideally adding the equal-volume control. If they do, the paper could be solid. As is, I would treat the ~4-point gains as provisional.","headline":"Useful pipeline and dataset, but the reported gains may be an artifact of a modified test set and a missing equal-volume baseline.","tokens_in":15470,"tokens_out":2775,"would_cite":false,"duration_ms":24659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic tabular math problems lift LLM accuracy ~4 points","keywords":["tabular math word problems","TMWP","data generation","template-based generation","LLM paraphrasing","chain-of-thought","fine-tuning","TabMWP-TeLL"],"falsifier":"Take a random sample of generated items where the paraphrased problem and the template problem share the same answer, have expert annotators judge whether the question, table, and solution express the same mathematical query, and compute the proportion of semantic mismatches; a rate above a few percent would invalidate the correctness guarantee. A cleaner experimental falsifier is to re-run the fine-tuning comparisons with the answer-consistency filter replaced by a stricter logic-preservation check, such as back-translation or symbolic validation, and see whether the reported gains persist.","tokens_in":14470,"feed_emoji":"📊","tokens_out":7736,"duration_ms":61377,"temperature":0.7,"pith_summary":"Tabular math word problems are a standard test of LLM reasoning, but hand-built datasets are expensive to scale. The paper proposes TeLL, a two-stage generator: abstract templates from real solved problems, instantiate them with fresh numbers, then ask an LLM to paraphrase the question and table into a natural background while leaving the arithmetic logic intact. Because the template dictates the answer, correctness is guaranteed by construction; the LLM supplies diversity. The concrete payoff is that fine-tuning three 7B-8B LLMs on the original TabMWP data plus the generated TabMWP-TeLL data lifts their test accuracy by 3.96%, 3.70%, and 3.78% respectively, with Llama 3 reaching 98.07%. The implication is that large volumes of reliable, varied tabular math training data can be synthesized instead of collected.","feed_headline":"Synthetic tabular math problems lift LLM accuracy ~4 points","feed_subtitle":"Adding a template-plus-paraphrase dataset to training lifts Llama 3, Qwen 2, and Mistral by up to 3.96 percent.","key_machinery":"The load-bearing object is the template tuple $P(x)$, which couples question, table, answer, and solution through shared placeholders. The pipeline's invariants are: templates are abstracted from real samples so the arithmetic is valid; an LLM extends templates to neighboring question types; instantiation fills random numbers and categories; and a final LLM paraphrases the problem into a contextual story under instructions not to change the logic. The step-by-step illustrative solution is a key part of the generated item: it is produced by an LLM from the original free-form solution, and the ablations show that models trained on these illustrative solutions beat models trained on the original free-form solutions by about 1.6-1.9 points.","core_discovery":"The central discovery is that a hybrid template/LLM pipeline can produce tabular math problems that are simultaneously correct and diverse, and that these synthetic problems transfer to real tests. Each template is a tuple $P(x)=(Q(x), T(x), A(x), S(x))$ with shared placeholders for numbers and categories. After instantiation, the LLM paraphrase step is instructed to keep the problem, data, and solution logic unchanged; an answer-consistency filter drops cases where the solution's answer differs from the template's answer, and a BLEU threshold removes near-duplicates of test questions. Fine-tuning Mistral-7B, Qwen 2-7B, and Llama 3-8B on TabMWP plus TabMWP-TeLL outperforms training on TabMWP alone on every model, with the largest gains on difficult grade 7-8 and stem-leaf plot problems, and a human check of 1,000 generated samples reports 97.5% correctness.","pith_inferences":["Editorial inference: the answer-consistency filter only catches items where the paraphrased solution produces a different number from the template; semantic drift that preserves the answer would pass, so the 97.5% human verification rate may be an upper bound on the clean fraction if such drift is common.","Editorial inference: the ablations suggest the illustrative solutions, not just the new questions, drive much of the gain; a factorial design that holds questions fixed while toggling solution style would isolate this contribution.","Editorial inference: because the source of templates is a single dataset, the template database may inherit that dataset's question-type distribution; applying the same pipeline to broader or noisier tabular sources would test how far the correctness guarantee degrades.","Editorial inference: the BLEU-based deduplication at $\\delta=0.95$ removes near-exact copies of test questions, but semantically equivalent rephrasings with lower BLEU could still leak; evaluating on a fresh paraphrase distribution, or using a semantic similarity measure, would give a sharper estimate of true generalization."],"forward_implications":["Adding TabMWP-TeLL to the training mix improves Mistral, Qwen 2, and Llama 3 by 3.96%, 3.70%, and 3.78% on the TabMWP test set, with Llama 3 reaching 98.07% overall.","The gains concentrate on challenging items: Llama 3's grade 7-8 accuracy rises to 97.42%, and stem-leaf plot subtypes improve by 11.92% on average over training on TabMWP alone.","Data scaling shows that even 20% of the generated data yields a substantial improvement, and accuracy keeps rising as more generated data is added.","The framework is not tied to the 25 question types instantiated here; the authors state it can be extended to new and unseen question types.","Human verification of 1,000 sampled generated problems reports a 97.5% correctness rate, with the remaining errors classed as incomplete paraphrases, incorrect paraphrased solutions, or grammar issues."],"supporting_citations":[{"why":"It supplies the source TMWP dataset, the 25 question types selected, and the test set used to measure all results.","marker":"Lu et al. 2023b"},{"why":"Yi is the LLM used for template augmentation and problem paraphrasing in the TeLL pipeline.","marker":"Young et al. 2024"},{"why":"Mistral-7B is one of the three base models fine-tuned to demonstrate the gains.","marker":"Jiang et al. 2023"},{"why":"Qwen 2-7B is one of the three base models fine-tuned to demonstrate the gains.","marker":"Yang et al. 2024"},{"why":"Llama 3-8B is the base model that reaches 98.07% after training with TabMWP-TeLL.","marker":"Dubey et al. 2024"},{"why":"QLoRA is the parameter-efficient fine-tuning strategy used in all experiments.","marker":"Dettmers et al. 2023"}],"fun_headline_variants":["Hybrid generation creates diverse, correct tabular math problems","Template-LLM framework improves LLM tabular math accuracy by ~4 pts","Synthetic tabular math data from templates + paraphrasing lifts LLMs","TabMWP-TeLL dataset boosts LLM performance on tabular word problems","Correct and diverse: generating tabular math problems via template-LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the LLM paraphrase step preserves the mathematical logic of the template problem, with only the final answer compared automatically; if the paraphrase changes what the question asks while still yielding the same numeric answer, the generated data can contain mismatched or mislabeled problems.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid generation creates diverse, correct tabular math problems","Template-LLM framework improves LLM tabular math accuracy by ~4 pts","Synthetic tabular math data from templates + paraphrasing lifts LLMs","TabMWP-TeLL dataset boosts LLM performance on tabular word problems","Correct and diverse: generating tabular math problems via template-LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1824,"prompt_tokens":1013,"completion_tokens":811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":629,"tokens_out":811,"duration_ms":6906,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:17:04.140719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of generated items where the paraphrased problem and the template problem share the same answer, have expert annotators judge whether the question, table, and solution express the same mathematical query, and compute the proportion of semantic mismatches; a rate above a few percent would invalidate the correctness guarantee. A cleaner experimental falsifier is to re-run the fine-tuning comparisons with the answer-consistency filter replaced by a stricter logic-preservation check, such as back-translation or symbolic validation, and see whether the reported gains persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"QLoRA is the parameter-efficient fine-tuning strategy used in all experiments."}],"review_version":1}