{"id":"c8bf481b-8da3-4c75-9aed-b467ddc6600f","arxiv_id":"2412.09906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SGFT fine-tunes a small model to produce calculation-free solution plans and uses a second model to answer from them, outperforming CoT fine-tuning with roughly 3% of the training data.","lead":"This paper introduces Solution Guidance Fine-Tuning (SGFT), which teaches a small language model to output high-level problem-solving steps without calculations, then feeds those steps to a second small model that computes the answer. On five reasoning benchmarks, the authors report that SGFT with about 3,000 training examples beats chain-of-thought fine-tuning with 30,000 examples, while running on a single consumer GPU.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SG advantage is not isolated from two confounds: the SG pipeline uses a stronger GPT-4o teacher and a second, untrained answer model, while the CoT baseline is a single fine-tuned model.","rationale":"The reader's weakest_assumption identifies exactly the right soft spot: the experiments vary several factors at once (teacher strength, data format/cleaning, single vs. two-model inference). I agree that the ablations in Figure 4 do not close the gap because they omit the symmetric condition where a CoT-finetuned model serves as the guidance generator. This is load-bearing because the abstract and Section 4.2 sell SG as the causal ingredient and claim a data-efficiency advantage over CoT fine-tuning. If the two-model setup alone accounts for the improvement, the method is still a reasonable plan-and-execute distillation scheme, but the distinctive claim about SG and its 10x data savings is not established. A single controlled comparison would settle it: CoT-finetuned generator + untuned answerer. I do not see an internal inconsistency in the method, and the reported gains are consistent across models and datasets, so rejection is not warranted; the appropriate disposition is the same conditional acceptance the reader gave, pending the missing control. The paper does provide code/data links and uses a standard evaluation setup, which is supportive, but none of that substitutes for the missing ablation.","tokens_in":12735,"tokens_out":4468,"duration_ms":48044,"concrete_test":"Use the existing ChatGLM3-6B_CoT checkpoint as the guidance model: for GSM8K test questions, let it generate its usual chain-of-thought (without forcing a final answer), then feed question + CoT to the original untuned ChatGLM3-6B to produce the final answer. If this two-model CoT baseline reaches or exceeds the 43.7 reported for SG+ChatGLM3-6B, the SG-specific data format is not responsible for the gain. As a companion check, run the inverse: have the SG-finetuned model both generate guidance and produce the final answer in one pass; if accuracy collapses, the second model is doing essential work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract, Section 1, Section 4.2) is that 3,000 SG examples outperform 30,000 CoT examples. For that to be true, the SG format itself—not the teacher or the inference setup—must drive the gain. The experiments do not establish this. In Table 3, the SG entry is always SG-finetuned generator + an untuned second SLM, so it uses roughly twice the parameters and inference compute of the single CoT-finetuned baseline. The teacher is GPT-4o, and Section 3.1 also describes manual cleaning and removal of numerical cases, so the training material is not merely 'less data'; it is a different, stronger, and partially hand-edited dataset. Figure 4's ablation compares SG-finetuned guidance + untuned answer against untuned guidance + CoT-finetuned answer, but never the key symmetric baseline: a CoT-finetuned model used as the guidance generator with the same untuned answer model. Without that baseline, the observed improvement could come from the two-model decomposition (guidance then execution) or from GPT-4o's planning quality, rather than from Solution Guidance as a data format. The claim that SG is the key ingredient is therefore conditional on a missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Solution Guidance (SG), a reasoning-data format in which a model produces 2–6 high-level solution steps or objectives without performing arithmetic, and Solution-Guidance Fine-Tuning (SGFT), a pipeline that uses GPT-4o to generate SG data for 1,000–3,000 GSM8K questions, fine-tunes a 6–7B small language model with the LISA method to emit such guidance, and then feeds the original question plus the generated guidance to a second, untrained small model that produces the final answer. Across GSM8K, SVAMP, MultiArith, StrategyQA, and CommonsenseQA, the paper reports that SGFT with 3,000 SG examples outperforms CoT fine-tuning with 30,000 examples for ChatGLM3-6B, Qwen2-7B, and Llama2-7B, and it also reports cross-model combinations and ablations over context and few-shot settings during fine-tuning.","tokens_in":13010,"tokens_out":5746,"duration_ms":63320,"significance":"If the central comparison is valid, the practical contribution is meaningful: a roughly 10x reduction in fine-tuning data, single-GPU training, and preservation of the base model's general abilities would make reasoning enhancement more accessible for small models. The paper has clear strengths: it evaluates on five standard benchmarks with three 6–7B families, includes held-out out-of-distribution datasets, provides code and data, and the SG idea of separating problem decomposition from computation is interesting and easy to reproduce. However, the manuscript does not yet establish that the reported gains come from the SG format itself, because the main comparison and the ablation both leave a key control unmeasured; the remaining comments identify the specific missing experiments.","major_comments":[{"comment":"The main comparison in Table 3 does not isolate the SG data format. Every SG row is the SG-fine-tuned generator plus an untuned second model, whereas every CoT baseline is a single fine-tuned model; additionally, the SG training data is produced by GPT-4o and manually cleaned (Section 3.1), while the CoT data source and cleaning are not described at the same level of detail. The observed margin could therefore come from the stronger teacher, from the selection/cleaning of training instances, or from the two-model decomposition rather than from SG as a data format. Please add the missing control: use a CoT-fine-tuned model as the guidance generator with the same untuned answer model, and, if feasible, generate the CoT training set with the same teacher and the same data protocol so that teacher strength and inference architecture are held fixed.","section":"§4.2, Table 3"},{"comment":"The ablation tests only four combinations: untrained guidance with untrained answering, untrained guidance with a CoT-fine-tuned answer model, SG-fine-tuned guidance with untrained answering, and SG-fine-tuned guidance with a CoT-fine-tuned answer model. The critical symmetric cell, a CoT-fine-tuned model used as the guidance generator with the same untuned answer model, is missing. Without this cell, the improvement attributed to SG could instead be an effect of the two-model decomposition itself. Figure 4 also omits the numeric scores and the model/dataset it refers to; please report those details and add the missing cell.","section":"§4.3, Figure 4"},{"comment":"No standard deviations, multiple seeds, or significance tests accompany the central accuracy comparisons. The decoding temperature is set to 0, but fine-tuning itself is stochastic, and several of the reported gaps are modest (for example, ChatGLM3-6B on SVAMP: 51.2 vs 47.5, and Qwen2-7B on MultiArith: 61.4 vs 57.2). Please report results over at least a few fine-tuning runs and, if the distributions permit, paired significance tests so that the claim of significant improvement is statistically supported.","section":"§4.1–§4.2, Tables 3–4"}],"minor_comments":[{"comment":"The abstract contains a grammatical error ('this paper introduce') and Table 2 contains '8,00 weeks' in the first example; both should be corrected.","section":"Abstract, Table 2"},{"comment":"The conclusion states that 1,000 SG examples outperform 30,000 CoT examples, while Table 3 and Section 4.2 emphasize 3,000 SG examples; please reconcile these numbers and make Figure 3 explicitly support the 1,000-example claim if it is intended.","section":"Conclusion vs §4.2"},{"comment":"Section 3.1 says a subset of 2,000 questions was randomly selected, while Section 4.1 says 1,000, 2,000, and 3,000 questions were used; please clarify whether 2,000 refers to a different selection or is a typo.","section":"§3.1 vs §4.1"},{"comment":"The paper uses 'GPT-4' and 'GPT-4o' interchangeably in several places; please use a single naming convention consistently.","section":"§3.1 and §4.1"},{"comment":"Figure 4 lacks axis labels and numeric values, and Table 4 does not report variance or the exact number of SG examples used; adding these details would make the ablation legible and reproducible.","section":"Figure 4 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The missing control is the key issue: the paper's headline claim is that the SG data format, not the teacher or the two-model architecture, drives the gain. That claim is plausible but not yet supported by the experiments as written. I would not reject the paper; the method is simple, reproducible, and the empirical pattern is worth publishing if the control experiments confirm the attribution. Please also ask the authors to describe the manual cleaning of SG data in the experimental setup so that the 'less data' claim is not overstated. The paper is appropriate for the journal's scope if the empirical claims are tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper describes SGFT, a fine-tuning pipeline where a small model is trained to emit calculation-free solution guidance (SG), and a second, untuned small model then uses that guidance to produce the answer. The reported numbers are consistently positive across three 6-7B models and five benchmarks, and the paper is clearly written with honest ablations on context and few-shot settings. That is real value.\n\nThe key claim is that 3,000 SG examples beat 30,000 CoT examples. That claim is not yet established. Figure 4 ablates four combinations but never the symmetric baseline: a CoT-finetuned model used as the guidance generator with the same untuned answer model. Without that control, the gain could come from the two-model decomposition itself, or from the fact that the SG training data was generated by GPT-4o and then hand-cleaned to remove numerical steps. The paper reports no error bars or significance tests, and the CoT baseline is a single configuration, so we don't know the variance. This is a moderate but important gap.\n\nNovelty is also limited: separating planning from execution is exactly Plan-and-Solve and Think-and-Execute. The paper cites Think-and-Execute but does not compare against it, and Plan-and-Solve is not cited. The specific SG data format is new, but the underlying idea is known.\n\nWhat I do credit: the method is practical (works on consumer GPUs with LISA), the reporting is transparent about limitations, and the results are plausible as a lower-bound improvement. This is a useful datapoint for the small-model reasoning literature, but the main quantitative headline needs a more careful experimental design.\n\nRecommendation: send to peer review. The paper deserves a serious referee who can ask for the missing baseline and for repeated runs. With those added, the contribution would be modest but solid.","headline":"A practical plan-and-execute distillation recipe with a strong but unproven data-efficiency claim.","tokens_in":13530,"tokens_out":2691,"would_cite":false,"duration_ms":28522,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"With only 3,000 solution-guidance examples, small language models outperform chain-of-thought fine-tuning with 30,000 examples on five reasoning benchmarks.","keywords":["solution guidance","small language models","reasoning","chain-of-thought fine-tuning","collaborative inference","data efficiency","GSM8K","LISA fine-tuning"],"falsifier":"Fine-tune a small model on 3,000 GPT-4o-generated Chain-of-Thought examples instead of Solution Guidance, keep the same two-model collaborative inference (the fine-tuned model as planner, an untrained model as answerer), and evaluate on GSM8K: if the CoT-based planner matches or beats the SG-based planner's 43.7 accuracy, the SG format is not the decisive factor; if it falls short, the format's contribution is confirmed.","tokens_in":12516,"feed_emoji":"🧩","tokens_out":8724,"duration_ms":74525,"temperature":0.7,"pith_summary":"The paper claims that the reasoning bottleneck for small language models (under 10 billion parameters) is problem decomposition, not computation, and that a model trained to emit concise, calculation-free solution plans can guide any other small model to the correct answer. It introduces Solution Guidance (SG), a data format of two-to-six step solution objectives, and Solution-Guidance Fine-Tuning (SGFT), which trains a small model on 1,000-3,000 SG examples to produce such plans. In collaborative inference, the fine-tuned planner's guidance is passed, together with the original question, to a second, unmodified small model that writes the final answer. Across GSM8K, SVAMP, MultiArith, StrategyQA, and CommonsenseQA, the approach outperforms chain-of-thought fine-tuning with 30,000 examples, and the paper reports that 1,000 SG examples already beat 30,000 CoT examples, cutting data requirements to roughly 3-10%. If correct, it makes stronger reasoning practical on a single consumer-grade GPU without large annotated CoT corpora.","feed_headline":"3,000 solution plans beat 30,000 CoT examples for small LLMs","feed_subtitle":"A 6-7B model that generates step plans, not calculations, beats chain-of-thought fine-tuning using 10x less data.","key_machinery":"The load-bearing object is the Solution Guidance (SG) data schema: a question paired with a two-to-six step list of solution objectives and their order, explicitly without calculations or final answers. The protocol that carries the result is two-model collaborative inference: a fine-tuned guidance model (Model_SG) reads the question and emits SG, and an untrained response model receives the original question plus SG and generates the answer. Fine-tuning uses the LISA layer-wise importance sampling method on a consumer GPU, with contextual prompts (zero-shot or few-shot) included in the training data to stop the planner from slipping into calculations. The division of labor is what the paper credits for the accuracy gain, since an SG-fine-tuned planner paired with an untrained answerer outperforms all other combinations in their ablation, including pairing the planner with a CoT-fine-tuned answerer.","core_discovery":"The central discovery is that separating planning from calculation improves small-model reasoning. A small language model fine-tuned on Solution Guidance, which lists step-by-step solution objectives without any arithmetic, can decompose unseen problems into generic plans, and a second, unmodified small model can execute those plans to produce final answers. The paper reports that SGFT with 3,000 SG examples, generated from GSM8K training questions by GPT-4o, beats CoT fine-tuning with 30,000 examples across all five tested benchmarks; for instance, ChatGLM3-6B guided by ChatGLM3-6B_SG reaches 43.7 on GSM8K versus 34.4 for the CoT-fine-tuned model, and the best cross-model pair, Qwen2-7B_SG generating guidance for ChatGLM3-6B, reaches 48.3. The authors argue this works because SG focuses on problem understanding and decomposition at the semantic and logical levels, avoiding the error propagation and noise that come from concatenating calculations in CoT chains.","pith_inferences":["The reported gains likely combine three factors that the paper does not isolate: the SG format, the much stronger GPT-4o teacher that generated the guidance, and the two-model split that offloads answer generation to an untrained model; a controlled comparison that swaps SG data for CoT data of the same size and same teacher would reveal how much of the improvement the format itself contributes.","The planner/answerer split resembles a planning-and-execution architecture; if the SG plan is the main driver, one testable extension is to have the answer model execute several alternative SG plans for the same question and vote on the result, which the authors list as future work under self-consistency.","Because SG is higher-level, language-agnostic guidance, it may extend beyond math word problems to tasks like multi-hop QA or tool-use planning, where decomposition rather than arithmetic is the bottleneck; the paper's own CommonsenseQA and StrategyQA results are consistent with that direction.","A direct scaling prediction follows from Figure 3: accuracy on GSM8K rises with SG training data, so the 3,000-example ceiling is likely not the practical optimum; testing 5,000 or 10,000 SG examples would show whether the gap over CoT widens or saturates."],"forward_implications":["With roughly 3,000 SG examples, practitioners can fine-tune a 6-7B model on a single consumer-grade GPU and exceed CoT fine-tuning that uses 30,000 examples, cutting data preparation cost by an order of magnitude.","Because the answer model needs no training, any small instruction-tuned model can serve as the responder, so a single SG planner can be reused with different answer models depending on deployment needs.","The guidance format transfers across benchmarks: training data drawn only from GSM8K improves out-of-distribution performance on SVAMP, MultiArith, StrategyQA, and CommonsenseQA, suggesting the learned plans capture reusable problem-solving patterns.","Separating planning from calculation keeps the planner from generating arithmetic errors, and the paper's ablation indicates that pairing an SG planner with an untrained answer model is better than pairing it with a CoT-fine-tuned answer model.","The data-efficiency result implies that low-data settings, where large CoT corpora are unavailable or too noisy, are exactly where SGFT has the largest relative advantage."],"supporting_citations":[{"why":"Defines Chain-of-Thought reasoning, the baseline method the paper compares against.","marker":"(Wei et al., 2022)"},{"why":"Introduces using large models as teachers to generate CoT data for fine-tuning small models, the paradigm behind the CoT fine-tuning baseline.","marker":"(Ho et al., 2022)"},{"why":"Provides the GSM8K dataset, the source of all SG training questions and the in-distribution benchmark.","marker":"(Cobbe et al., 2021)"},{"why":"Provides the SVAMP benchmark used as an out-of-distribution math evaluation set.","marker":"(Patel et al., 2021)"},{"why":"Provides the MultiArith benchmark used as an out-of-distribution math evaluation set.","marker":"(Roy and Roth, 2016)"},{"why":"Provides the CommonsenseQA benchmark used to test common-sense reasoning transfer.","marker":"(Talmor et al., 2019)"},{"why":"Provides the StrategyQA benchmark used to test implicit-reasoning transfer.","marker":"(Geva et al., 2021)"},{"why":"Supplies the LISA layer-wise importance sampling method used for memory-efficient fine-tuning.","marker":"(Pan et al., 2024)"},{"why":"Supplies the few-shot selection procedure used to choose the initial seven questions for SG generation.","marker":"(Brown et al., 2020)"}],"fun_headline_variants":["Plan, don't calculate: 3k guides beat 30k chains for small LLMs","Small LLMs get smarter with 10x less data via solution guidance","Separate planning from math: SGFT lifts small model reasoning","3k plans beat 30k CoT: split thinking from doing for SLMs","Smart small LLMs: 3k solution guides outperform 30k reasoning chains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central claim rests on the assumption that the solution-guidance data format, rather than the stronger GPT-4o teacher or the two-model inference setup, is what drives the accuracy gains.","fun_headline_variants_meta":{"raw":{"variants":["Plan, don't calculate: 3k guides beat 30k chains for small LLMs","Small LLMs get smarter with 10x less data via solution guidance","Separate planning from math: SGFT lifts small model reasoning","3k plans beat 30k CoT: split thinking from doing for SLMs","Smart small LLMs: 3k solution guides outperform 30k reasoning chains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1511,"prompt_tokens":1001,"completion_tokens":510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":406}},"tokens_in":617,"tokens_out":510,"duration_ms":5218,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:35:15.501426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a small model on 3,000 GPT-4o-generated Chain-of-Thought examples instead of Solution Guidance, keep the same two-model collaborative inference (the fine-tuned model as planner, an untrained model as answerer), and evaluate on GSM8K: if the CoT-based planner matches or beats the SG-based planner's 43.7 accuracy, the SG format is not the decisive factor; if it falls short, the format's contribution is confirmed.","supporting_citations":[{"cited_title":"Bhattamishra, and Navin Goyal","cited_arxiv_id":null,"evidence_quote":"Provides the SVAMP benchmark used as an out-of-distribution math evaluation set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the StrategyQA benchmark used to test implicit-reasoning transfer."}],"review_version":1}