Pith. sign in

REVIEW 3 major objections 6 minor 15 references

Reasoning models can be adapted with only input-output pairs—no verifier, no reasoning traces—by instruction tuning then linearly merging back toward the original weights, choosing the ratio from non-empty reasoning rates on a calibration s

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:46 UTC pith:LIADHZZI

load-bearing objection A practical IFT-then-merge recipe for adapting RLMs without verifiers — worth engaging, but the calibration signal is trace presence, not reasoning quality, and the abstract overstates cost. the 3 major comments →

arxiv 2607.14895 v1 pith:LIADHZZI submitted 2026-07-16 cs.LG cs.CL

Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

Yu-Du Feng , Niels M\"undler-Sasahara , Mark Vero , Martin Vechev This is my paper
classification cs.LG cs.CL
keywords reasoning language modelsinstruction fine-tuningmodel merginglinear interpolationreasoning-trace collapseverifier-free adaptationcalibration-based alpha selectioncode and summarization tasks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper is trying to establish a cheap, verifier-free recipe for adapting reasoning language models to tasks where only input-output pairs are available. It argues that the familiar failure mode—instruction tuning makes a reasoning model stop thinking—can be repaired by linearly interpolating the fine-tuned weights with the original reasoning model's weights. The only signal it uses to pick the interpolation coefficient is the fraction of calibration responses that contain a non-empty reasoning trace. The central finding is that this signal suffices: across four open reasoning models, the merged model keeps most of the task gain from instruction tuning while restoring held-out math reasoning to within about a point of the original model, all in under an hour and under $3. If true, large piles of existing supervised data become usable for reasoning models without any of the machinery usually needed.

Core claim

The paper's central claim is that a two-step procedure—standard instruction fine-tuning (supervised fine-tuning with empty reasoning traces) followed by linear model merging back toward the original reasoning model—recovers most or all of the reasoning capability lost by fine-tuning, while retaining a significant share of the target-task improvement. The merge ratio is chosen as the largest coefficient at which at least 90% of a small target-task calibration set still elicits a non-empty reasoning trace. The claim is supported on four open reasoning models and two tasks (Rust coding and text summarization), with MATH500 as a held-out check of general reasoning: the merged model keeps MATH500

What carries the argument

The load-bearing mechanism is linear weight interpolation between two checkpoints of the same model: θ_α = (1−α)θ_original + α θ_IFT. The central object is the reasoning rate ρ(M′,D), the fraction of responses containing a non-empty reasoning trace, measured on a small target-task calibration set. The search picks α* = max{α : ρ(M_α, D_cal) ≥ 0.9}, exploiting the empirical monotonic decrease of reasoning rate with α. This lets the pipeline select a point close to the IFT model—where task knowledge is strongest—without ever checking answer correctness. The cheap calibration trick is that once a model stops reasoning, it emits the end-of-reasoning marker immediately after the start marker, so

Load-bearing premise

The recipe assumes the fraction of non-empty reasoning traces on a small calibration set is a trustworthy proxy for restored reasoning quality, and that this rate drops monotonically as the merge ratio grows; if a model can produce traces without useful reasoning—as the paper's own prefill experiment shows—the chosen α may restore talking, not thinking.

What would settle it

Run the pipeline on a verifiable reasoning task such as MATH500 itself, select α by the paper's rule (largest α with reasoning rate ≥ 0.9 on a calibration split), then measure answer accuracy on a held-out split. If the selected α keeps MATH500 accuracy at the IFT model's depressed level despite non-empty traces, or if the reasoning rate does not decrease monotonically with α so the binary search returns a wrong point, the calibration signal is shown to be insufficient. The paper's report that pre-filling 'Okay' yields traces without recovering performance is already a partial negative instanc

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Standard instruction tuning on reasoning models is not just suboptimal but actively harmful to reasoning behavior; the paper documents drops of up to 77 points on MATH500 and 100% to 0% reasoning rates.
  • The recovered-reasoning benefit shows up concretely in task performance: on Rust coding, the merged model improves by an average of 7.0 points over the unadapted model, versus 3.8 from IFT alone, because the model both knows the task and still thinks about it.
  • The method costs less than $3 on a single H200 GPU, 22.8% and 33.0% less than the on-policy distillation and KL-regularization baselines, with hyperparameter search for the merge ratio included.
  • The pipeline works in both verifiable (coding) and hard-to-verify (summarization) domains, since it never needs a reward or correctness signal.
  • Merging techniques beyond linear (SLERP, TIES) preserve more MATH500 ability but give less target-task gain, so plain linear merging is the recommended default in this setting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a testable extension is to apply the same α-selection rule on a domain where reasoning quality itself is verifiable (e.g., MATH500) and check whether the chosen α tracks answer accuracy, not just trace non-emptiness. The paper's own 'Okay'-prefill experiment shows traces can be produced without recovered performance, so trace rate is at best a proxy.
  • Editorial inference: if this holds broadly, the practical meaning is that almost any existing supervised fine-tuning dataset—human-written solutions, style data, knowledge patches—can be applied to a reasoning model with a one-hour training run and a simple merge step, demoting 'reasoning trace availability' from a blocker to a non-issue for many applications.
  • Editorial inference: the paper's exploratory result that sequential application of the method loses reasoning faster suggests a scaling ceiling: a single train-and-merge step works, but stacking many such steps is not yet a solution. A natural next question is whether repeatedly merging against the original model, or using a fused anchor, behaves differently.
  • Editorial inference: the method's boundary is visible in its own discussion—on domains where reasoning gains define performance (like mathematics), input-output-only adaptation fails. This points to a division of labour: trace supervision remains necessary where the thinking itself is the product; this recipe is for inserting knowledge or aligning style.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a verifier-free method for adapting reasoning language models (RLMs) to new tasks using only input-output training data. The pipeline first performs standard instruction fine-tuning (IFT) on the target task, then linearly merges the IFT checkpoint with the original RLM. The merge coefficient α is selected on a held-out target-task calibration set as the largest α such that the non-empty reasoning-trace rate ρ(Mα,Dcal) is at least ρ_min=0.9. The method is evaluated on four open RLMs (OpenThinker 7B, Apriel Nemotron 15B, Olmo3 7B, DeepSeek R1 Qwen 7B) on Rust coding and text summarization, with MATH500 as a held-out proxy for general reasoning preservation. The main empirical finding is that standard IFT often collapses both target-task reasoning and MATH500, while the merged model restores MATH500 to within about one point of the base model in seven of eight settings while retaining a large fraction of the IFT target-task gain, at lower cost than on-policy distillation and KL-regularized IFT baselines.

Significance. If the reported results hold, the paper offers a practical, low-cost recipe for adapting RLMs to domains without verifiers, potentially opening up the large corpus of existing SFT data for reasoning-model adaptation. The manuscript has notable strengths: it releases code and datasets, reports a reproducible protocol (two seeds, ten evaluation runs), validates the LLM-as-a-judge setup with human correlations, gives a detailed runtime/cost breakdown, and is unusually candid about limitations, including a failed application to mathematical proofs. However, the central mechanistic claim — that calibration on non-empty trace rate is what selects a model that recovers lost reasoning — is not directly established. The positive results are empirical across the eight settings; the paper would be substantially strengthened by an analysis linking the calibration signal to actual reasoning quality and by scoping the claims accordingly.

major comments (3)
  1. [§3, Eq. (2)–(3); §6 “Interpretability”] The merge-ratio selection rule (Eq. 3) chooses α⋆ = max{α : ρ(Mα,Dcal) ≥ 0.9}, where ρ is the non-empty reasoning-trace rate (Eq. 2). The paper’s own §6 demonstrates that trace presence can be decoupled from useful reasoning: pre-filling a start-of-reasoning token with “Okay” produces a non-empty trace yet “performance of the untuned model was not recovered.” The paper never shows, across the α grid, that the set {α : ρ(Mα,Dcal) ≥ 0.9} coincides with the region where MATH500 is restored. It is therefore possible that the selected α* preserves “talking” without preserving reasoning; the good Table 1 outcomes could be particular to these eight settings rather than a consequence of the calibration criterion. Please add an α-sweep for at least one or two representative settings (e.g., OpenThinker 7B on Rust and text summarization) showing MATH500 and target-task accuracy alongside ρ(Mα,Dcal)
  2. [§4.2, Table 1; §6 “Lightweight adaptation on reasoning domains”] The central claim that the method “recovers most or all of the lost reasoning capability” is operationalized only as (i) a non-empty trace rate on the target task and (ii) MATH500, a held-out mathematics benchmark. No evidence is provided that the target-task reasoning traces themselves are useful — e.g., that trace correctness correlates with final-answer correctness. Moreover, §6 states that the method “failed to improve” on mathematical proofs, so the claim should be scoped to tasks where IFT already supplies the output-side knowledge. Please either add a trace-usefulness analysis (for example, comparing per-example reasoning content of M vs. Mα* on a sample of target-task examples) or explicitly qualify the generalizability claim in the Abstract and Conclusion.
  3. [§4.1 vs. Algorithm 1] Algorithm 1 prescribes grid search over A_K = {j/K}, while §4.1 states the final results use “up to eight search steps and binary search.” The binary search relies on the asserted monotonicity of ρ(Mα,Dcal) in α, which is observed for some settings but not demonstrated for all (Figure 2 suggests near-monotonicity, but Olmo3 maintains 100% reasoning for all α, and the text itself notes a sharp drop around a critical ratio). Please clarify which search procedure actually produced Table 1, provide evidence for monotonicity or a caveat, and report the selected α⋆ per setting (currently absent from Table 1), as this is needed to interpret the merging results.
minor comments (6)
  1. [Table 1] The text says results are averaged over two seeds and ten evaluation runs, but Table 1 reports only point changes with no variance measures. Please include standard deviations or confidence intervals, at least for the main Merge rows.
  2. [Table 1 (Olmo3 rows)] For Olmo3 7B, IFT does not degrade reasoning, so all methods reduce to the same IFT checkpoint. These rows cannot discriminate between methods and should be explicitly flagged as degenerate, so the reader does not count them as evidence for the merging step.
  3. [§4.2, Runtime and cost] The claim “less than USD $3” is an average over eight settings; Table 5 shows Apriel Rust (112.7 min) and Qwen Rust (89.1 min) individually exceed $3. Please state “on average” or report the full distribution.
  4. [§3, Eq. (2)] The empty trace ε is used in Eq. (2) but is not defined until later in the same section. Define it immediately before or in the equation.
  5. [§3, hyperparameters] ρ_min = 0.9 and the number of binary-search steps K are hand-set and not ablated. A brief sensitivity analysis over ρ_min would strengthen the method’s practical guidance.
  6. [§4.1] Typo: “linear merginging” should be “linear merging.”

Circularity Check

0 steps flagged
full rationale

The derivation chain is empirical and self-contained rather than definitional. The merge ratio alpha* is selected by Eq. (3) as the largest alpha with rho(M_alpha, D_cal) >= rho_min, using the non-empty reasoning-trace rate defined in Eq. (2). The central reported outcomes (MBPP-Rust scores, SummEval, MATH500) are measured on held-out test sets disjoint from D_cal, so they are not equal by construction to the fitted quantity; the paper also reports reasoning rates on evaluation datasets, which are correlated with the selection objective but not the same dataset. The self-citation of Alexandrov et al. 2024 (with Vechev as co-author) is used only as motivation for merging to mitigate forgetting, not as a load-bearing proof of the method, and the paper independently evaluates merging variants (linear, SLERP, TIES) and compares against OPD/KL baselines. The paper's limitation statements in Section 6 are genuine but do not create circularity: prefilling a response with 'Okay' produces traces without recovering untuned-model performance, and preliminary mathematical-proof experiments failed, showing that trace presence is an imperfect proxy for useful reasoning; this is a construct-validity/correctness risk, not a case where a reported prediction reduces to a fitted parameter. The cost comparison's omission of OPD/KL hyperparameter search is a fairness issue, not circularity. Overall, no specific reduction of a claimed result to its own inputs was found.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The method rests mainly on an empirical linear-mode-connectivity assumption for reasoning behavior, a reasoning-rate proxy, and the choice of evaluation benchmarks. No new entities or physical/formal objects are introduced. The principal free parameter is ρ_min, which determines the selected merge ratio.

free parameters (4)
  • ρ_min (minimum reasoning rate) = 0.9
    Equation (3) and Algorithm 1; hand-set threshold that defines the selected merge ratio. It directly controls the trade-off between IFT task gain and reasoning preservation, and it is not swept in the main experiments.
  • Binary search steps K = 8
    Section 4.1: 'using up to eight search steps'. Search resolution for α; affects the selected checkpoint and the reported cost.
  • Short-prefix token count for reasoning-rate calibration = unspecified ('first few tokens')
    Section 3 Optimizations; the number of prefix tokens used to decide whether a trace is non-empty is not reported, so the exact calibration behavior is not fully reproducible.
  • Calibration set size Dcal = unspecified for text summarization; 750 for Rust
    The calibration set is used to estimate reasoning rate and select α; its size influences the reliability of ρ, and the paper does not specify it for the summarization task.
axioms (4)
  • domain assumption Linear weight interpolation between an RLM and its IFT checkpoint lies on a trajectory with gradually varying reasoning behavior and target-task skill.
    Used in §3 Merging and §2 prior work. Without this, α-selection by reasoning rate would not produce useful models. The paper cites Frankle et al. 2020 and Ilharco et al. 2023 but does not prove it for RLMs.
  • domain assumption Reasoning rate is monotonically non-increasing with α, and short-prefix evaluation estimates it faithfully.
    §3 Optimizations: 'We observe that the reasoning rate decreases monotonically as the merge ratio increases' and short-prefix check. Empirical, not guaranteed across models and tasks.
  • domain assumption MATH500 is a sufficient held-out proxy for general reasoning capability and forgetting.
    §4.1: 'We use MATH500 as a held-out dataset to measure the preservation of general reasoning capability.' Only one math benchmark; it does not cover other capabilities.
  • domain assumption Gemini 3 Pro LLM-as-a-judge scores correlate adequately with human judgments for SummEval.
    §4.1 and §A.2; Spearman correlations are around 0.56–0.68, adequate but not perfect.

pith-pipeline@v1.3.0-alltime-deepseek · 24512 in / 17744 out tokens · 162927 ms · 2026-08-02T00:46:11.350191+00:00 · methodology

0 comments
read the original abstract

Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.

Figures

Figures reproduced from arXiv: 2607.14895 by Mark Vero, Martin Vechev, Niels M\"undler-Sasahara, Yu-Du Feng.

Figure 1
Figure 1. Figure 1: Our core method for RLM training consists of a lightweight two-step pipeline: We first perform standard IFT on Dtrain, which produces a fine-tuned model MIFT with higher task accuracy but possibly compromised reasoning behavior and forgetting. In the shown example, Apriel 15B correctly inserts missing type casts in Rust code after fine-tuning, but does not emit any reasoning about the code. We then merge M… view at source ↗
Figure 2
Figure 2. Figure 2: We evaluate various merge ratios on the final test set. We observe that reasoning on the target task drops sharply after a merging threshold is crossed (left), while target-task performance follows a smoother curve (right). For text summarization, task performance is measured by SummEval. On Rust, the benefit of restored reasoning for task performance is clearly visible. fully fine-tuning all model weights… view at source ↗
Figure 3
Figure 3. Figure 3: OpenThinker 7B ablations on Rust coding, from left to right: training epochs, learning rate η, batch size B, LoRA vs. full fine-tuning, and merging techniques. The overall trend of reasoning loss in standard IFT and recovery of reasoning and performance is stable across all settings. 6. Discussion Inconsistent reasoning loss In our experiments, we ob￾serve that OpenThinker 7B loses its reasoning behavior i… view at source ↗
Figure 4
Figure 4. Figure 4: The non-tuned Apriel 15B model produces a non-empty reasoning trace but fails to produce compiling Rust code because it does not recognize that the implicit type of variable low is usize, which conflicts with the target return type isize. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The IFT model emits an empty reasoning trace and then a direct correct solution by immediately casting the running variables to the target type isize (highlighted in red). 17 [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The model merged with our method produces a non-empty reasoning trace and correctly recognizes the implicit types of low and high and the potential conflict with the function signature (highlighted in blue). It thus inserts a crucial cast at the return position (highlighted in red). 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt used for both training and evaluation in the Rust code generation task. The task instruction varies per example and is inserted as {{RUST_PROMPT}}. User {{PROBLEM}} Please reason step by step, and put your final answer within \boxed{} [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt used for MATH-500 evaluation. The problem is inserted as {{PROBLEM}}. User You are an expert in writing summarization. Your task is to read the following Article and write a summary about it. Output your complete summary after the <SUMMARY> tag, like this: <SUMMARY> // your summary here Article: {{ARTICLE}} [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt used for both training and evaluation in the text summarization task. The article is inserted as {{ARTICLE}}. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt used to evaluate summary relevance with Gemini 3 Pro. The article and generated summary are inserted as {{ARTICLE}} and {{SUMMARY}}. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt used to evaluate summary fluency with Gemini 3 Pro. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt used to evaluate summary consistency with Gemini 3 Pro. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt used to evaluate summary coherence with Gemini 3 Pro. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 3 linked inside Pith

  1. [2]

    Identify the article's main idea and secondary details

  2. [3]

    Now low equals high, return 4

    is7 <8, so low becomes4. Now low equals high, return 4. So the algorithm seems correct. Now, implementing this in Rust. The function signature is given as fn left_insertion(a: Vec<isize>, x: isize) -> isize. So the code would be: Initialize low as 0, high as a.len(). Then loop while low < high. Compute mid as (low + high) / 2. But in Rust, integer divisio...

  3. [4]

    Identify if there are any secondary details or redundancy in the summary

  4. [5]

    Assign a score from 1 to 100 based on the Evaluation Criteria above. Output your detailed thought process and formal justification based on the Evaluation Criteria, and finally output your final score in the format shown below: <think> // your thought process and justification </think> <Final score> Relevance: // your final score Article: {{ARTICLE}} Summ...

  5. [7]

    Read the summary and judge if it captures the main idea of the article

  6. [11]

    Read the summary and judge whether the text is easy to read and whether the individual sentences are natural

  7. [12]

    Only output the score

    Assign a score from 1 to 5 based on the Evaluation Criteria above. Only output the score. Do not include any additional explanations or text. Use the following format: Fluency: <score> Article: {{ARTICLE}} Summary: {{SUMMARY}} Figure 11.Prompt used to evaluate summary fluency with Gemini 3 Pro. 21 Leveraging Instruction Tuning and Merging for Reasoning Mo...

  8. [14]

    Read the summary and check whether each fact is supported by the article

  9. [15]

    Only output the score

    Assign a score from 1 to 5 based on the Evaluation Criteria above. Only output the score. Do not include any additional explanations or text. Use the following format: Consistency: <score> Article: {{ARTICLE}} Summary: {{SUMMARY}} Figure 12.Prompt used to evaluate summary consistency with Gemini 3 Pro. 22 Leveraging Instruction Tuning and Merging for Reas...

  10. [16]

    Read and understand the article

  11. [17]

    Read the summary and assess whether the ideas are presented in a clear and logical order

  12. [18]

    Only output the score

    Assign a score from 1 to 5 based on the Evaluation Criteria above. Only output the score. Do not include any additional explanations or text. Use the following format: Coherence: <score> Article: {{ARTICLE}} Summary: {{SUMMARY}} Figure 13.Prompt used to evaluate summary coherence with Gemini 3 Pro. 23

  13. [2016]

    Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M.-H., Zi, Y ., Anderson, C

    URLhttps://arxiv.org/abs/1606.06565. Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L., Pinckney, D., Yee, M.-H., Zi, Y ., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022. URL https://arxiv.org/abs/2208.08227. Chen, M., Two...

    Pith/arXiv arXiv 2022

  14. [2024]

    Alexandrov, A., Raychev, V ., Müller, M

    URLhttps://arxiv.org/abs/2306.13649. Alexandrov, A., Raychev, V ., Müller, M. N., Zhang, C., Vechev, M. T., and Toutanova, K. Mitigating catastrophic forgetting in language transfer via model merging. In EMNLP (Findings), 2024. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schul- man, J., and Mané, D. Concrete problems in ai safety,

    Pith/arXiv arXiv 2024

  15. [2026]

    Zhou, X., Liu, Z., Sims, A., Wang, H., Pang, T., Li, C., Wang, L., Lin, M., and Du, C

    URLhttps://arxiv.org/abs/2605.21384. Zhou, X., Liu, Z., Sims, A., Wang, H., Pang, T., Li, C., Wang, L., Lin, M., and Du, C. Reinforcing general rea- soning without verifiers.CoRR, 2025a. Zhou, X., Liu, Z., Sims, A., Wang, H., Pang, T., Li, C., Wang, L., Lin, M., and Du, C. Reinforcing gen- eral reasoning without verifiers, 2025b. URL https: //arxiv.org/ab...

    Pith/arXiv arXiv 2021