arxiv: 2604.13538 · v1 · submitted 2026-04-15 · 💻 cs.CL

Recognition: unknown

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Tatsuya Ichinose , Youmi Ma , Masanari Oi , Ryuto Koike , Naoaki Okazaki

Authors on Pith no claims yet

Pith reviewed 2026-05-10 14:08 UTC · model grok-4.3

classification 💻 cs.CL

keywords instruction tuningcontrastive decodingdataset synthesislarge language modelspost-trainingpre-trained knowledgechat vector

0 comments

The pith

Contrastive decoding between post-trained and pre-trained LLMs isolates instruction-following behavior to synthesize superior instruction-tuning datasets.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that responses generated by LLMs for instruction tuning mix pre-training world knowledge with post-training instruction skills, which reduces their effectiveness for tuning. It introduces CoDIT, a method that uses contrastive decoding between a post-trained model and its pre-trained version to suppress the shared knowledge and amplify pure instruction-following behavior in the generated responses. Experiments show that models trained on these CoDIT datasets outperform those trained on directly generated responses as well as existing public instruction-tuning datasets across multiple benchmarks. The approach is also shown to distill the chat vector from parameter space into text space, enabling transfer of instruction capabilities across different model architectures. A reader would care because this offers a targeted way to improve the quality of synthetic training data for better LLM alignment and performance.

Core claim

CoDIT applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. This suppresses pre-trained knowledge shared between the models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses and existing publicly available instruction-tuning datasets across multiple benchmarks. CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning across

What carries the argument

Contrastive decoding between a post-trained model and its pre-trained counterpart during response generation, which suppresses shared pre-trained knowledge and amplifies post-training instruction-following behavior.

Load-bearing premise

Contrastive decoding between the post-trained and pre-trained models cleanly suppresses only pre-trained knowledge without introducing new biases, hallucinations, or loss of useful information.

What would settle it

If identical models trained on CoDIT-generated datasets show no consistent outperformance over models trained on directly generated responses when evaluated on held-out benchmarks, the central claim would be falsified.

Figures

Figures reproduced from arXiv: 2604.13538 by Masanari Oi, Naoaki Okazaki, Ryuto Koike, Tatsuya Ichinose, Youmi Ma.

**Figure 2.** Figure 2: Score distribution of the synthetic dataset evaluated by gpt-oss-120b on a scale of 1 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Cosine similarity between the teacher’s chat vector [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Score distribution of synthesized responses using Qwen3-8B. Consistent with [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗

**Figure 5.** Figure 5: Hyperparameter tuning for α based on MT-Bench scores. The average performance of student models (Qwen3-8B-Base and Llama-3.1-8B) is plotted against different α values for each teacher model. The optimal α is selected where the MT-Bench score is maximized. B Training Details All training was conducted on four NVIDIA H100 SXM5 GPUs, utilizing DeepSpeed ZeRO (Rajbhandari et al., 2020) for memory optimization.… view at source ↗

**Figure 6.** Figure 6: , demonstrate that both student models achieved consistently higher scores across all tested values of α compared to the baselines. This indicates that the improvement is primarily driven by the core mechanism of CoDIT rather than exhaustive hyperparameter optimization, confirming the robustness of our approach. D Evaluation Prompts for WildBench Due to several grammatical and contextual errors identified … view at source ↗

**Figure 7.** Figure 7: Cosine similarity between the parameter updates and the teacher’s chat vector for [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗

read the original abstract

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes CoDIT, which applies contrastive decoding between a post-trained LLM and its pre-trained counterpart to generate instruction responses. This is hypothesized to suppress shared pre-trained knowledge while amplifying post-training instruction-following behavior, yielding synthetic datasets that produce stronger instruction-tuned models than direct generation or existing public datasets. The authors further claim a theoretical and empirical interpretation of CoDIT as distilling a 'chat vector' from parameter space to text space for cross-architecture transfer.

Significance. If the empirical outperformance claims hold under rigorous validation, the method could provide an efficient, training-free approach to synthesizing higher-quality instruction data by leveraging existing model pairs. The chat-vector framing offers a potentially useful lens for understanding and transferring post-training effects, which may influence future work on alignment and capability isolation.

major comments (3)

[§4] §4 (Experiments): The central claim of consistent outperformance over direct generation and public datasets is asserted without reported model sizes, dataset scales, number of evaluation runs, error bars, or statistical tests, rendering the quantitative results difficult to assess for robustness or effect size.
[§3] §3 (CoDIT Method): The load-bearing assumption that contrastive decoding cleanly isolates instruction-following capabilities without suppressing useful world knowledge or introducing decoding artifacts (hallucinations, reduced diversity) lacks direct supporting measurements on the generated responses themselves, such as factuality or instruction-adherence metrics.
[§5] §5 (Theoretical Interpretation): The claim that CoDIT distills the chat vector to text space is presented as both theoretical and empirical, yet the connection between the contrastive objective and the vector distillation appears interpretive rather than derived from a formal equivalence or proof; this framing requires a clearer mathematical link to be load-bearing for the cross-architecture transfer result.

minor comments (3)

[§3] Notation for contrastive decoding hyperparameters (e.g., scaling factors) is introduced without an explicit table or equation reference, which could be clarified for reproducibility.
[§4] Benchmark comparison figures would benefit from including variance across seeds or runs to support the 'consistent' outperformance narrative.
The related-work discussion on synthetic data generation and contrastive methods could cite additional recent papers on logit manipulation for alignment.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments, which have helped us identify areas for improvement in the presentation of our results and theoretical framing. We address each major comment below.

read point-by-point responses

Referee: [§4] §4 (Experiments): The central claim of consistent outperformance over direct generation and public datasets is asserted without reported model sizes, dataset scales, number of evaluation runs, error bars, or statistical tests, rendering the quantitative results difficult to assess for robustness or effect size.

Authors: We agree that the experimental details were insufficiently reported, making it challenging to evaluate the robustness of our claims. In the revised version, we will report the specific model sizes used, the scale of the generated datasets, the number of evaluation runs performed, include error bars representing standard deviations, and conduct statistical significance tests. These additions will be incorporated into Section 4. revision: yes
Referee: [§3] §3 (CoDIT Method): The load-bearing assumption that contrastive decoding cleanly isolates instruction-following capabilities without suppressing useful world knowledge or introducing decoding artifacts (hallucinations, reduced diversity) lacks direct supporting measurements on the generated responses themselves, such as factuality or instruction-adherence metrics.

Authors: This is a valid point; while our primary evidence is the improved downstream performance of models trained on CoDIT data, direct validation on the synthetic responses would strengthen the hypothesis. We will add in the revision an evaluation of the generated responses using automatic metrics for factuality and instruction adherence (e.g., using an LLM-as-a-judge approach). We will also report diversity metrics to address potential artifacts. This analysis will be included in Section 3. revision: yes
Referee: [§5] §5 (Theoretical Interpretation): The claim that CoDIT distills the chat vector to text space is presented as both theoretical and empirical, yet the connection between the contrastive objective and the vector distillation appears interpretive rather than derived from a formal equivalence or proof; this framing requires a clearer mathematical link to be load-bearing for the cross-architecture transfer result.

Authors: We acknowledge that the connection is conceptual and empirical rather than a strict formal proof. In the revised manuscript, we will revise Section 5 to present it more clearly as an interpretive framework, highlighting the mathematical similarity between the contrastive decoding objective and the chat vector subtraction, supported by the empirical cross-architecture transfer results. We will add a more detailed explanation of the link while noting its interpretive nature. revision: partial

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper defines CoDIT explicitly as contrastive decoding applied between a post-trained LLM and its pre-trained base to suppress shared pre-training knowledge while amplifying post-training instruction-following behavior. This definition stands independently of the later empirical results or the interpretive claim that the process distills a 'chat vector' into text space. No equations or steps reduce the output dataset to a fitted parameter or self-referential construction; performance gains are measured against external baselines and public datasets rather than being forced by the method's own inputs. The chat-vector framing is presented as an after-the-fact theoretical and empirical observation, not a load-bearing premise that defines the method by construction. No self-citation chains, ansatzes smuggled via citation, or renaming of known results appear as central to the argument.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the domain assumption that the difference between pre-trained and post-trained models can be isolated by contrastive decoding to produce purer instruction data; no free parameters or new entities with independent evidence are introduced in the abstract.

axioms (1)

domain assumption Contrastive decoding between post-trained and pre-trained models suppresses shared pre-trained knowledge while amplifying instruction-following behavior acquired during post-training.
This is the core hypothesis invoked to justify the method and its expected benefit.

invented entities (1)

chat vector no independent evidence
purpose: Represents instruction-tuning capabilities in parameter space that CoDIT distills into text space.
Introduced as an interpretive lens for the method; no independent falsifiable evidence is supplied in the abstract.

pith-pipeline@v0.9.0 · 5505 in / 1369 out tokens · 38297 ms · 2026-05-10T14:08:48.693447+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 6 canonical work pages · 2 internal anchors

[1]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

URL https://lmsys.org/blog/2023-03-30-vicuna/ . https://lmsys.org/blog/ 2023-03-30-vicuna/. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. InThe Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://ope...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.12948 2023
[2]

URLhttps://openreview.net/forum?id=CybBmzWBX0. Team Olmo : Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, et al. Olmo
[3]

Olmo 3

arXiv:2512.13961, 2025. URLhttps://arxiv.org/abs/2512.13961. Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, and Dahua Lin. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. InForty-first International Conference on Machine Learnin...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2025.emnlp-main 2025
[4]

Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee

URLhttps://aclanthology.org/2025.emnlp-main.412/. Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee. Chat vector: A simple approach to equip LLMs with instruction following and model alignment in new languages. InProceedings of the 62nd Annual Meeting of the Association for Computational L...

work page doi:10.18653/v1/2024.acl-long.590 2025
[5]

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A

URLhttps://openreview.net/forum?id=IZDiRbVSVN. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023. URLhttps://arxiv.org/abs/2311.10702. Houcheng Jiang, Junfeng Fan...

work page doi:10.18653/v1/2025.findings-acl.343 2023
[6]

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He

URLhttps://openreview.net/forum?id=HPuSIXJaa9. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. InThe International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. URL https: //arxiv.org/abs/1910.02054. Guillaume Sanchez, Alexander Spa...

work page doi:10.18653/v1/2024.findings-naacl.235 2020
[7]

s t r e n g t h s \

URLhttps://openreview.net/forum?id=Pnk7vMbznK. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505. 09388. Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. MAmmoT...

work page doi:10.18653/v1/2025.findings-naacl.459 2025