Pith. sign in

REVIEW 3 major objections 5 minor 43 references

When contrastive candidates only differ in a few entity slots, preference training must supervise those slots—not the whole sequence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 04:23 UTC pith:H6PJN2PT

load-bearing objection Clean co-design of KG-path contrastive data and a matching slot-level preference objective; the main soft spot is that chosen/rejected labels rest on a frequency prior validated only by GPT-4 agreement and a random baseline. the 3 major comments →

arxiv 2607.03166 v1 pith:H6PJN2PT submitted 2026-07-03 cs.CL cs.AIcs.LG

KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment

Jinkyeong Choi , Chaebin Jeong , Donghyeon Park This is my paper
classification cs.CL cs.AIcs.LG
keywords Large Language ModelsReasoningDomain AdaptationPreference OptimizationKnowledge GraphResolution MismatchSlot-Parallel AlignmentContrastive Synthesis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that template-style contrastive data for language-model reasoning is scalable but structurally mismatched to how models are usually trained: candidates often share almost the entire template and differ only at a few entity slots, while sequence-level preference methods still spread the learning signal across every token. It formalizes this gap as the Resolution Mismatch Problem and proposes KARMA: build chosen-versus-rejected answers by enumerating schema-constrained paths on a domain knowledge graph, verbalizing them under a shared template so differences stay localized to intermediate entities, and ranking paths with a recurrence-based support score. Slot-Parallel Alignment (SPA) then applies a decoupled objective that puts preference and confidence losses on those entity slots while treating the shared template as ordinary language modeling. On biomedical, computer-science, and chemistry benchmarks, this co-design beats the base model and same-data supervised fine-tuning, and compares favorably with sequence-level and token-level preference methods. A sympathetic reader cares because the claim is that scaling reliable reasoning supervision is less about generating more candidates and more about matching the resolution of the loss to where the candidates actually disagree.

Core claim

Aligning the granularity of preference supervision with the entity-slot structure of knowledge-graph path candidates—via KARMA synthesis plus Slot-Parallel Alignment—produces stronger domain reasoning gains than applying sequence-level or token-level preference objectives to the same data, and stronger gains than supervised fine-tuning on either open data or the KARMA chosen answers alone.

What carries the argument

Slot-Parallel Alignment (SPA): a decoupled slot-level objective on packed, template-sharing candidates that routes preference and anchor-confidence losses to construction-exposed entity slots, keeps a language-modeling loss on shared template tokens, and optionally uses slot-aware masked attention to approximate per-candidate log-likelihoods in one forward pass.

Load-bearing premise

That ranking paths by how often their intermediate entities recur in a schema-constrained pool is a trustworthy way to label one path as chosen and the others as rejected.

What would settle it

Hold the knowledge graph, schemas, templates, and SPA objective fixed, replace support-based top-K path selection with random or reverse-support selection, and check whether the reported gains over same-data SFT and over DPO/MDPO/TDPO disappear on the same biomedical, chemistry, and computer-science benchmarks.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Same-data sequence-level SFT on template-sharing candidates can underperform because supervision collapses onto repeated template tokens rather than discriminative entities.
  • Preference methods that only refine token importance still leave residual mismatch when candidates differ only at known entity slots by construction.
  • Domain knowledge graphs with typed multi-hop schemas become a practical generator of slot-aligned contrastive reasoning data without hand-writing many templates.
  • Training efficiency can trade a small accuracy cost for fewer forward passes by packing slot variants under a slot-aware attention mask without changing the supervision signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If recurrence support is only a weak proxy for factual correctness, hybrid ranking that mixes support with retrieval or judge scores could further clean the chosen-rejected labels without changing SPA.
  • The same slot-resolution idea may transfer to other structured generators—tables, programs, or tool traces—wherever alternatives share a scaffold and differ in named slots.
  • Domains without a ready multi-hop knowledge graph will need automatic graph construction or hybrid retrieval before KARMA-style co-design applies.
  • Because all reported runs use one 7B instruction model with LoRA, a natural next test is whether slot-level resolution still dominates sequence- and token-level objectives at larger scales or other model families.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper identifies a Resolution Mismatch Problem: template-based contrastive candidates often differ only at a few entity slots, yet sequence-level preference objectives spread supervision over mostly shared tokens. It proposes KARMA, which enumerates schema-constrained multi-hop paths on domain knowledge graphs, selects top-K paths by intermediate-entity recurrence under an overlap budget, and verbalizes them under a shared template so differences are localized to entity slots. Slot-Parallel Alignment (SPA) then applies a decoupled hybrid objective (slot-level squared-margin preference, anchor confidence, and template LM loss), with optional slot-aware masked attention for packed evaluation. On biomedical, CS, and chemistry benchmarks with Qwen2.5-7B-Instruct + LoRA, KARMA is reported to beat the base model and same-data SFT, and to compare favorably with DPO, multi-rejected DPO, and token-level DPO, with ablations on path selection, objective resolution, and packing.

Significance. If the co-design claim holds, the work offers a practical recipe for domain reasoning supervision that turns a known failure mode of template contrastive data into an explicit interface for preference learning, without human process labels or teacher distillation. The path-selection ablation (Table 2, +16.5 avg on biomedical) and objective-resolution ablation (Table 3) are useful evidence that both the structural prior and the slot-level objective matter. The optional packing analysis (Table 4) is a concrete efficiency contribution. Strengths include a clear problem framing, end-to-end co-design of data and loss, multiple controlled ablations on the same KARMA candidates, and an explicit limitations section on KG coverage and schema expressivity. The main significance is methodological rather than theoretical: a falsifiable recipe for slot-aligned synthetic preference data in KG-rich domains.

major comments (3)
  1. [Section 3.2.2, Algorithm 2, Appendix E] Section 3.2.2 and Algorithm 2 define the chosen path as the support-maximizing path under overlap budget δ, where support is the sum of intermediate-entity frequencies inside the same source–target pool. Appendix E reports only GPT-4 agreement of 74.8% with this ranking on 10k pools, and Table 2 compares only against random sampling. Neither establishes that the chosen intermediate entities are factually or clinically more correct than rejected ones. Because SPA routes supervision exactly to those slots (Eqs. 5–7), mislabeled popularity could be learned as “reasoning.” A load-bearing addition is an external check (expert subsample, KG-grounded correctness of intermediates, or correlation of support with an independent factuality metric) or a clear reframing that the method optimizes structural recurrence rather than verified reasoning chains.
  2. [Abstract; Table 1; Section 4.2] The abstract and §4.2 claim that KARMA “outperforms base LLM and same-data SFT baselines” across the evaluated domains, but Table 1 is mixed: base LLM exceeds KARMA on GPQA-Bio (0.700 vs 0.667), and Open-Data SFT exceeds KARMA on CSBench (0.506 vs 0.495). Gains are large on several reasoning-heavy sets (e.g., MedQA, MMLU-Pro-Bio) but near-flat on PubMedQA. The central co-design claim remains plausible, yet the prose should be tightened to “majority of benchmarks / largest gains on multi-hop reasoning” and should discuss the GPQA-Bio regression rather than averaging it away.
  3. [Section 4.1; Table 3; Section 6] All main and ablation results use a single base model (Qwen2.5-7B-Instruct) with LoRA. Table 3’s claim that slot-level resolution beats sequence- and token-level objectives is therefore conditioned on one architecture and adapter setup. Given that the paper’s contribution is co-design of data resolution and objective resolution, at least one additional model family or scale (or a clear negative result) is needed before the resolution-alignment conclusion can be treated as method-level rather than model-specific. The Limitations section notes this; it should be treated as a required experiment or a hard scope restriction in the claims.
minor comments (5)
  1. [Section 3.3.3, Eq. (6)] Eq. (6) uses a squared-margin preference loss rather than the usual logistic DPO form. A short justification (stability under multi-rejected slots, interaction with length-normalized scores) would help readers map SPA to the DPO family.
  2. [Tables 1–3; Appendix B.2] Tables report means over 5 seeds but no standard deviations or significance tests. Adding error bars or ±std in Tables 1–3 would make small margins (e.g., MMLU-Pro-CS 0.256 tied with TDPO) easier to interpret.
  3. [Figure 1; Figure 2] Figure 1 and Figure 2 are dense; the “Resolution Mismatch” panel would benefit from a single concrete token-level example (shared template tokens vs. differing entity spans) next to the loss cartoons.
  4. [Title page] Title line breaks and spacing (“KNOWLEDGE GRAPH-BASEDAUTOMATED”, “REASONINGMATERIALIZATION”) should be fixed in the camera-ready PDF.
  5. [Section 4.2; Table 9] Clarify whether Open-Data SFT uses the same number of training tokens/samples as KARMA-Data SFT; otherwise the Table 1 contrast partly confounds data volume with structure.

Circularity Check

0 steps flagged

No circularity: empirical co-design paper; preference labels and claims are not forced by construction or self-citation.

full rationale

KARMA/SPA is an empirical methods paper, not a first-principles derivation. The load-bearing chain is: (i) observe that template-based contrastive candidates differ mainly at entity slots while sequence-level DPO dilutes signal (framing, not a theorem); (ii) construct candidates by schema-constrained KG path enumeration plus a training-free support heuristic (entity recurrence + overlap budget δ; Alg. 2 / §3.2.2); (iii) optimize a designed slot-level hybrid loss (Eqs. 5–9) on those candidates; (iv) measure accuracy on held-out public benchmarks (MedQA, MMLU/MMLU-Pro, GPQA, CSBench, PubMedQA). Preference labels come from the KG recurrence statistic, not from the policy being trained, so the training objective does not define its own targets. Support is proposed as a structural prior and checked externally (GPT-4 agreement 74.8% on 10k pools, Appendix E; random-selection ablation, Table 2; same-data SFT/DPO/MDPO/TDPO comparisons, Tables 1 and 3)—these are empirical controls, not self-definitional reductions. There is no fitted parameter renamed as a prediction, no uniqueness theorem imported from the authors, no load-bearing self-citation chain, and no renaming of a known closed-form result as a derived law. Concerns that support may track popularity rather than factual correctness of intermediates are correctness/validity risks, not circularity. The paper is self-contained against external benchmarks; score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The central performance claim rests on a small set of free loss weights chosen by grid search, on the modeling assumption that entity recurrence is a valid preference prior, and on the construction that schema-constrained paths under a shared template localize differences to entity slots. No new physical entities are postulated; the invented constructs are methodological.

free parameters (5)
  • λ_anchor = 0.5
    Weight on the anchor-confidence term in L_SPA; selected as 0.5 via sensitivity grid on biomedical (Table 14) and then frozen for all domains.
  • λ_template = 0.1
    Weight on the shared-template LM loss; selected as 0.1 on the same biomedical grid.
  • β (margin) = 0.1
    Target margin inside the squared-margin slot preference loss (Eq. 6); set to 0.1.
  • K (candidates per pool) = 4
    Number of paths retained after support-based selection; fixed at 4 for all reported experiments.
  • δ (overlap budget) = 1
    Maximum shared intermediate entities allowed when greedily admitting paths; set to 1 in the academic-domain generator.
axioms (4)
  • domain assumption Within a schema-constrained source-target pool, the cumulative recurrence of intermediate entities is a reliable structural prior for ranking paths as chosen versus rejected.
    Load-bearing for the preference labels used by SPA; justified only by GPT-4 agreement (74.8%) and the random-selection ablation (Section 3.2.2, Appendix E).
  • standard math Verbalizing paths that share source, target, and relation schema under one fixed template localizes all contrastive differences to intermediate entity slots.
    True by construction of the synthesis pipeline (Section 3.2.3); the rest of SPA is designed around this property.
  • domain assumption A domain knowledge graph with typed multi-hop connectivity sufficient for the chosen schemas exists and is free of systematic factual error that would invert chosen/rejected labels.
    Stated as a limitation (Section 6); without it the synthesis pipeline cannot run.
  • domain assumption The target reasoning capability is expressible as discrete entity-slot structures under an explicit schema and verbalization template.
    Explicit limitation; open-ended generation is out of scope (Section 6).
invented entities (3)
  • Resolution Mismatch Problem no independent evidence
    purpose: Names the gap between sequence-level preference optimization and candidates that differ only at a few entity slots.
    Framing device introduced in the introduction and Fig. 1; not an independently measured phenomenon outside the paper's own data construction.
  • Slot-Parallel Alignment (SPA) no independent evidence
    purpose: Decoupled hybrid objective (slot preference + anchor + template LM) plus optional slot-aware mask that matches supervision resolution to KARMA candidates.
    Core methodological invention; evaluated only inside this paper's experiments.
  • KARMA synthesis pipeline no independent evidence
    purpose: End-to-end procedure that turns schema-constrained KG paths into ordered, slot-aligned contrastive verbalizations.
    New combination of known KG-path and template techniques; no external replication yet.

pith-pipeline@v1.1.0-grok45 · 23660 in / 3670 out tokens · 39973 ms · 2026-07-12T04:23:32.041003+00:00 · methodology

0 comments
read the original abstract

Template-based contrastive synthesis is scalable, but its candidates often differ only in a few entity-slots while sequence-level optimization spreads supervision over mostly shared templates. We formalize this as the Resolution Mismatch Problem and propose KARMA, which enumerates schema-constrained paths over domain knowledge graphs and verbalizes them into slot-aligned contrastive candidates. Slot-Parallel Alignment (SPA) then applies a decoupled slot-level objective to route preference supervision to discriminative entity-slots, with slot-aware masked attention serving as an optional packed-evaluation implementation. Across biomedical, computer-science, and chemistry benchmarks, KARMA outperforms base LLM and same-data SFT baselines, and compares favorably with sequence and token-level preference methods.

Figures

Figures reproduced from arXiv: 2607.03166 by Chaebin Jeong, Donghyeon Park, Jinkyeong Choi.

Figure 1
Figure 1. Figure 1: The Resolution Mismatch Problem in preference optimization. (a) DPO compares a single chosen-rejected [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of KARMA and SPA. KARMA synthesizes structurally aligned contrastive candidates from [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Slot-aware masked attention for packed candidate evaluation. Green cells denote allowed attention, red cells [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 5 linked inside Pith

  1. [1]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances in Neural Information Processing Systems, 2022

    2022

  2. [2]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Solving math word problems with process- and outcome-based feedback

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe twelfth international conference on learning representations, 2023

    2023

  6. [6]

    Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations

    Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9426–9439, 2024

    2024

  7. [7]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2023

  8. [8]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  9. [9]

    The false promise of imitating proprietary llms, 2023

    Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023

    2023

  10. [10]

    AI models collapse when trained on recursively generated data.Nature, 631(8022):755–759, 2024

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. AI models collapse when trained on recursively generated data.Nature, 631(8022):755–759, 2024

    2024

  11. [11]

    Meta- Synth: Meta-prompting-driven agentic scaffolds for diverse synthetic data generation

    Haris Riaz, Sourav Sanjukta Bhabesh, Vinayak Arannil, Miguel Ballesteros, and Graham Horwood. Meta- Synth: Meta-prompting-driven agentic scaffolds for diverse synthetic data generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Findings of the Association for Computa- tional Linguistics: ACL 2025, pages 18770–...

    2025

  12. [12]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems, 2023

    2023

  13. [13]

    From r to q∗: Your language model is secretly a Q-function.arXiv preprint arXiv:2404.12358, 2024

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q∗: Your language model is secretly a Q-function.arXiv preprint arXiv:2404.12358, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Token-level direct preference optimization

    Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. InProceedings of the 41st International Conference on Machine Learning, 2024

    2024

  15. [15]

    T-REG: Preference optimization with token-level reward regularization

    Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, and Tao Meng. T-REG: Preference optimization with token-level reward regularization. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025

    2025

  16. [16]

    Paths-over-graph: Knowledge graph empowered large language model reasoning

    Xingyu Tan, Xiaoyang Wang, Qing Liu, Xiwei Xu, Xin Yuan, and Wenjie Zhang. Paths-over-graph: Knowledge graph empowered large language model reasoning. InProceedings of the ACM on Web Conference 2025, WWW ’25, page 3505–3522, New York, NY , USA, 2025. Association for Computing Machinery

    2025

  17. [17]

    Ni, Heung-Yeung Shum, and Jian Guo

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. InThe Twelfth International Conference on Learning Representations, 2024. 10

    2024

  18. [18]

    Reasoning on graphs: Faithful and interpretable large language model reasoning

    Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  19. [19]

    Pan, Wen Zhang, and Huajun Chen

    Junjie Wang, Mingyang Chen, Binbin Hu, Dan Yang, Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff Z. Pan, Wen Zhang, and Huajun Chen. Learning to plan for retrieval-augmented large language models from knowledge graphs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the Association for Computational Lingui...

    2024

  20. [21]

    The WebNLG challenge: Generating text from RDF data

    Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. In Jose M. Alonso, Alberto Bugarín, and Ehud Reiter, editors,Proceedings of the 10th International Conference on Natural Language Generation, pages 124–133, Santiago de Compostela, Spain, September 2017. Association for Co...

    2017

  21. [22]

    Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training

    Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings of the 2021 Confer...

    2021

  22. [23]

    Selective preference optimization via token-level reward function estimation

    Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. Selective preference optimization via token-level reward function estimation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natu- ral Language Processing, pages 7032–70...

    2025

  23. [24]

    SparsePO: Controlling preference alignment of LLMs via sparse token masks

    Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou Ammar, and Jun Wang. SparsePO: Controlling preference alignment of LLMs via sparse token masks. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25477–25503, Suzhou, C...

    2025

  24. [25]

    Token-importance guided direct preference optimization

    Ning Yang, Hai Lin, Yibo Liu, Baoliang Tian, Guoqing Liu, and Haijun Zhang. Token-importance guided direct preference optimization. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  25. [26]

    Yu, and Meng Cao

    Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Xiaoming Simon Wang, Jiulong Shan, Al- bin Madappally Jose, Xiaojiang Liu, Lijie Wen, Philip S. Yu, and Meng Cao. TIS-DPO: Token-level importance sampling for direct preference optimization with estimated weights. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  26. [27]

    WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    Pith/arXiv arXiv 2023

  27. [28]

    Generating uncontextualized and contextualized questions for document-level event argument extraction

    Md Nayem Uddin, Enfa Rose George, Eduardo Blanco, and Steven Corman. Generating uncontextualized and contextualized questions for document-level event argument extraction. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2024

  28. [29]

    Rethinking DPO: The role of rejected responses in preference misalignment

    Jae Hyeon Cho, JunHyeok Oh, Myunsoo Kim, and Byung-Jun Lee. Rethinking DPO: The role of rejected responses in preference misalignment. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors,Findings of the Association for Computational Linguistics: EMNLP 2025, pages 8159–8176, Suzhou, China, November 2025. Association f...

    2025

  29. [30]

    Preference ranking optimization for human alignment

    Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Arti...

    2024

  30. [31]

    LiPO: Listwise preference optimization through learning-to-rank

    Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J Liu, and Xuanhui Wang. LiPO: Listwise preference optimization through learning-to-rank. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Proceedings of the 2025 Conference of the Nations of the Americas Chapter of ...

    2025

  31. [32]

    TGDPO: Harnessing token- level reward guidance for enhancing direct preference optimization

    Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, and Jiaya Jia. TGDPO: Harnessing token- level reward guidance for enhancing direct preference optimization. InProceedings of the 42nd International Conference on Machine Learning, 2025

    2025

  32. [33]

    OCEAN: Offline chain-of-thought evaluation and alignment in large language models

    Junda Wu, Xintong Li, Ruoyu Wang, Yu Xia, Yuxin Xiong, Jianing Wang, Tong Yu, Xiang Chen, Branislav Kveton, Lina Yao, Jingbo Shang, and Julian McAuley. OCEAN: Offline chain-of-thought evaluation and alignment in large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  33. [34]

    GLaM: Fine- tuning large language models for domain knowledge graph alignment via neighborhood partitioning and generative subgraph encoding

    Stefan Dernbach, Khushbu Agarwal, Alejandro Zuniga, Michael Henry, and Sutanay Choudhury. GLaM: Fine- tuning large language models for domain knowledge graph alignment via neighborhood partitioning and generative subgraph encoding. InProceedings of the AAAI Symposium Series, 2024

    2024

  34. [35]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  35. [36]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J...

    2024

  36. [37]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14), 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14), 2021

    2021

  37. [38]

    PubMedQA: A dataset for biomedi- cal research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedi- cal research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro...

    2019

  38. [39]

    Association for Computational Linguistics

  39. [40]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  40. [41]

    CS-bench: A comprehensive benchmark for large language models towards computer science mastery

    Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, and Weiran Xu. CS-bench: A comprehensive benchmark for large language models towards computer science mastery. InThe Thirteenth International Conference on Learning ...

    2025

  41. [42]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning.IEEE Transactions on Audio, Speech and Language Processing, 2025

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.IEEE Transactions on Audio, Speech and Language Processing, 2025

    2025

  42. [43]

    Do models really learn to follow instructions? an empirical study of instruction tuning

    Po-Nien Kung and Nanyun Peng. Do models really learn to follow instructions? an empirical study of instruction tuning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1317–1328, Toronto, Canada, July 2023. Association for...

    2023

  43. [44]

    Graph chain-of-thought: Augmenting large language models by reasoning on graphs

    Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics: ACL 2024, pages 163–...

    2024