Pith. sign in

REVIEW 1 major objections 2 cited by

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

T0 review · 1 major / 0 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Skill-RM reformulates reward modeling as execution of a reusable Reward-Evaluation Skill to unify heterogeneous criteria.

desk verdict Skill-RM frames reward modeling as execution of a reusable agent skill to unify rule-based, reference, and rubric signals, but the abstract supplies no metrics or implementation details to check whether the gains are real. read the letter →

arxiv 2606.03980 v1 pith:G7RBLVKP submitted 2026-06-02 cs.LG cs.CL

Tao Chen , Gangwei Jiang , Pengyu Cheng , Siyuan Huang , Yihao Liu , Jingwei Ni , Jiaqi Guo , Mengyu Zhou
show 5 more authors
Kai Tang Junling Liu Qinliang Su Xiaoxi Jiang Guanjun Jiang
This is my paper · ORCID
classification cs.LGcs.CL
keywords rewardmodelagentskillheterogeneousevaluationLLMpost-trainingreinforcementlearningbest-of-Nselectionbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Skill-RM as a way to handle the variety of evaluation signals used in reward models for LLM training. Instead of separate rule checkers, references, checklists, and rubrics, it casts reward computation as one structured agent task that a single reusable skill can run. The skill selects and combines evidence on the fly for each input. Experiments report better results than standard judge models on reward benchmarks plus downstream uses such as best-of-N selection and reinforcement learning. The central idea is that an agentic interface can deliver consistent, transparent scoring across task types.

What carries the argument

The Reward-Evaluation Skill, a reusable agentic module that dynamically selects and aggregates evidence from rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics.

What would settle it

A controlled test set of new heterogeneous criteria where Skill-RM produces lower agreement with human labels or lower downstream task performance than the strongest single-criterion baseline.

Watch

Extended reading notes

Core claim

Skill-RM supplies a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. Treating reward computation as a structured agentic task gives a consistent interface for orchestrating heterogeneous resources and dynamically selecting and aggregating evidence tailored to each input, which yields consistency and transparency across diverse tasks.

Load-bearing premise

Reformulating reward computation as execution of a reusable Reward-Evaluation Skill will integrate heterogeneous evidence types while preserving or improving evaluation quality without introducing new inconsistencies or selection biases.

Editorial extensions

If this is right

  • Skill-RM delivers higher scores than traditional judge baselines on standard reward benchmarks.
  • The same model improves best-of-N selection quality when used as the ranking signal.
  • Reinforcement learning pipelines obtain stronger training signals from the dynamically orchestrated evidence.
  • Evaluation becomes consistent and transparent across tasks that previously required separate verifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The agentic interface could be applied to other LLM evaluation settings that mix rules and rubrics, such as safety classifiers.
  • A single skill might replace collections of task-specific verifiers in large-scale alignment pipelines.
  • Dynamic evidence selection raises the possibility of measuring which evidence types contribute most to final scores on different domains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The manuscript proposes Skill-RM, a framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. This agentic approach provides a consistent interface for dynamically selecting and aggregating heterogeneous evidence (rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics) to produce reward signals for LLM post-training. The paper claims that this yields superior performance over traditional judge baselines on reward benchmarks and in downstream applications including best-of-N selection and reinforcement learning, with the code released at a public repository.

Significance. If the reported gains are reproducible and not artifacts of post-hoc choices, the work could offer a practical unification of disparate reward evaluation methods, reducing the need for task-specific verifiers in RFT and RL pipelines. The agentic formulation is a conceptual contribution that may generalize beyond the evaluated settings.

major comments (1)
  1. [Abstract] Abstract: the central claim of consistent outperformance on reward benchmarks and downstream tasks is asserted without any reported metrics, baseline names, dataset sizes, ablation results, or statistical significance tests. This absence prevents verification that the gains derive from the proposed orchestration mechanism rather than implementation details or evaluation choices.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the review and the opportunity to clarify the presentation of our results. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim of consistent outperformance on reward benchmarks and downstream tasks is asserted without any reported metrics, baseline names, dataset sizes, ablation results, or statistical significance tests. This absence prevents verification that the gains derive from the proposed orchestration mechanism rather than implementation details or evaluation choices.

    Authors: The abstract is written as a high-level summary, consistent with standard practice for concise overviews. The full manuscript (Sections 4 and 5) reports the requested details: concrete metrics on multiple reward benchmarks, comparisons against named traditional judge baselines, dataset sizes and splits, ablation studies isolating the contribution of dynamic evidence orchestration, and statistical significance testing. These results support that the observed gains stem from the agentic formulation rather than implementation artifacts. We are willing to incorporate one or two key quantitative highlights into the abstract in a revision if the editor prefers a more results-oriented abstract. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

The paper proposes Skill-RM as a methodological reformulation of reward modeling into an agentic skill-execution task, with performance claims resting on external benchmark experiments rather than any internal derivation chain. No equations, fitted parameters, self-citations, or uniqueness theorems appear in the abstract or described full text that reduce outputs to inputs by construction. The framework description and experimental results stand as independent content.

Assumptions & free parameters 0 free parameters · 0 assumptions · 1 invented entities

Abstract introduces the Reward-Evaluation Skill as a core construct without detailing supporting axioms or free parameters; no invented entities beyond the skill itself are described.

invented entities (1)
  • Reward-Evaluation Skill
    purpose: Reusable agent skill that dynamically selects and aggregates heterogeneous evidence for reward computation
    Core new component introduced to unify evaluation criteria; no independent evidence supplied in abstract

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill." pith.science (2026). https://pith.science/paper/G7RBLVKP

@misc{pith2026260603980,
  author       = {Pith},
  title        = {Pith review of: Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7RBLVKP}},
  note         = {Machine review of arXiv:2606.03980}
}
read the original abstract

Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored. To this end, we propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill. By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources, dynamically selecting and aggregating evidence tailored to the specific requirements of each input. This approach enables the reward model to move beyond static evaluation, ensuring consistency and transparency across diverse tasks. Extensive experiments on reward benchmarks and downstream applications, including best-of-N selection and reinforcement learning, demonstrate that Skill-RM consistently outperforms traditional judge baselines. Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence. The code is at https://github.com/Qwen-Applications/Skill-RM.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

    cs.AI 2026-08 conditional novelty 6.0 of 10

    SkillTV-Bench provides a multi-domain, skill-aware trajectory verification benchmark, and SkillTV-Evolve's evolved JudgeSkill improves an agent judge's accuracy by 14.8 points on a held-out set.

  2. Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Skill Self-Play uses an evolving skill library to guide LLM self-training, improving tool-call and reasoning accuracy beyond unguided self-play on five model backbones.

Reference graph

Works this paper leans on

70 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    On diversified preferences of large language model alignment , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  2. [2]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Training a helpful and harmless assistant with reinforcement learning from human feedback , author=. arXiv preprint arXiv:2204.05862 , year=

    work page Pith review arXiv

  3. [3]

    Search Self-play: Pushing the Frontier of Agent Capability without Supervision

    Search Self-Play: Pushing the Frontier of Agent Capability without Supervision , author =. The Fourteenth International Conference on Learning Representations , year =. 2510.18821 , archivePrefix =

    work page Pith review arXiv

  4. [4]

    2026 , booktitle =

    Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance , author=. 2026 , booktitle =

    2026

  5. [5]

    Findings of the Association for Computational Linguistics , year=

    Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game , author=. Findings of the Association for Computational Linguistics , year=

  6. [6]

    Everyone deserves a reward: Learning customized human preferences.arXiv preprint arXiv:2309.03126, 2023

    Everyone deserves a reward: Learning customized human preferences , author=. arXiv preprint arXiv:2309.03126 , year=

    work page arXiv

  7. [7]

    Advances in Neural Information Processing Systems , volume =

    Training Language Models to Follow Instructions with Human Feedback , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  8. [8]

    Advances in Neural Information Processing Systems , volume =

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author =. Advances in Neural Information Processing Systems , volume =. 2023 , url =

    2023

Show all 70 references
  1. [9]

    Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =

    A General Theoretical Paradigm to Understand Learning from Human Preferences , author =. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , pages =. 2024 , volume =

    2024

  2. [10]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and Zhang, Hao and Gonzalez, Joseph and Stoica, Ion , booktitle =. Judging. 2023 , url =

    2023

  3. [11]

    and Hajishirzi, Hannaneh , booktitle =

    Lambert, Nathan and Pyatkin, Valentina and Morrison, Jacob and Miranda, LJ and Lin, Bill Yuchen and Chandu, Khyathi and Dziri, Nouha and Kumar, Sachin and Zick, Tom and Choi, Yejin and Smith, Noah A. and Hajishirzi, Hannaneh , booktitle =. 2025 , address =. doi:10.18653/v1/202...

    2025 doi

  4. [12]

    and Hajishirzi, Hannaneh and Lambert, Nathan , booktitle =

    Malik, Saumya and Pyatkin, Valentina and Land, Sander and Morrison, Jacob and Smith, Noah A. and Hajishirzi, Hannaneh and Lambert, Nathan , booktitle =. 2026 , url =

    2026

  5. [13]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , month = nov, year =

    Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts , author =. Findings of the Association for Computational Linguistics: EMNLP 2024 , month = nov, year =. doi:10.18653/v1/2024.findings-emnlp.620 , url =

    2024 doi

  6. [14]

    2026 , url =

    Liu, Chris Yuhao and Zeng, Liang and Xiao, Yuzhen and He, Jujie and Liu, Jiacai and Wang, Chaojie and Yan, Rui and Shen, Wei and Zhang, Fuxiang and Xu, Jiacheng and Liu, Yang , booktitle =. 2026 , url =

    2026

  7. [15]

    2024 , url =

    Kim, Seungone and Shin, Jamin and Cho, Yejin and Jang, Joel and Longpre, Shayne and Lee, Hwaran and Yun, Sangdoo and Shin, Seongjin and Kim, Sungdong and Thorne, James and Seo, Minjoon , booktitle =. 2024 , url =

    2024

  8. [16]

    2024 , address =

    Kim, Seungone and Suk, Juyoung and Longpre, Shayne and Lin, Bill Yuchen and Shin, Jamin and Welleck, Sean and Neubig, Graham and Lee, Moontae and Lee, Kyungjae and Seo, Minjoon , booktitle =. 2024 , address =. doi:10.18653/v1/2024.emnlp-main.248 , url =

    2024 doi

  9. [17]

    2024 , eprint =

    Generative Reward Models , author =. 2024 , eprint =

    2024

  10. [18]

    Self-Generated Critiques Boost Reward Modeling for Language Models , author =. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , month = apr, year =....

    2025 doi

  11. [19]

    2026 , url =

    Chen, Xiusi and Li, Gaotang and Wang, Ziqi and Jin, Bowen and Qian, Cheng and Wang, Yu and Wang, Hongru and Zhang, Yu and Zhang, Denghui and Zhang, Tong and Tong, Hanghang and Ji, Heng , booktitle =. 2026 , url =

    2026

  12. [20]

    2025 , url =

    Hong, Ilgee and Yu, Changlong and Qiu, Liang and Yan, Weixiang and Xu, Zhenghao and Jiang, Haoming and Zhang, Qingru and Lu, Qin and Liu, Xin and Zhang, Chao and Zhao, Tuo , booktitle =. 2025 , url =

    2025

  13. [21]

    2025 , eprint =

    Inference-Time Scaling for Generalist Reward Modeling , author =. 2025 , eprint =. doi:10.48550/arXiv.2504.02495 , url =

    2025 doi

  14. [22]

    doi:10.48550/arXiv.2506.03637 , url =

    Yu, Zhuohao and Zeng, Jiali and Gu, Weizheng and Wang, Yidong and Wang, Jindong and Meng, Fandong and Zhou, Jie and Zhang, Yue and Zhang, Shikun and Ye, Wei , year =. doi:10.48550/arXiv.2506.03637 , url =. 2506.03637 , archivePrefix =

    doi

  15. [23]

    and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Kaplan, Jared , year =

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain,...

    doi arXiv

  16. [24]

    2024 , url =

    Ye, Seonghyeon and Kim, Doyoung and Kim, Sungdong and Hwang, Hyeonbin and Kim, Seungone and Jo, Yongrae and Thorne, James and Kim, Juho and Seo, Minjoon , booktitle =. 2024 , url =

    2024

  17. [25]

    2025 , address =

    Saad-Falcon, Jon and Vivek, Rajan Pathe and Berrios, William and Naik, Nandita Shankar and Franklin, Matija and Vidgen, Bertie and Singh, Amanpreet and Kiela, Douwe and Mehri, Shikib , booktitle =. 2025 , address =. doi:10.18653/v1/2025.findings-emnlp.176 , url =

    2025 doi

  18. [26]

    Findings of the Association for Computational Linguistics: ACL 2024 , month = aug, year =

    Benchmarking Cognitive Biases in Large Language Models as Evaluators , author =. Findings of the Association for Computational Linguistics: ACL 2024 , month = aug, year =. doi:10.18653/v1/2024.findings-acl.29 , url =

    2024 doi

  19. [27]

    Advances in Neural Information Processing Systems , volume =

    Checklists Are Better Than Reward Models for Aligning Language Models , author =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

    2025

  20. [28]

    2026 , eprint =

    Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric , author =. 2026 , eprint =

    2026

  21. [29]

    2023 , eprint =

    Instruction-Following Evaluation for Large Language Models , author =. 2023 , eprint =

    2023

  22. [30]

    Advances in Neural Information Processing Systems , volume =

    Generalizing Verifiable Instruction Following , author =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

    2025

  23. [31]

    He, Yun and Li, Wenzhe and Zhang, Hejia and Li, Songlin and Mandyam, Karishma and Khosla, Sopan and Xiong, Yuanhao and Wang, Nanshu and Peng, Xiaoliang and Li, Beibin and Bi, Shengjie and Patil, Shishir G. and Qi, Qi and Feng, Shengyu and Katz-Samuels, Julian and Pang, Richard...

    doi

  24. [32]

    2025 , address =

    Peng, Hao and Qi, Yunjia and Wang, Xiaozhi and Xu, Bin and Hou, Lei and Li, Juanzi , booktitle =. 2025 , address =. doi:10.18653/v1/2025.emnlp-main.1542 , url =

    2025 doi

  25. [33]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =

    Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems , author =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = jul, year =. doi:10.1865...

    2025 doi

  26. [34]

    2025 , url =

    Liu, Yantao and Yao, Zijun and Min, Rui and Cao, Yixin and Hou, Lei and Li, Juanzi , booktitle =. 2025 , url =

    2025

  27. [35]

    2025 , url =

    Tan, Sijun and Zhuang, Siyuan and Montgomery, Kyle and Tang, William Yuan and Cuadron, Alejandro and Wang, Chenguang and Popa, Raluca and Stoica, Ion , booktitle =. 2025 , url =

    2025

  28. [36]

    doi:10.48550/arXiv.2603.04738 , url =

    Wen, Bosi and Niu, Yilin and Wang, Cunxiang and Ling, Xiaoying and Zhang, Ying and Ke, Pei and Wang, Hongning and Huang, Minlie , year =. doi:10.48550/arXiv.2603.04738 , url =. 2603.04738 , archivePrefix =

    doi arXiv

  29. [37]

    doi:10.48550/arXiv.2510.07743 , url =

    Liu, Tianci and Xu, Ran and Yu, Tony and Hong, Ilgee and Yang, Carl and Zhao, Tuo and Wang, Haoyu , year =. doi:10.48550/arXiv.2510.07743 , url =. 2510.07743 , archivePrefix =

    doi

  30. [38]

    doi:10.48550/arXiv.2510.17314 , url =

    Xie, Lipeng and Huang, Sen and Zhang, Zhuo and Zou, Anni and Zhai, Yunpeng and Ren, Dingchao and Zhang, Kezun and Hu, Haoyuan and Liu, Boyin and Chen, Haoran and Liu, Zhaoyang and Ding, Bolin , year =. doi:10.48550/arXiv.2510.17314 , url =. 2510.17314 , archivePrefix =

    doi

  31. [39]

    Incentivizing Agentic Reasoning in

    Xu, Ran and Chen, Jingjing and Ye, Jiayu and Wu, Yu and Yan, Jun and Yang, Carl and Yu, Hongkun , booktitle =. Incentivizing Agentic Reasoning in. 2026 , url =

    2026

  32. [40]

    2026 , eprint =

    Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward , author =. 2026 , eprint =. doi:10.48550/arXiv.2602.12430 , url =

    2026 doi arXiv

  33. [41]

    doi:10.48550/arXiv.2602.20867 , url =

    Jiang, Yanna and Li, Delong and Deng, Haiyu and Ma, Baihe and Wang, Xu and Wang, Qin and Yu, Guangsheng , year =. doi:10.48550/arXiv.2602.20867 , url =. 2602.20867 , archivePrefix =

    doi arXiv

  34. [42]

    2025 , month = oct, howpublished =

    Introducing. 2025 , month = oct, howpublished =

    2025

  35. [43]

    2025 , month = oct, howpublished =

    Equipping Agents for the Real World with Agent Skills , author =. 2025 , month = oct, howpublished =

    2025

  36. [44]

    2025 , month = nov, howpublished =

    2025

  37. [45]

    doi:10.48550/arXiv.2602.12670 , url =

    Li, Xiangyi and Chen, Wenbo and Liu, Yimin and Zheng, Shenghan and Chen, Xiaokun and He, Yifeng and Li, Yubo and You, Bingran and Shen, Haotian and Sun, Jiankai and Wang, Shuyi and Li, Binxu and Zeng, Qunhong and Wang, Di and Zhao, Xuandong and Wang, Yuanli and Ben Chaim, Roey...

    doi arXiv

  38. [46]

    Can External Validation Tools Improve Annotation Quality for

    Findeis, Arduin and Weers, Floris and Yin, Guoli and Ye, Ke and Pang, Ruoming and Gunter, Tom , booktitle =. Can External Validation Tools Improve Annotation Quality for. 2025 , address =. doi:10.18653/v1/2025.acl-long.779 , url =

    2025 doi

  39. [47]

    Advances in Neural Information Processing Systems , volume =

    Reward Reasoning Models , author =. Advances in Neural Information Processing Systems , volume =. 2025 , url =

    2025

  40. [48]

    2026 , eprint =

    Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models , author =. 2026 , eprint =. doi:10.48550/arXiv.2602.04649 , url =

    2026 doi

  41. [49]

    2021 , eprint =

    Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and Jiang, Xu and Cobbe, Karl and Eloundou, Tyna and Krueger, Gretchen and Button, Kevin...

    2021

  42. [50]

    The Twelfth International Conference on Learning Representations , year =

    Let's Verify Step by Step , author =. The Twelfth International Conference on Learning Representations , year =

  43. [51]

    2024 , address =

    Jiang, Yuxin and Wang, Yufei and Zeng, Xingshan and Zhong, Wanjun and Li, Liangyou and Mi, Fei and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wang, Wei , booktitle =. 2024 , address =. doi:10.18653/v1/2024.acl-long.257 , url =

    2024 doi

  44. [52]

    2023 , address =

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle =. 2023 , address =. doi:10.18653/v1/2023.emnlp-main.153 , url =

    2023 doi

  45. [53]

    Evaluating Judges as Evaluators: The

    Zhou, Yilun and Xu, Austin and Wang, Peifeng and Xiong, Caiming and Joty, Shafiq , booktitle =. Evaluating Judges as Evaluators: The. 2025 , volume =

    2025

  46. [54]

    and Yang, Jiangjiang and Le Bras, Ronan and Tafjord, Oyvind and Wilhelm, Christopher and Soldaini, Luca and Smith, Noah A

    Lambert, Nathan and Morrison, Jacob and Pyatkin, Valentina and Huang, Shengyi and Ivison, Hamish and Brahman, Faeze and Miranda, Lester James Validad and Liu, Alisa and Dziri, Nouha and Lyu, Xinxi and Gu, Yuling and Malik, Saumya and Graf, Victoria and Hwang, Jena D. and Yang,...

    2025

  47. [55]

    doi:10.48550/arXiv.2307.09288 , url =

    Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and Bikel, Dan and Blecher, Lukas and Canton Ferrer, Cristian and Chen, Moya and Cucu...

    doi arXiv

  48. [56]

    2024 , volume =

    Lee, Harrison and Phatale, Samrat and Mansoor, Hassan and Mesnard, Thomas and Ferret, Johan and Lu, Kellie Ren and Bishop, Colton and Hall, Ethan and Carbune, Victor and Rastogi, Abhinav and Prakash, Sushant , booktitle =. 2024 , volume =

    2024

  49. [57]

    2024 , url =

    Xu, Can and Sun, Qingfeng and Zheng, Kai and Geng, Xiubo and Zhao, Pu and Feng, Jiazhan and Tao, Chongyang and Lin, Qingwei and Jiang, Daxin , booktitle =. 2024 , url =

    2024

  50. [58]

    Advances in Neural Information Processing Systems , volume =

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems , volume =. 2022 , url =

    2022

  51. [59]

    2021 , eprint =

    Evaluating Large Language Models Trained on Code , author =. 2021 , eprint =

    2021

  52. [60]

    2024 , url =

    Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyu...

    2024 arXiv

  53. [61]

    2023 , address =

    Min, Sewon and Krishna, Kalpesh and Lyu, Xinxi and Lewis, Mike and Yih, Wen-tau and Koh, Pang and Iyyer, Mohit and Zettlemoyer, Luke and Hajishirzi, Hannaneh , booktitle =. 2023 , address =. doi:10.18653/v1/2023.emnlp-main.741 , url =

    2023 doi

  54. [62]

    2024 , address =

    Luong, Trung Quoc and Zhang, Xinbo and Jie, Zhanming and Sun, Peng and Jin, Xiaoran and Li, Hang , booktitle =. 2024 , address =. doi:10.18653/v1/2024.acl-long.410 , url =

    2024 doi

  55. [63]

    Shao, Zhihong and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Song, Junxiao and Bi, Xiao and Zhang, Haowei and Zhang, Mingchuan and Li, Y. K. and Wu, Y. and Guo, Daya , year =. doi:10.48550/arXiv.2402.03300 , url =. 2402.03300 , archivePrefix =

    doi arXiv

  56. [64]

    2025 , eprint =

    Group Sequence Policy Optimization , author =. 2025 , eprint =

    2025

  57. [65]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = aug, year =

    Hashemi, Helia and Eisner, Jason and Rosset, Corby and. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , month = aug, year =. doi:10.18653/v1/2024.acl-long.745 , url =

    2024 doi

  58. [66]

    doi:10.48550/arXiv.2305.16291 , url =

    Wang, Guanzhi and Xie, Yuqi and Jiang, Yunfan and Mandlekar, Ajay and Xiao, Chaowei and Zhu, Yuke and Fan, Linxi and Anandkumar, Anima , year =. doi:10.48550/arXiv.2305.16291 , url =. 2305.16291 , archivePrefix =

    doi arXiv

  59. [67]

    Agent Skills: A Data-Driven Analysis of

    Ling, George and Zhong, Shanshan and Huang, Richard , year =. Agent Skills: A Data-Driven Analysis of. doi:10.48550/arXiv.2602.08004 , url =. 2602.08004 , archivePrefix =

    doi

  60. [68]

    2026 , eprint =

    A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applications , author =. 2026 , eprint =. doi:10.48550/arXiv.2605.07358 , url =

    2026 doi arXiv

  61. [69]

    2026 , eprint =

    Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale , author =. 2026 , eprint =. doi:10.48550/arXiv.2603.02176 , url =

    2026 doi

  62. [70]

    2026 , eprint =

    From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills , author =. 2026 , eprint =. doi:10.48550/arXiv.2604.24026 , url =

    2026 doi arXiv

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.