Pith. sign in

REVIEW 6 major objections 6 minor 87 references

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

T0 review · 6 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes skill entropy, a measure of how hard it is for an LLM to switch reasoning skills mid-task, and shows the same signal both exposes a systematic accuracy drop and, as an RL reward, roughly doubles cross-skill scores.

desk verdict Skill entropy is a genuinely new and useful object, but every downstream result sits on an unvalidated factorization — referee it, and make the authors validate Eq. (7) directly. read the letter →

arxiv 2608.05139 v1 pith:74Q7XCHU submitted 2026-08-05 cs.CL cs.LG

Yinghui He , Ling Yang , Jiarui Liu , Yongjin Yang , Lechen Zhang , Yingcheng Wu , Zhenfei Yin , Mengdi Wang
show 1 more author
Sanjeev Arora
This is my paper
classification cs.CLcs.LG
keywords skillentropycross-skilllong-horizonreasoningskill-switchinggapLLMevaluationbenchmarkSkill2-BenchreinforcementlearningrewardGRPOskill-annotated
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that long-horizon reasoning failures in language models are not explained by how well models handle individual skills, but by how hard it is to switch between skills inside one reasoning chain. To make that difficulty measurable, it introduces skill entropy, a directed score that compares a fixed reference model's accuracy on each skill in isolation with its accuracy when the same skills are chained back to back. The paper uses this score to build Skill2-Bench, a benchmark over 558 skills across nine domains, and shows that model accuracy falls nearly monotonically as a task's skill entropy rises, with a drop of 4 to 13 points when a skill is exercised inside a cross-skill task rather than in isolation. Because the drop persists even for skills models master alone, the paper concludes that skill switching is an orthogonal capability, and demonstrates that the same entropy signal, used as a reinforcement-learning reward, lifts Qwen3-4B-Instruct's Skill2-Bench score from 34.4% to 68.4% and Qwen3-1.7B's from 14.6% to 40.1%.

What carries the argument

The load-bearing object is the directed pairwise skill entropy $\mathrm{SkE}(s_a,s_b)$, the smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model (Claude-opus-4.7). Because measuring all $558\times558$ ordered skill pairs directly would need roughly $3.1\times10^5$ reference-model runs, the paper factors each pair as $\mathrm{SkE}(s_a,s_b)\approx \mathrm{SkE}(s_a,d_{s_b})\cdot\mathrm{SkE}(d_{s_a},s_b)$, replacing the skill-pair cost with the product of a leaving cost and a landing cost coarse-grained through the skills' source domains; this cuts the budget to about $5\times10^3$ evaluations. Task-level entropy is the average of the pairwise values along the task's switches, $\mathrm{SkE}(\tau)=\frac{1}{L-1}\sum_{i=1}^{L-1}\mathrm{SkE}(s_i,s_{i+1})$, and the skill-entropy reward $r_{\mathrm{ent}}=1-|\hat\rho-\rho^\star|$ compares the rank of the predicted skill sequence's entropy with the gold sequence's rank on the training distribution. The model is trained to emit a $\langle\mathrm{skill}\rangle$ label before each $\langle\mathrm{answer}\rangle$, so the reward grades the structure of the plan, not only its final outcome.

What would settle it

Measure two-step cross-skill accuracy directly for a random sample of a few hundred ordered skill pairs under the same reference model and compare the resulting entropies with the factorized values $\mathrm{SkE}(s_a,s_b)\approx\mathrm{SkE}(s_a,d_{s_b})\cdot\mathrm{SkE}(d_{s_a},s_b)$; if the product systematically over- or under-estimates the direct measurements, or if tasks re-bucketed using direct values no longer show the monotone accuracy drop, the benchmark levels and the RL reward are artifacts of the factorization rather than of model skill-switching difficulty.

Watch

Extended reading notes

Core claim

Skill entropy is defined as a Laplace-smoothed ratio, $\mathrm{SkE}(s_a,s_b) = (\frac{1}{2}(\mathrm{Acc}(s_a)+\mathrm{Acc}(s_b))+\alpha)/(\mathrm{Acc}(s_a,s_b)+\alpha)$ with $\alpha=0.1$, comparing a fixed reference model's accuracy on each skill alone with its per-step accuracy on two-step chains that first use $s_a$ and then $s_b$; values above 1 mean the switch adds difficulty, and the order of the two skills matters. The central claim is that this score captures an orthogonal capability, skill switching, that single-skill benchmarks miss. Averaged along a task's directed skill sequence, task-level entropy sorts tasks into low, medium, and high difficulty, and across 8 frontier and 4 open-source models per-step accuracy declines nearly monotonically as that entropy rises; the same skill inside a cross-skill task loses 4 to 13 points relative to its single-skill score, with the dominant failure being that models reuse the previous step's skill and answer modality instead of switching. On the training side, the paper claims the same signal is reusable: a GRPO loop in which the model declares the skill of each step before its answer and receives a reward for matching the gold sequence's entropy rank outperforms answer-only RL and skill-aware baselines, transferring to open-ended domains and to off-the-shelf OpenR1-Math data.

Load-bearing premise

The entire difficulty scale rests on the assumption that the cost of leaving one skill and the cost of landing on another can be multiplied to give the cost of the switch, an approximation that reduces the evaluation budget from about 310,000 ordered skill pairs to about 5,000 evaluations and is never verified against directly measured skill-pair values.

Editorial extensions

If this is right

  • Accuracy on cross-skill long-horizon tasks declines nearly monotonically as task-level skill entropy rises across nearly all of the 8 frontier and 4 open-source models tested, with smaller models dropping the most.
  • Exercising a skill inside a cross-skill task costs models 4 to 13 percentage points relative to the same skill in isolation, even for skills models nearly saturate in single-skill mode.
  • Adding the skill-entropy reward to answer-only GRPO adds +9.6 points (Qwen3-4B-Instruct) and +7.9 points (Qwen3-1.7B) on Skill2-Bench, and the gain persists under different reward-weight splits.
  • The reward transfers beyond its training distribution: it improves open-ended domains never seen in RL training, external long-horizon and general reasoning benchmarks, and a version of the pipeline applied to off-the-shelf OpenR1-Math data improves six math benchmarks by +1.9% on average over GRPO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization $\mathrm{SkE}(s_a,s_b)\approx\mathrm{SkE}(s_a,d_{s_b})\cdot\mathrm{SkE}(d_{s_a},s_b)$ is never checked against direct skill-pair measurements, so the sharpest test of the framework would be to compute a few hundred ordered pairs directly and compare the two sides; any systematic divergence would flow into both the benchmark's difficulty buckets and every entropy reward value.
  • Read as a training objective, the skill-entropy reward teaches a model to segment its reasoning into clean, task-appropriate skill transitions, which suggests it could apply wherever long structured traces exist, such as agent tool-use sequences or multi-turn dialogue, with any skill taxonomy rather than only the 558-skill bank.
  • Because tasks are rejection-sampled to land in the low/medium/high entropy buckets, the monotone accuracy drop could partly reflect the construction procedure; scoring tasks that were not selected for their entropy would separate a property of models from a property of the sampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper introduces Skill Entropy (SkE), a directed pairwise measure of skill-switch difficulty defined as a smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model (Eq. 3), and a task-level version that averages pairwise entropies along a skill sequence (Eq. 4). It builds Skill2-Bench, a 300-task cross-skill long-horizon benchmark over 558 skills in 9 domains, stratified into low/medium/high entropy levels, and reports that 12 models' accuracy drops as task entropy rises (Table 2). The paper then proposes Skill-Entropy RL, which trains models to emit per-step skill tags and optimizes a reward combining answer correctness and rank-based skill-entropy alignment (Eqs. 5-6); it reports large gains on Qwen3-4B-Instruct (34.4 to 68.4) and Qwen3-1.7B (14.6 to 40.1), plus gains on off-the-shelf OpenR1-Math data. The central technical object, however, is everywhere computed through an unvalidated multiplicative domain factorization (Eq. 7) that reduces the pairwise evaluation from about 3.1e5 to about 5k cells.

Significance. If the skill-entropy signal is valid, the paper contributes a reusable difficulty scale for cross-skill long-horizon reasoning and a training reward that plausibly transfers across domains and datasets. Strengths: the evaluation protocol is carefully controlled (single-skill versus cross-skill comparison, deterministic scorers, verifier-filtered task synthesis, human-validated LLM judge for open-ended scoring, baselines sharing the same data), and the authors provide code and extensive appendices. The claimed headline gains are large and internally consistent with the GRPO ablation. However, the significance is conditional on the unvalidated Eq. (7), the small Monte-Carlo sample sizes, the self-judging of open-ended domains in the RL evaluation, and potential train/test seed overlap. The monotonic trend for the reference model itself is partly definitional; the empirical content resides in the other 11 models and in reference-model robustness checks.

major comments (6)
  1. [B.4 (Eq. 7)] Section B.4 defines the product approximation SkE(s_a,s_b) ≈ SkE(s_a,d_sb)·SkE(d_sa,s_b) and uses it to replace the 3.1e5 ordered skill-pair evaluations with about 5k evaluations, but the approximation is never validated against direct skill-pair measurements. Because Eq. (4) for every task-level entropy, the low/medium/high stratification of Skill2-Bench, and the r_ent reward in Eq. (6) all consume this approximated table, a systematic failure of the factorization would propagate into both the benchmark and training claims. The robustness checks in Tables 7-9 compare reference models on the approximated table and therefore cannot detect such a failure. I request a direct validation on a stratified sample of ordered skill pairs (e.g., 200-500 pairs spanning the domain-pair matrix), reporting rank correlation and stratum agreement between Eq. (7) and the direct Eq. (3) estimate.
  2. [B.3] The entropy estimates use N_s=N_d=N_c=5 Monte-Carlo samples per cell at temperature 0.0, with no error bars or sensitivity analysis. With five binary per-step scores per cell, a single correct/incorrect answer moves the accuracy estimate by 20 percentage points, and the smoothed ratios in Eq. (3) are consequently noisy; task-level entropies and the thresholds θ_l and θ_h inherit this noise. Please provide bootstrap confidence intervals for the pairwise and task-level entropies and report how often tasks change stratum under resampling, or increase N_c for cells near the thresholds.
  3. [D.5] The RL evaluation uses 'the evaluated model itself serving as the LLM judge for open-ended steps.' This invalidates the open-ended domain columns in Table 3 and the overall Skill2-Bench score, because a model can be rewarded for answers that match its own judging preferences rather than the rubric. The human validation in Table 10 covers Claude-opus-4.7 as judge, not Qwen3-4B or Qwen3-1.7B self-scoring. Use a fixed external judge (e.g., Claude-opus-4.7) for all models, and report the verifiable-only score as a robustness check.
  4. [D.5] The 9K training tasks and the 300-task test set are both synthesized from the same seed datasets (OpenR1-Math, MMLU-Pro, LiveCodeBench, etc.), and the text only states that training tasks are 'held out from the test set' without specifying deduplication at the level of seed questions. If a seed question appears in a training task and in a test task's step, the absolute Skill2-Bench gains in Table 3 are inflated. Please report the number of test steps whose seed question (or a near-duplicate) appears in the training data, and rerun with exact seed-level deduplication.
  5. [C / D.8] Table 14 reports a Single-Skill oracle for the open-ended domains (e.g., Creative Writing 96.0 for Qwen3-4B), directly contradicting Section C, which states that single-skill accuracy is not reported for open-ended domains because their questions are context-dependent and not comparable. The subsequent conclusion that Skill-Entropy RL exceeds the single-skill oracle overall (68.4 vs. 62.2) therefore rests on numbers the paper itself declares non-comparable. Remove the open-ended columns from the oracle table or explain how comparable single-skill evaluations were obtained.
  6. [3.1 / 3.3] For the reference model Claude-opus-4.7, the monotonic low-to-high accuracy decrease in Table 2 is partly definitional, because the entropy scale is constructed from that model's own accuracy drops on two-step skill pairs (Eq. 3). The genuinely independent evidence is the behavior of the other 11 models, and the reference-model robustness checks in Tables 7-9 reuse the same approximated entropy table. I recommend reporting the entropy-accuracy monotonicity separately for non-reference models and, where possible, evaluating monotonicity on a held-out reference model not used to build the scale.
minor comments (6)
  1. [Throughout] The notation 'Skill2-Bench' and 'Skill^2-Bench' are used interchangeably; pick one and use it consistently.
  2. [References] References [56] and [63] are incomplete author lists ('Xia et al.' and 'Zhang et al.'); add full citations.
  3. [Table 2] The header layout with the merged 'SettingCoding Logic Math Planning Science Avg.' line is hard to parse; separate the single-skill/cross-skill indicator from the per-domain columns.
  4. [Figure 2] The right panel labels 'Harder Domain' and 'Harder Skill-Switching' are ambiguous; clarify which axis each label refers to.
  5. [Eq. (3)] The choice of Laplace smoothing α=0.1 is not justified or ablated; a brief sensitivity check for α would strengthen the definition.
  6. [D.7] The reward-weight ablation in Table 13 sweeps only four splits without error bars; report repeated runs or at least note that these are single-run results.

Circularity Check

1 steps flagged · score 2.0 of 10

One minor self-definitional validation for the reference model; central benchmark and RL claims remain independently testable.

  1. self definitional [Section 3.1 Eq. (3); Section 3.3 Table 2; Section B.5]
    "SkE(s_a,s_b) = (1/2(Accuracy(s_a)+Accuracy(s_b))+alpha) / (Accuracy(s_a,s_b)+alpha) ... (>1, hard to switch from s_a to s_b, <=1, easy to switch from s_a to s_b). (3)"

    High SkE is by definition a low value of Accuracy(s_a,s_b) for the reference model (Claude-opus-4.7). Task-level SkE (Eq. 4) averages these ratios, and the low/medium/high strata are thresholds on that average. Table 2 then reports Claude-opus-4.7's own accuracy as 72.2/69.2/68.6 across those strata and treats the monotone drop as confirmation that 'skill entropy tracks cross-skill task difficulty.' For the reference model, the drop is a restatement of the construction: the strata were built to separate tasks by the reference model's cross-skill accuracy. The claim is genuinely empirical for the 11 other models, whose drops are not encoded in the definition, so the circularity is limited to one data point in the validation.

full rationale

The central object SkE is defined as a smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model, and both the benchmark difficulty strata and the RL reward reuse this same reference-derived table. Any validation that relied only on the reference model's own drop would be circular. That circularity is present but narrow: Table 2's monotone trend is also observed across 11 non-reference models, which is an independent empirical finding, and the external benchmarks in Table 16 plus the OpenR1-Math results provide out-of-distribution checks. The load-bearing approximation in Eq. (7) factorizes pairwise skill entropy through domains and is never validated against direct skill-pair measurements; this is a serious correctness risk, but it is an unverified assumption rather than a circular reduction. Similarly, the default reward weights (0.7, 0.3) are chosen as the peak of a sweep on the target benchmark (Section D.7), which inflates the reported gain through test-set tuning but does not make the result definitionally equivalent to an input. Overall, no central derivation reduces to its own inputs; the self-referential validation of the reference model is a minor flaw, so the circularity score is 2.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a small number of hand-chosen or fitted quantities: the smoothing constant, the reward weights tuned on the target benchmark, and the sample sizes for entropy estimation. The most load-bearing axiom is the multiplicative separability of pairwise entropy, which is never tested against direct skill-pair measurements. The reference-model dependence is partially mitigated by robustness checks but remains a domain assumption.

free parameters (5)
  • Laplace smoothing alpha = 0.1
    Added in Equation (3) to stabilize the ratio; no sensitivity analysis is reported.
  • Reward weights (lambda_ans, lambda_ent) = 0.7, 0.3
    Chosen as the peak of a four-point sweep on the target benchmark (Table 13), so the default is fitted to the evaluation metric.
  • Monte Carlo sample sizes N_s, N_d, N_c = 5 per cell
    Only five samples per skill/domain pair are used to estimate accuracies in Section B.3, and no confidence intervals are reported for the resulting entropy values.
  • Entropy level thresholds (theta_l, theta_h) = Empirical distribution boundaries
    Low/medium/high splits are read off the empirical distribution of pairwise entropies (Section 3.1), making the difficulty levels data-dependent.
  • Embedding cosine threshold = 0.5
    Skills below this similarity to a bank entry get zero entropy reward (Section D.4); a hand-set threshold.
assumptions (4)
  • ad hoc to paper Multiplicative factorization of pairwise skill entropy (Equation 7)
    SkE(s_a,s_b) is approximated as SkE(s_a,d_sb)*SkE(d_sa,s_b), assuming independence of leaving and landing costs; this reduces computation from 3.1e5 to 5k evaluations but is not validated against direct pairwise computation.
  • domain assumption A fixed strong reference model (Claude-opus-4.7) defines task difficulty
    Skill entropy is computed from the reference model's accuracy drop; the measure is relative to that model, though robustness checks with two other models give 82-89% partition overlap.
  • domain assumption LLM-generated skill labels and verifier checks ensure benchmark quality
    Skills are labeled and clustered by an LLM with manual review; tasks are filtered by an LLM verifier on answer preservation, scenario consistency, dependency, and leakage. These checks are not independently audited.
  • domain assumption Seed datasets (OpenR1-Math, MMLU-Pro, etc.) are representative of their domains
    The benchmark inherits the quality and coverage of the selected seed datasets, which may not span each domain fully.
invented entities (1)
  • Skill Entropy (SkE) independent evidence
    purpose: Measures the difficulty of switching from one reasoning skill to another; used to stratify benchmark tasks and to provide an RL reward.
    The measure is grounded in human judgments (63% agreement versus 75% inter-annotator) and stable across alternative reference models (83-89% partition overlap), but it is defined relative to a chosen reference model and is not a first-principles quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning." pith.science (2026). https://pith.science/paper/74Q7XCHU

@misc{pith2026260805139,
  author       = {Pith},
  title        = {Pith review of: Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/74Q7XCHU}},
  note         = {Machine review of arXiv:2608.05139}
}
read the original abstract

Long-horizon reasoning in recent LLMs demands that the model switch between distinct skills inside a reasoning chain, such as first doing a math derivation, then using the result to plan a schedule. We call such problems cross-skill long-horizon tasks: multi-step tasks whose steps require different reasoning skills and depend on earlier outputs. Existing benchmarks often evaluate individual skills, lacking a principled way to measure how well a model switches between skills. We address this gap from both the evaluation and training sides. We introduce Skill Entropy, a measure of the difficulty of switching from one skill to another. We then propose Skill^2-Bench, a benchmark of cross-skill long-horizon tasks built over 558 skills across 9 verifiable and open-ended domains. Each task is assigned a task-level skill-entropy score and grouped into three difficulty levels. Evaluating 8 frontier and 4 open-source models on Skill^2-Bench reveals a skill-switching gap: accuracy decreases on higher-entropy tasks. We then turn skill entropy from a benchmark scale into a training signal. We propose Skill-Entropy RL, an RL framework where the model predicts not only the answer at each step but also the skill used to produce it. The reward combines step-level correctness with a skill-entropy reward that measures the alignment between the model-predicted skill sequence and the gold skill sequence. On Qwen3-4B-Instruct and Qwen3-1.7B, Skill-Entropy RL improves the Skill^2-Bench score from 34.4% to 68.4% and from 14.6% to 40.1%, respectively, outperforming competitive baselines. The same pipeline can be applied to off-the-shelf training data such as OpenR1-Math, indicating that skill entropy is a reusable training signal. Code available at: https://github.com/Gen-Verse/Skill-Entropy-RL

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 29 canonical work pages

  1. [1]

    Rahmani, and Marek Rei

    Lisa Alazraki, Lihu Chen, Ana Brassard, Joe Stacey, Hossein A. Rahmani, and Marek Rei. AgentCoMa: A compositional benchmark mixing commonsense and mathematical reasoning in real-world scenarios.arXiv preprint arXiv:2508.19988, 2025

    arXiv 2025

  2. [2]

    STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

    Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur, Chad DeLuca, and Hima Patel. STaD: Scaffolded task design for identifying compositional skill gaps in LLMs.arXiv preprint arXiv:2604.18177, 2026

    work page Pith review arXiv 2026

  3. [3]

    HeroBench: A benchmark for long-horizon planning and structured reasoning in virtual worlds.arXiv preprint arXiv:2508.12782, 2025

    Petr Anokhin, Roman Khalikov, Stefan Rebrikov, Viktor Volkov, Artyom Sorokin, and Vincent Bissonnette. HeroBench: A benchmark for long-horizon planning and structured reasoning in virtual worlds.arXiv preprint arXiv:2508.12782, 2025

    arXiv 2025

  4. [4]

    Concepts or Skills? Rethinking Instruction Selection for Multi-modal Models

    Andrew Bai, Justin Cui, Ruochen Wang, and Cho-Jui Hsieh. Concepts or skills? rethinking instruction selection for multi-modal models.arXiv preprint arXiv:2508.10339, 2025

    work page Pith review arXiv 2025

  5. [5]

    Symbolic mixture-of-experts: Adaptive skill-based routing for heterogeneous reasoning.arXiv preprint arXiv:2503.05641, 2025

    Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, and Mohit Bansal. Symbolic mixture-of-experts: Adaptive skill-based routing for heterogeneous reasoning.arXiv preprint arXiv:2503.05641, 2025

    arXiv 2025

  6. [6]

    SkillCraft: Can LLM agents learn to use tools skillfully?arXiv preprint arXiv:2603.00718, 2026

    Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, Junlong Li, Kangrui Wang, Zihan Wang, Zhengyu Chen, Klara Kaleb, Ning Miao, Siyang Gao, Cong Lu, Manling Li, Junxian He, and Yee Whye Teh. SkillCraft: Can LLM agents learn to use tools skillfully?arXiv preprint arXiv:2603.00718, 2026

    arXiv 2026

  7. [7]

    Self-evolving curriculum for LLM reasoning.arXiv preprint arXiv:2505.14970, 2025

    Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piche, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for LLM reasoning.arXiv preprint arXiv:2505.14970, 2025

    arXiv 2025

  8. [8]

    WebSRC: A dataset for web-based structural reading comprehension.arXiv preprint arXiv:2101.09465, 2021

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A dataset for web-based structural reading comprehension.arXiv preprint arXiv:2101.09465, 2021

    arXiv 2021

Show all 87 references
  1. [9]

    Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P

    Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzh...

    2025 arXiv

  2. [10]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    Metacognitive capabilities of LLMs: An exploration in mathematical problem solving.Advances in Neural Information Processing Systems, 2024

    Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Sanjeev Arora. Metacognitive capabilities of LLMs: An exploration in mathematical problem solving.Advances in Neural Information Proc...

    2024

  4. [12]

    Thinkless: LLM learns when to think.arXiv preprint arXiv:2505.13379, 2025

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: LLM learns when to think.arXiv preprint arXiv:2505.13379, 2025. 12 Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

    2025 arXiv

  5. [13]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  6. [14]

    R-Bench: Graduate-level multi-disciplinary benchmarks for LLM & MLLM complex reasoning evaluation.arXiv preprint arXiv:2505.02018, 2025

    Meng-Hao Guo, Jiajun Xu, Yi Zhang, Jiaxi Song, Haoyang Peng, Yi-Xuan Deng, Xinzhi Dong, Kiy- ohiroNakayama,ZhengyangGeng,ChenWang,BolinNi,Guo-WeiYang,YongmingRao,Houwen Peng, Han Hu, Gordon Wetzstein, and Shi-min Hu. R-Bench: Graduate-level multi-disciplinary benchmarks for LL...

    2025 arXiv

  7. [15]

    AdaptMI: Adaptive skill-based in-context math instruction for small language models.arXiv preprint arXiv:2505.00147, 2025

    Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. AdaptMI: Adaptive skill-based in-context math instruction for small language models.arXiv preprint arXiv:2505.00147, 2025

    2025 arXiv

  8. [16]

    STAT: Skill-targeted adaptive training.arXiv preprint arXiv:2510.10023, 2025

    Yinghui He, Abhishek Panigrahi, Yong Lin, and Sanjeev Arora. STAT: Skill-targeted adaptive training.arXiv preprint arXiv:2510.10023, 2025

    2025

  9. [17]

    Self-distillation zero: Self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002, 2026

    Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision.arXiv preprint arXiv:2604.12002, 2026

    2026 arXiv

  10. [18]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  11. [19]

    Evaluating memory in LLM agents via incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025

    Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions.arXiv preprint arXiv:2507.05257, 2025

    2025 arXiv

  12. [20]

    Open-R1: A fully open reproduction of DeepSeek-R1

    Hugging Face. Open-R1: A fully open reproduction of DeepSeek-R1. https://github.com/huggingface/open-r1, 2025

    2025

  13. [21]

    LiveCodeBench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  14. [22]

    DRP: Distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models.arXiv preprint arXiv:2505.13975, 2025

    Yuxuan Jiang, Dawei Li, and Francis Ferraro. DRP: Distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models.arXiv preprint arXiv:2505.13975, 2025

    2025 arXiv

  15. [23]

    Instruct-SkillMix: A powerful pipeline for LLM instruction tuning.arXiv preprint arXiv:2408.14774, 2024

    Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-SkillMix: A powerful pipeline for LLM instruction tuning.arXiv preprint arXiv:2408.14774, 2024

    2024 arXiv

  16. [24]

    Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q

    Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Q...

    2025 arXiv

  17. [25]

    Benchmark profiling: Mechanistic diagnosis of LLM benchmarks.arXiv preprint arXiv:2510.01232, 2025

    Dongjun Kim, Gyuho Shim, Yongchan Chun, Minhyuk Kim, Chanjun Park, and Heuiseok Lim. Benchmark profiling: Mechanistic diagnosis of LLM benchmarks.arXiv preprint arXiv:2510.01232, 2025

    2025

  18. [26]

    MSCoRe: A benchmark for multi-stage collaborative reasoning in LLM agents.arXiv preprint arXiv:2509.17628, 2025

    Yuzhen Lei, Hongbin Xie, Jiaxing Zhao, Shuangxue Liu, and Xuan Song. MSCoRe: A benchmark for multi-stage collaborative reasoning in LLM agents.arXiv preprint arXiv:2509.17628, 2025

    2025

  19. [27]

    START: Self-taught reasoner with tools.arXiv preprint arXiv:2503.04625, 2025

    Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. START: Self-taught reasoner with tools.arXiv preprint arXiv:2503.04625, 2025. 13 Toward Skill-Native LLMs: Skill Entropy for Benchmarking and ...

    2025 arXiv

  20. [28]

    SkillsBench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670, 2026

    XiangyiLi, WenboChen, YiminLiu, ShenghanZheng, XiaokunChen, YifengHe, YuboLi, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Binxu Li, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kon...

    2026 arXiv

  21. [29]

    Benchmark test-time scaling of general LLM agents.arXiv preprint arXiv:2602.18998, 2026

    Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, and Chenyan Xiong. Benchmark test-time scaling of general LLM agents.arXiv preprint arXiv:2602.18998, 2026

    2026

  22. [30]

    MTR-Bench: A comprehensive benchmark for multi-turn reasoning evaluation.arXiv preprint arXiv:2505.17123, 2025

    Xiaoyuan Li, Keqin Bao, Yubo Ma, Moxin Li, Wenjie Wang, Rui Men, Yichang Zhang, Fuli Feng, Dayiheng Liu, and Junyang Lin. MTR-Bench: A comprehensive benchmark for multi-turn reasoning evaluation.arXiv preprint arXiv:2505.17123, 2025

    2025 arXiv

  23. [31]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  24. [32]

    ZebraLogic: On the scaling limits of LLMs for logical reasoning.arXiv preprint arXiv:2502.01100, 2025

    Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. ZebraLogic: On the scaling limits of LLMs for logical reasoning.arXiv preprint arXiv:2502.01100, 2025

    2025 arXiv

  25. [33]

    AgentBench: Evaluating LLMs as agents.arXiv preprint arXiv:2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench:...

    2023 arXiv

  26. [34]

    GAIA: A benchmark for general AI assistants.arXiv preprint arXiv:2311.12983, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for general AI assistants.arXiv preprint arXiv:2311.12983, 2023

    2023 arXiv

  27. [35]

    Bench- marking and understanding compositional relational reasoning of LLMs.arXiv preprint arXiv:2412.12841, 2024

    Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, and Hongliang Liang. Bench- marking and understanding compositional relational reasoning of LLMs.arXiv preprint arXiv:2412.12841, 2024

    2024 arXiv

  28. [36]

    Reasoning curriculum: Bootstrapping broad LLM reasoning from math.arXiv preprint arXiv:2510.26143, 2025

    Bo Pang, Deqian Kong, Silvio Savarese, Caiming Xiong, and Yingbo Zhou. Reasoning curriculum: Bootstrapping broad LLM reasoning from math.arXiv preprint arXiv:2510.26143, 2025

    2025

  29. [37]

    Compositional semantic parsing on semi-structured tables

    Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305, 2015

    2015 arXiv

  30. [38]

    Learning to reason across parallel samples for LLM reasoning.arXiv preprint arXiv:2506.09014, 2025

    Jianing Qi, Xi Ye, Hao Tang, Zhigang Zhu, and Eunsol Choi. Learning to reason across parallel samples for LLM reasoning.arXiv preprint arXiv:2506.09014, 2025

    2025

  31. [39]

    EmoAgent: Assessing and safeguarding human-AI interaction for mental health safety

    Jiahao Qiu, Yinghui He, Xinzhe Juan, Yimin Wang, Yuhan Liu, Zixin Yao, Yue Wu, Xun Jiang, Ling Yang, and Mengdi Wang. EmoAgent: Assessing and safeguarding human-AI interaction for mental health safety. InProceedings of the 2025 Conference on Empirical Methods in Natural Langua...

    2025

  32. [40]

    LogicSkills: A structured benchmark for formal reasoning in large language models.arXiv preprint arXiv:2602.06533, 2026

    Brian Rabern, Philipp Mondorf, and Barbara Plank. LogicSkills: A structured benchmark for formal reasoning in large language models.arXiv preprint arXiv:2602.06533, 2026

    2026

  33. [41]

    Reasoning models are test exploiters: Rethinking multiple-choice.arXiv preprint arXiv:2507.15337, 2025

    Narun Raman, Taylor Lundy, and Kevin Leyton-Brown. Reasoning models are test exploiters: Rethinking multiple-choice.arXiv preprint arXiv:2507.15337, 2025. 14 Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning

    2025

  34. [42]

    Revisiting compositional generalization capability of large language models considering instruction following ability.arXiv preprint arXiv:2506.15629, 2025

    Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. Revisiting compositional generalization capability of large language models considering instruction following ability.arXiv preprint arXiv:2506.15629, 2025

    2025 arXiv

  35. [43]

    AI-assisted generation of difficult math questions.arXiv preprint arXiv:2407.21009, 2024

    Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. AI-assisted generation of difficult math questions.arXiv preprint arXiv:2407.21009, 2024

    2024 arXiv

  36. [44]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  37. [45]

    DARE- bench: Evaluating modeling and instruction fidelity of LLMs in data science.arXiv preprint arXiv:2602.24288, 2026

    Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu, Zhewei Yao, Yuxiong He, and Feng Yan. DARE- bench: Evaluating modeling and instruction fidelity of LLMs in data science.arXiv preprint arXiv:2602.24288, 2026

    2026

  38. [46]

    PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint arXiv:2504.01848, 2025

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research.arXiv preprint ...

    2025 arXiv

  39. [47]

    Challenging BIG-Bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    arXiv

  40. [48]

    Reinforcement learning for self-improving agent with skill library.arXiv preprint arXiv:2512.17102, 2025

    Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. Reinforcement learning for self-improving agent with skill library.arXiv preprint arXiv:2512.17102, 2025

    2025 arXiv

  41. [49]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2024

    2024 arXiv

  42. [50]

    OdysseyBench: Evaluating LLM agents on long-horizon complex office application workflows

    WeixuanWang, DonggeHan, DanielMadrigalDiaz, JinXu, VictorRühle, andSaravanRajmohan. OdysseyBench: Evaluating LLM agents on long-horizon complex office application workflows. arXiv preprint arXiv:2508.09124, 2025

    2025 arXiv

  43. [51]

    MMLU-Pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    YuboWang, XueguangMa, GeZhang, YuanshengNi, AbhranilChandra, ShiguangGuo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understandin...

    2024 arXiv

  44. [52]

    Reinforcingmulti-turn reasoning in LLM agents via turn-level reward design.arXiv preprint arXiv:2505.11821, 2025

    QuanWei, SiliangZeng, ChenliangLi, WilliamBrown, OanaFrunza, WeiDeng, AndersonSchnei- der,YuriyNevmyvaka,YangKatieZhao,AlfredoGarcia,andMingyiHong. Reinforcingmulti-turn reasoning in LLM agents via turn-level reward design.arXiv preprint arXiv:2505.11821, 2025

    2025

  45. [53]

    Towardscompositionalgeneralization of LLMs via skill taxonomy guided data synthesis.arXiv preprint arXiv:2601.03676, 2026

    YifanWei, LiDu, XiaoyanYu, YangFeng, andAngshengLi. Towardscompositionalgeneralization of LLMs via skill taxonomy guided data synthesis.arXiv preprint arXiv:2601.03676, 2026

    2026

  46. [54]

    Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in large language models

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. 15 Toward Skill-Nati...

    2023

  47. [55]

    CritICL: Inference-time weak-to-strong generalization from small language model failure modes

    Yufan Wu, Zhengyi Hu, Lang Wei, Ruichen Li, Qifan Yang, Yinghui He, and Ting Zhu. CritICL: Inference-time weak-to-strong generalization from small language model failure modes. In Conference on Language Modeling (COLM), 2026

    2026

  48. [56]

    SkillRL: Evolving agents via recursive skill-augmented reinforcement learning, 2026

    Xia et al. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning, 2026

    2026

  49. [57]

    LaRS: Latent reasoning skills for chain-of-thought reasoning.arXiv preprint arXiv:2312.04684, 2024

    Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Xian Wu, Peter Stone, and Yanjun Qi. LaRS: Latent reasoning skills for chain-of-thought reasoning.arXiv preprint arXiv:2312.04684, 2024

    2024 arXiv

  50. [58]

    DeepCritic: Deliberate critique with large language models.arXiv preprint arXiv:2505.00662, 2025

    Wenkai Yang, Jingwen Chen, Yankai Lin, and Ji-Rong Wen. DeepCritic: Deliberate critique with large language models.arXiv preprint arXiv:2505.00662, 2025

    2025 arXiv

  51. [59]

    Transferability for general reasoning: An automated curriculum for multi-domain RLVR.arXiv preprint arXiv:2606.25178, 2026

    Yongjin Yang, Jiarui Liu, Yinghui He, Lechen Zhang, Bernhard Schölkopf, and Zhijing Jin. Transferability for general reasoning: An automated curriculum for multi-domain RLVR.arXiv preprint arXiv:2606.25178, 2026

    2026 arXiv

  52. [60]

    LongProc: Benchmarking long-context language models on long procedural generation

    Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. LongProc: Benchmarking long-context language models on long procedural generation. arXiv preprint arXiv:2501.05414, 2025

    2025

  53. [61]

    Skill-Mix: A flexible and expandable family of evaluations for AI models.arXiv preprint arXiv:2310.17567, 2023

    Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-Mix: A flexible and expandable family of evaluations for AI models.arXiv preprint arXiv:2310.17567, 2023

    2023 arXiv

  54. [62]

    From𝑓(𝑥) and 𝑔(𝑥) to 𝑓(𝑔(𝑥)) : LLMs learn new skills in RL by composing old ones.arXiv preprint arXiv:2509.25123, 2025

    Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. From𝑓(𝑥) and 𝑔(𝑥) to 𝑓(𝑔(𝑥)) : LLMs learn new skills in RL by composing old ones.arXiv preprint arXiv:2509.25123, 2025

    2025

  55. [63]

    Skill-aware data selection and fine-tuning for data-efficient reasoning distillation

    Zhang et al. Skill-aware data selection and fine-tuning for data-efficient reasoning distillation. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2026

    2026

  56. [64]

    Skill-awaredataselectionandfine-tuning for data-efficient reasoning distillation, 2026

    LechenZhang,YunxiangZhang,WeiHu,andLuWang. Skill-awaredataselectionandfine-tuning for data-efficient reasoning distillation, 2026. URLhttps://arxiv.org/abs/2601.10109

    2026

  57. [65]

    Lee, Chenlei Leng, and Fanghui Liu

    Yuanhe Zhang, Ilja Kuzborskij, Jason D. Lee, Chenlei Leng, and Fanghui Liu. DAG-Math: Graph-of-thought guided mathematical reasoning in LLMs.arXiv preprint arXiv:2510.19842, 2025

    2025

  58. [66]

    RLVMR: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. RLVMR: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents.arXiv preprint arXiv:2507.22844, 2025

    2025 arXiv

  59. [67]

    Can models learn skill composition from examples?arXiv preprint arXiv:2409.19808, 2024

    Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples?arXiv preprint arXiv:2409.19808, 2024

    2024 arXiv

  60. [68]

    A survey of process reward models: From outcome signals to process supervisions for large language models.arXiv preprint arXiv:2510.08049, 2025

    Congmin Zheng, Jiachen Zhu, Zhuoying Ou, Yuxiang Chen, Kangning Zhang, Rong Shan, Zeyu Zheng, Mengyue Yang, Jianghao Lin, Yong Yu, and Weinan Zhang. A survey of process reward models: From outcome signals to process supervisions for large language models.arXiv preprint arXiv:2...

    arXiv

  61. [69]

    NATURAL PLAN: Benchmarking LLMs on natural language planning.arXiv preprint arXiv:2406.04520, 2024

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V Le, Ed H Chi, and Denny Zhou. NATURAL PLAN: Benchmarking LLMs on natural language planning.arXiv preprint arXiv:2406.04520, 2024. 16 Toward Skill-Native LLMs:...

    2024 arXiv

  62. [70]

    SkillRouter: Skill routing for LLM agents at scale.arXiv preprint arXiv:2603.22455, 2026

    YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu. SkillRouter: Skill routing for LLM agents at scale.arXiv preprint arXiv:2603.22455, 2026

    2026 arXiv

  63. [71]

    Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Lihan Feng, Yuyi Ao, Leonardo F. R. Ribeiro, Markus Dreyer, Sean Ammirati, and Chenyan Xiong. SkillLearnBench: Benchmarking continual learning methods for agent skill generation on real-world tasks.arXiv preprint arXiv:2604.2008...

    2026 arXiv

  64. [72]

    It clearly outlines the key factors and their interrelationships

    Clarity and Coherence (20 points): The narrative is well-structured, easy to follow, and logically organized. It clearly outlines the key factors and their interrelationships

  65. [73]

    Specific examples from the data are used to support the narrative

    Depth of Analysis (30 points): The narrative delves into the motivations and dynamics of the competing nations or factions, providing a nuanced understanding of the situation. Specific examples from the data are used to support the narrative

  66. [74]

    It captures the reader’s interest and effectively conveys the potential crisis

    Creativity and Engagement (20 points): The narrative is engaging and creative, using vivid language and storytelling techniques to bring the scenario to life. It captures the reader’s interest and effectively conveys the potential crisis

  67. [75]

    It uses the data and insights from the previous steps to construct a plausible and coherent narrative

    Relevance and Accuracy (30 points): The narrative accurately reflects the information provided in the previous steps and is relevant to the task of predicting the outcome of the international crisis. It uses the data and insights from the previous steps to construct a plausibl...

  68. [76]

    It should reflect the brand’s commitment to sustainability

    Relevance to Eco-Friendly Attributes (30 points): The title should clearly communicate the eco- friendly attributes of the products. It should reflect the brand’s commitment to sustainability

  69. [77]

    It should not exceed 10 words

    Conciseness and Descriptiveness (30 points): The title should be concise and descriptive, effectively conveying the essence of the product line. It should not exceed 10 words

  70. [78]

    Alignment with Unique Selling Points (20 points): The title should highlight the unique selling points of the products, such as innovative materials, energy efficiency, or reduced carbon footprint

  71. [79]

    no-interference

    Appeal to Target Market (20 points): The title should resonate emotionally with the target market, using language that appeals to their values and interests. Total possible points: 100 </rubric> B.2. Skill bank This subsection details how the per-domain skill banksS𝑑 introduce...

  72. [80]

    DON’T CHANGE THE ANSWER, CORE LOGIC, OR THE SKILL REQUIRED

    ONLY ADAPT STYLE AND TONES TO MATCH THE SCENARIO. DON’T CHANGE THE ANSWER, CORE LOGIC, OR THE SKILL REQUIRED

  73. [81]

    THE ADAPTED QUESTION SHOULD HAVE THE EXACT SAME ANSWER AS BEFORE (PASS KEYWORD MATCHING!)

  74. [82]

    Conference trip on constraints

    IF THE ORIGINAL QUESTION IS MULTIPLE CHOICE, YOU MUST INCLUDE ALL ANSWER OPTIONS (A, B, C, D, E , etc.) VERBATIM IN THE ADAPTED QUESTION -- DO NOT OMIT THEM. Output Format: <reason> [Your thought on how to adapt the question to match the scenario] </reason> <question> [Your ad...

  75. [83]

    29 Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning 2.Scenario consistent.All rewritten steps plausibly belong to the samescenario

    Answer preserved.Each rewritten step still admits the original ground-truth answer of its seed question. 29 Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning 2.Scenario consistent.All rewritten steps plausibly belong to the samescenar...

  76. [84]

    The core logic, numerical values, and (for multiple-choice) the option letters and contents must be unchanged

    ANSWER_PRESERVED: For every step i, the rewritten question {rewritten_task}[i].question still admits the original ground-truth answer {seed_pairs}[i].answer of its seed question. The core logic, numerical values, and (for multiple-choice) the option letters and contents must b...

  77. [85]

    FAIL if any step references entities, settings, or framings that contradict the scenario or that read as an unrelated problem pasted in

    SCENARIO_CONSISTENT: All rewritten steps plausibly belong to the same scenario described above. FAIL if any step references entities, settings, or framings that contradict the scenario or that read as an unrelated problem pasted in

  78. [86]

    using the value from the previous step

    DEPENDENCY_PRESENT: For every step i > 1, the rewritten question cannot be solved without the answer of step i-1 -- either it explicitly references the previous answer (e.g., "using the value from the previous step", a named quantity carried over) or the scenario state it reli...

  79. [87]

    MAE” is the mean absolute error between the LLM and mean human score in[0, 1]. “Binary agree

    NO_ANSWER_LEAKAGE: The scenario text and every earlier step (question and any narrative around it) do not reveal the ground-truth answer of a later step, either verbatim or by paraphrase, computation, or option elimination. FAIL if a later answer can be read off the scenario o...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.