{"id":"cab13fe0-616a-45e3-9d6e-f36960dc7c23","arxiv_id":"2608.05139","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Skill entropy, a reference-model-based measure of skill-switching difficulty, calibrates a new cross-skill benchmark and serves as an RL reward, more than doubling small models' scores.","lead":"Long-horizon reasoning tasks often require switching between different skills, like math then planning. This paper introduces a 'skill entropy' difficulty measure, a benchmark built on it, and a training method that uses it as a reward, improving small models' cross-skill reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (7)'s multiplicative domain factorization is the unvalidated cornerstone of every task-level entropy value; if it mis-ranks skill pairs, both the benchmark's difficulty strata and the skill-entropy reward inherit the error. A direct skill-pair validation is required.","rationale":"The reader's verdict is CONDITIONAL, anchored on the unvalidated factorization in Eq. (7). Reading the paper in good faith, I find this to be the single most load-bearing concern. The paper's two headline contributions—Skill2-Bench as an entropy-calibrated benchmark and Skill-Entropy RL as a training method—both depend on task-level skill entropy, which is computed by averaging pairwise entropies obtained through the product approximation. If the approximation is inaccurate, the benchmark's difficulty labels are not measurements of the defined quantity, and the reward signal used in training is not grading what the paper claims to grade. The paper's own robustness checks are real evidence that the entropy signal is stable across reference models and partially aligned with human judgment, but they validate the approximated table, not the approximation. The empirical trends in Tables 2 and 3 provide downstream evidence that the entropy signal carries useful information, but they cannot distinguish a correct factorization from a systematically biased one that happens to correlate with task difficulty through domain distance or length. The concrete test I propose directly measures the unvalidated assumption by comparing Eq. (7) products against direct Eq. (3) skill-pair measurements, and then checks whether the benchmark stratification and RL reward survive the swap. Because this concern is exactly the reader's weakest assumption and does not introduce a new issue, the appropriate verdict remains CONDITIONAL: the work is promising but should not be accepted at face value until the factorization is validated or the paper discloses and mitigates its potential failure mode.","tokens_in":33540,"tokens_out":8267,"duration_ms":113754,"concrete_test":"Sample a stratified set of about 120 ordered skill pairs spanning all nine domain pairs and the full predicted-entropy range; for each, run N = 20 two-step evaluations with Claude-opus-4.7 and compute direct SkE(s_a,s_b) from Eq. (3). Compare against the Eq. (7) product: report Spearman rank correlation and mean absolute error. Then re-assign the 300 Skill2-Bench test tasks using the direct pairwise entropies, recompute the low/medium/high strata and the Table 2 accuracy-by-entropy trend, and check whether the monotonic drop and the RL reward rankings survive. Require pair-level rank correlation above roughly 0.7 and stratum overlap above 80%; otherwise the factorization, and every entropy-derived label and reward, is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central object of the paper is SkE(s_a,s_b), defined by Eq. (3) as a smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model. But no task-level entropy quoted anywhere—neither the low/medium/high strata of Skill2-Bench nor the r_ent reward in Eq. (6)—uses Eq. (3) directly. Appendix B.4 replaces the 3.1e5 ordered skill pairs with Eq. (7): SkE(s_a,s_b) ≈ SkE(s_a,d_sb)·SkE(d_sa,s_b). This asserts that the cost of leaving s_a and the cost of landing on s_b are multiplicatively separable and independent. The paper gives no direct comparison of the product against measured skill-pair entropies. The robustness checks in Tables 7–9 (reference-model overlap, cell-rank correlations, human agreement) all operate on the approximated table, so they cannot detect a systematic failure of the factorization. If Eq. (7) mis-ranks even a modest fraction of pairs—for example, because two specific skills in the same source/target domains interact in ways the domain averages miss—then every task-level SkE(τ) in Eq. (4), every difficulty level in the benchmark, and every skill-entropy reward used in training inherit the error. The empirical monotonic trend in Table 2 and the RL gains in Table 3 are consistent with a correct measure, but they do not validate the factorization itself: a wrong difficulty label could still correlate with accuracy if the dominant signal is domain distance or task length. Compounding this, each factor is estimated from only N_c = 5 two-step instances per (skill, domain) pair, so the product is noisy and no confidence intervals are reported. Thus the load-bearing condition—that Eq. (7) approximates Eq. (3) well enough to preserve pair ordering—is currently unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Skill Entropy (SkE), a directed pairwise measure of skill-switch difficulty defined as a smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model (Eq. 3), and a task-level version that averages pairwise entropies along a skill sequence (Eq. 4). It builds Skill2-Bench, a 300-task cross-skill long-horizon benchmark over 558 skills in 9 domains, stratified into low/medium/high entropy levels, and reports that 12 models' accuracy drops as task entropy rises (Table 2). The paper then proposes Skill-Entropy RL, which trains models to emit per-step skill tags and optimizes a reward combining answer correctness and rank-based skill-entropy alignment (Eqs. 5-6); it reports large gains on Qwen3-4B-Instruct (34.4 to 68.4) and Qwen3-1.7B (14.6 to 40.1), plus gains on off-the-shelf OpenR1-Math data. The central technical object, however, is everywhere computed through an unvalidated multiplicative domain factorization (Eq. 7) that reduces the pairwise evaluation from about 3.1e5 to about 5k cells.","tokens_in":33893,"tokens_out":10361,"duration_ms":119994,"significance":"If the skill-entropy signal is valid, the paper contributes a reusable difficulty scale for cross-skill long-horizon reasoning and a training reward that plausibly transfers across domains and datasets. Strengths: the evaluation protocol is carefully controlled (single-skill versus cross-skill comparison, deterministic scorers, verifier-filtered task synthesis, human-validated LLM judge for open-ended scoring, baselines sharing the same data), and the authors provide code and extensive appendices. The claimed headline gains are large and internally consistent with the GRPO ablation. However, the significance is conditional on the unvalidated Eq. (7), the small Monte-Carlo sample sizes, the self-judging of open-ended domains in the RL evaluation, and potential train/test seed overlap. The monotonic trend for the reference model itself is partly definitional; the empirical content resides in the other 11 models and in reference-model robustness checks.","major_comments":[{"comment":"Section B.4 defines the product approximation SkE(s_a,s_b) ≈ SkE(s_a,d_sb)·SkE(d_sa,s_b) and uses it to replace the 3.1e5 ordered skill-pair evaluations with about 5k evaluations, but the approximation is never validated against direct skill-pair measurements. Because Eq. (4) for every task-level entropy, the low/medium/high stratification of Skill2-Bench, and the r_ent reward in Eq. (6) all consume this approximated table, a systematic failure of the factorization would propagate into both the benchmark and training claims. The robustness checks in Tables 7-9 compare reference models on the approximated table and therefore cannot detect such a failure. I request a direct validation on a stratified sample of ordered skill pairs (e.g., 200-500 pairs spanning the domain-pair matrix), reporting rank correlation and stratum agreement between Eq. (7) and the direct Eq. (3) estimate.","section":"B.4 (Eq. 7)"},{"comment":"The entropy estimates use N_s=N_d=N_c=5 Monte-Carlo samples per cell at temperature 0.0, with no error bars or sensitivity analysis. With five binary per-step scores per cell, a single correct/incorrect answer moves the accuracy estimate by 20 percentage points, and the smoothed ratios in Eq. (3) are consequently noisy; task-level entropies and the thresholds θ_l and θ_h inherit this noise. Please provide bootstrap confidence intervals for the pairwise and task-level entropies and report how often tasks change stratum under resampling, or increase N_c for cells near the thresholds.","section":"B.3"},{"comment":"The RL evaluation uses 'the evaluated model itself serving as the LLM judge for open-ended steps.' This invalidates the open-ended domain columns in Table 3 and the overall Skill2-Bench score, because a model can be rewarded for answers that match its own judging preferences rather than the rubric. The human validation in Table 10 covers Claude-opus-4.7 as judge, not Qwen3-4B or Qwen3-1.7B self-scoring. Use a fixed external judge (e.g., Claude-opus-4.7) for all models, and report the verifiable-only score as a robustness check.","section":"D.5"},{"comment":"The 9K training tasks and the 300-task test set are both synthesized from the same seed datasets (OpenR1-Math, MMLU-Pro, LiveCodeBench, etc.), and the text only states that training tasks are 'held out from the test set' without specifying deduplication at the level of seed questions. If a seed question appears in a training task and in a test task's step, the absolute Skill2-Bench gains in Table 3 are inflated. Please report the number of test steps whose seed question (or a near-duplicate) appears in the training data, and rerun with exact seed-level deduplication.","section":"D.5"},{"comment":"Table 14 reports a Single-Skill oracle for the open-ended domains (e.g., Creative Writing 96.0 for Qwen3-4B), directly contradicting Section C, which states that single-skill accuracy is not reported for open-ended domains because their questions are context-dependent and not comparable. The subsequent conclusion that Skill-Entropy RL exceeds the single-skill oracle overall (68.4 vs. 62.2) therefore rests on numbers the paper itself declares non-comparable. Remove the open-ended columns from the oracle table or explain how comparable single-skill evaluations were obtained.","section":"C / D.8"},{"comment":"For the reference model Claude-opus-4.7, the monotonic low-to-high accuracy decrease in Table 2 is partly definitional, because the entropy scale is constructed from that model's own accuracy drops on two-step skill pairs (Eq. 3). The genuinely independent evidence is the behavior of the other 11 models, and the reference-model robustness checks in Tables 7-9 reuse the same approximated entropy table. I recommend reporting the entropy-accuracy monotonicity separately for non-reference models and, where possible, evaluating monotonicity on a held-out reference model not used to build the scale.","section":"3.1 / 3.3"}],"minor_comments":[{"comment":"The notation 'Skill2-Bench' and 'Skill^2-Bench' are used interchangeably; pick one and use it consistently.","section":"Throughout"},{"comment":"References [56] and [63] are incomplete author lists ('Xia et al.' and 'Zhang et al.'); add full citations.","section":"References"},{"comment":"The header layout with the merged 'SettingCoding Logic Math Planning Science Avg.' line is hard to parse; separate the single-skill/cross-skill indicator from the per-domain columns.","section":"Table 2"},{"comment":"The right panel labels 'Harder Domain' and 'Harder Skill-Switching' are ambiguous; clarify which axis each label refers to.","section":"Figure 2"},{"comment":"The choice of Laplace smoothing α=0.1 is not justified or ablated; a brief sensitivity check for α would strengthen the definition.","section":"Eq. (3)"},{"comment":"The reward-weight ablation in Table 13 sweeps only four splits without error bars; report repeated runs or at least note that these are single-run results.","section":"D.7"}],"recommendation":"major_revision","confidential_remarks":"This is a strong paper with a potentially useful measure and training signal, but the unvalidated factorization and the self-judging issue are load-bearing. I would not reject because both are fixable with additional experiments. The paper's scope fits the journal; however, given the large headline gains, the authors should also consider external review of the benchmark's causal interpretation and the train/test overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one-line take: this is a real contribution, not a repackaging. The authors define Skill Entropy as a directed pairwise difficulty, build Skill2-Bench over 558 skills, and use the same entropy signal as an RL reward. The training results are large and mostly well-controlled: the GRPO row is a clean ablation, the open-ended transfer and OpenR1-Math results argue for generality, and the LLM judge is checked against human raters. The reference-model robustness tables (partition overlap, rank correlations, human agreement) are more than this kind of paper usually does. Credit where due: this is careful work.\n\nThe soft spot is exactly where the stress test points. Equation (7) replaces every skill-pair entropy with a product of a skill-to-domain and a domain-to-skill entropy. That cuts roughly 3e5 evaluations to about 5k, which is sensible, but the factorization is never validated against directly measured skill-pair values. All the robustness checks run on the approximated table, so they cannot catch a systematic failure of the separability assumption. If two skills in the same source and target domains interact in a way the domain averages miss, then every task-level entropy, every difficulty stratum, and every skill-entropy reward inherits the error. The monotonic trend in Table 2 and the RL gains are consistent with a correct measure, but they do not establish it. This is the one thing I would want fixed before the numbers are taken at face value.\n\nMinor but real: each accuracy entering the entropy is estimated from N=5 Monte-Carlo samples with no error bars, so the pairwise table is noisier than it looks; the reward weights (lambda_ans=0.7, lambda_ent=0.3) are selected on the same benchmark where the gains are reported, which is a mild selection risk; and for the reference model itself the accuracy-vs-entropy trend is partly tautological — though the eleven other models make the trend empirical overall.\n\nCitation pattern is group-heavy, but the cited prior work is genuinely relevant, so I would not penalize it much.\n\nBottom line: the paper deserves a serious referee. I would send it out with a request for direct validation of Eq. (7) on a sampled subset of skill pairs, confidence intervals for the entropy estimates, and ideally a reward-weight robustness check on an unseen benchmark. The benchmark itself will be useful regardless, and the training idea is worth engaging with.","headline":"Skill entropy is a genuinely new and useful object, but every downstream result sits on an unvalidated factorization — referee it, and make the authors validate Eq. (7) directly.","tokens_in":34514,"tokens_out":2122,"would_cite":true,"duration_ms":25185,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes skill entropy, a measure of how hard it is for an LLM to switch reasoning skills mid-task, and shows the same signal both exposes a systematic accuracy drop and, as an RL reward, roughly doubles cross-skill scores.","keywords":["skill entropy","cross-skill long-horizon reasoning","skill-switching gap","LLM evaluation benchmark","Skill2-Bench","reinforcement learning reward","GRPO","skill-annotated reasoning"],"falsifier":"Measure two-step cross-skill accuracy directly for a random sample of a few hundred ordered skill pairs under the same reference model and compare the resulting entropies with the factorized values $\\mathrm{SkE}(s_a,s_b)\\approx\\mathrm{SkE}(s_a,d_{s_b})\\cdot\\mathrm{SkE}(d_{s_a},s_b)$; if the product systematically over- or under-estimates the direct measurements, or if tasks re-bucketed using direct values no longer show the monotone accuracy drop, the benchmark levels and the RL reward are artifacts of the factorization rather than of model skill-switching difficulty.","tokens_in":33307,"feed_emoji":"🧠","tokens_out":13234,"duration_ms":116075,"temperature":0.7,"pith_summary":"This paper argues that long-horizon reasoning failures in language models are not explained by how well models handle individual skills, but by how hard it is to switch between skills inside one reasoning chain. To make that difficulty measurable, it introduces skill entropy, a directed score that compares a fixed reference model's accuracy on each skill in isolation with its accuracy when the same skills are chained back to back. The paper uses this score to build Skill2-Bench, a benchmark over 558 skills across nine domains, and shows that model accuracy falls nearly monotonically as a task's skill entropy rises, with a drop of 4 to 13 points when a skill is exercised inside a cross-skill task rather than in isolation. Because the drop persists even for skills models master alone, the paper concludes that skill switching is an orthogonal capability, and demonstrates that the same entropy signal, used as a reinforcement-learning reward, lifts Qwen3-4B-Instruct's Skill2-Bench score from 34.4% to 68.4% and Qwen3-1.7B's from 14.6% to 40.1%.","feed_headline":"Skill entropy predicts and fixes LLM skill-switching drops","feed_subtitle":"A measure of how hard it is to change reasoning skill mid-task doubles Qwen3-4B's score on a 558-skill benchmark.","key_machinery":"The load-bearing object is the directed pairwise skill entropy $\\mathrm{SkE}(s_a,s_b)$, the smoothed ratio of single-skill to cross-skill accuracy under a fixed reference model (Claude-opus-4.7). Because measuring all $558\\times558$ ordered skill pairs directly would need roughly $3.1\\times10^5$ reference-model runs, the paper factors each pair as $\\mathrm{SkE}(s_a,s_b)\\approx \\mathrm{SkE}(s_a,d_{s_b})\\cdot\\mathrm{SkE}(d_{s_a},s_b)$, replacing the skill-pair cost with the product of a leaving cost and a landing cost coarse-grained through the skills' source domains; this cuts the budget to about $5\\times10^3$ evaluations. Task-level entropy is the average of the pairwise values along the task's switches, $\\mathrm{SkE}(\\tau)=\\frac{1}{L-1}\\sum_{i=1}^{L-1}\\mathrm{SkE}(s_i,s_{i+1})$, and the skill-entropy reward $r_{\\mathrm{ent}}=1-|\\hat\\rho-\\rho^\\star|$ compares the rank of the predicted skill sequence's entropy with the gold sequence's rank on the training distribution. The model is trained to emit a $\\langle\\mathrm{skill}\\rangle$ label before each $\\langle\\mathrm{answer}\\rangle$, so the reward grades the structure of the plan, not only its final outcome.","core_discovery":"Skill entropy is defined as a Laplace-smoothed ratio, $\\mathrm{SkE}(s_a,s_b) = (\\frac{1}{2}(\\mathrm{Acc}(s_a)+\\mathrm{Acc}(s_b))+\\alpha)/(\\mathrm{Acc}(s_a,s_b)+\\alpha)$ with $\\alpha=0.1$, comparing a fixed reference model's accuracy on each skill alone with its per-step accuracy on two-step chains that first use $s_a$ and then $s_b$; values above 1 mean the switch adds difficulty, and the order of the two skills matters. The central claim is that this score captures an orthogonal capability, skill switching, that single-skill benchmarks miss. Averaged along a task's directed skill sequence, task-level entropy sorts tasks into low, medium, and high difficulty, and across 8 frontier and 4 open-source models per-step accuracy declines nearly monotonically as that entropy rises; the same skill inside a cross-skill task loses 4 to 13 points relative to its single-skill score, with the dominant failure being that models reuse the previous step's skill and answer modality instead of switching. On the training side, the paper claims the same signal is reusable: a GRPO loop in which the model declares the skill of each step before its answer and receives a reward for matching the gold sequence's entropy rank outperforms answer-only RL and skill-aware baselines, transferring to open-ended domains and to off-the-shelf OpenR1-Math data.","pith_inferences":["The factorization $\\mathrm{SkE}(s_a,s_b)\\approx\\mathrm{SkE}(s_a,d_{s_b})\\cdot\\mathrm{SkE}(d_{s_a},s_b)$ is never checked against direct skill-pair measurements, so the sharpest test of the framework would be to compute a few hundred ordered pairs directly and compare the two sides; any systematic divergence would flow into both the benchmark's difficulty buckets and every entropy reward value.","Read as a training objective, the skill-entropy reward teaches a model to segment its reasoning into clean, task-appropriate skill transitions, which suggests it could apply wherever long structured traces exist, such as agent tool-use sequences or multi-turn dialogue, with any skill taxonomy rather than only the 558-skill bank.","Because tasks are rejection-sampled to land in the low/medium/high entropy buckets, the monotone accuracy drop could partly reflect the construction procedure; scoring tasks that were not selected for their entropy would separate a property of models from a property of the sampling."],"forward_implications":["Accuracy on cross-skill long-horizon tasks declines nearly monotonically as task-level skill entropy rises across nearly all of the 8 frontier and 4 open-source models tested, with smaller models dropping the most.","Exercising a skill inside a cross-skill task costs models 4 to 13 percentage points relative to the same skill in isolation, even for skills models nearly saturate in single-skill mode.","Adding the skill-entropy reward to answer-only GRPO adds +9.6 points (Qwen3-4B-Instruct) and +7.9 points (Qwen3-1.7B) on Skill2-Bench, and the gain persists under different reward-weight splits.","The reward transfers beyond its training distribution: it improves open-ended domains never seen in RL training, external long-horizon and general reasoning benchmarks, and a version of the pipeline applied to off-the-shelf OpenR1-Math data improves six math benchmarks by +1.9% on average over GRPO."],"supporting_citations":[{"why":"Supplies the math seed questions for the benchmark's math domain and the off-the-shelf training data for the transfer experiment.","marker":"[20]"},{"why":"Supplies GRPO, the reinforcement-learning algorithm the skill-entropy reward is trained with.","marker":"[44]"},{"why":"The prior compositional evaluation showing models fail when skills are combined, which skill entropy formalizes into a directed switch-difficulty score.","marker":"[61]"},{"why":"Supplies the science multiple-choice seed questions and scorer for the science domain.","marker":"[51]"},{"why":"Supplies the coding seed questions and the sandboxed unit-test scorer for the coding domain.","marker":"[21]"},{"why":"Supplies the planning seed questions and constraint-satisfaction scorer for the planning domain.","marker":"[69]"},{"why":"Supplies the logic seed questions for the logic domain.","marker":"[32]"},{"why":"One of the six external math benchmarks used to test whether the skill-entropy reward transfers to off-the-shelf training data.","marker":"[18]"}],"fun_headline_variants":["Skill entropy metric doubles Qwen3-4B reasoning score","Skill entropy reveals and fixes LLM skill-switch failures","New RL reward doubles LLM long-horizon reasoning accuracy","Skill entropy: from benchmark to training for LLM reasoning","Measure skill-switching difficulty to boost LLM reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire difficulty scale rests on the assumption that the cost of leaving one skill and the cost of landing on another can be multiplied to give the cost of the switch, an approximation that reduces the evaluation budget from about 310,000 ordered skill pairs to about 5,000 evaluations and is never verified against directly measured skill-pair values.","fun_headline_variants_meta":{"raw":{"variants":["Skill entropy metric doubles Qwen3-4B reasoning score","Skill entropy reveals and fixes LLM skill-switch failures","New RL reward doubles LLM long-horizon reasoning accuracy","Skill entropy: from benchmark to training for LLM reasoning","Measure skill-switching difficulty to boost LLM reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1994,"prompt_tokens":1227,"completion_tokens":767,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":843,"completion_tokens_details":{"reasoning_tokens":685}},"tokens_in":843,"tokens_out":767,"duration_ms":8412,"temperature":1.0,"reasoning_tokens":685,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:21:56.909811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure two-step cross-skill accuracy directly for a random sample of a few hundred ordered skill pairs under the same reference model and compare the resulting entropies with the factorized values $\\mathrm{SkE}(s_a,s_b)\\approx\\mathrm{SkE}(s_a,d_{s_b})\\cdot\\mathrm{SkE}(d_{s_a},s_b)$; if the product systematically over- or under-estimates the direct measurements, or if tasks re-bucketed using direct values no longer show the monotone accuracy drop, the benchmark levels and the RL reward are artifacts of the factorization rather than of model skill-switching difficulty.","supporting_citations":[],"review_version":1}