{"id":"cafd9073-c4ce-440b-ab66-1ed7bdfb4fed","arxiv_id":"2610.05777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On two enterprise agent benchmarks, the best skill form and mining-evidence choice depend on the domain, with workflows and labeled contrastive evidence helping on one benchmark and no configuration winning on the other.","lead":"This study mines reusable agent skills from recorded business-workflow execution traces and compares six ways of deciding which traces to show the skill-mining model and how to format the resulting skill. It finds that the best choice depends on the task domain, with no single configuration winning everywhere, so practitioners should test configurations on their own deployments.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The workflow-versus-ontology contrast is not identified as a representation effect: the 1.7 pp ThinkingBox lead bundles scope, skill count, and read-rate differences; a global-ontology arm would settle it.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my read agrees with the identified weakest assumption. The paper is unusually honest: the key confounds are stated in Sections 3.2 and 5.2 (sentences b161, b165, b169), the Random Blind arm addresses outcome-based task selection, and the task-level explanations are explicitly framed as post hoc hypotheses (b181). The central context-dependence conclusion is robust to the representation confound, because the same one-global-skill-versus-many-per-group-skills structure applies in both benchmarks, yet the preferred form reverses across benchmarks; that reversal already establishes some domain-dependent interaction. What the confound threatens is the causal attribution to representation specifically, which is exactly what the paper's Contribution 1 and the headline 'workflow beats ontology by 1.7 pp' claim. The audit in Figure 4 strengthens the concern by showing systematic read-rate differences between the forms, so the contrast bundles representation with deployment and routing properties. The single global-ontology arm would hold scope and skill count fixed and settle whether the form itself, rather than the library structure, drives the difference. Keeping the verdict CONDITIONAL rather than ACCEPT is also justified by the absence of shipped code or data, since the proposed check is concrete and the described pipeline is detailed enough to run.","tokens_in":25751,"tokens_out":7968,"duration_ms":104240,"concrete_test":"Run one additional arm on both benchmarks: a global operational ontology mined per fold with the same dataset-wide scope, the same five folds, the same Success/Goldilocks/Blind evidence selection, and exactly one skill per fold as the workflow arm, using the existing ontology meta-skill with a global scope contract. Evaluate under the same protocol. If the global-ontology minus global-workflow paired difference reproduces the current +1.7 pp (ThinkingBox) and -0.7 pp (APEX) contrasts, the representation effect is identified. If it moves toward zero or reverses, the original workflow-versus-ontology difference was driven by scope, skill count, or evidence quantity rather than by procedural-versus-declarative form.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's modest conclusion, that no single configuration wins on both benchmarks, is supported by the reversal in point estimates. But the headline quantitative contrast used to motivate it, workflows beat ontologies by +1.7 pp on ThinkingBox-Bench and trail on APEX-Agents (Table 3), does not identify representation as the cause. Each workflow arm mines one global skill per fold; each ontology arm mines one skill per task group, so scope, library size, and per-skill evidence quantity change with the form. The paper acknowledges this at b161, and Figure 4 adds a further bundle: in ThinkingBox-Bench, workflow skills are read in nearly every run, while ontology reads are absent or off-scope in neobank, car insurance, retail, and hotel. The contrast therefore also includes routing burden and skill selection. Success-versus-Goldilocks similarly changes task selection and trace composition (b165), and labeled-versus-blind changes both labels and mining instructions (b169). These confounds do not refute the context-dependence claim, but they do mean the paper's stated contribution of controlled comparisons of skill form and mining evidence is stronger than the design supports.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper reports an empirical measurement study of skill mining from enterprise agent traces. Holding one mining pipeline and a frozen meta-skill fixed, the authors cross two skill forms (an ordered workflow plan and a declarative operational ontology) with three mining-evidence levels (passing trajectories only; a mix of passes and failures with labels; the same mix with labels withheld) and evaluate the six arms on two external benchmarks, ThinkingBox-Bench (five domains) and APEX-Agents (three domains). Skills are mined once per fold and tested on held-out tasks in a five-fold design, and the metric is the per-task paired difference in pass rate between skill-enabled and no-skill runs. Main findings: on ThinkingBox-Bench, workflow plans, Goldilocks evidence, and labeled evidence show aggregate paired advantages of 1.7, 2.4, and 3.1 percentage points respectively, while APEX-Agents shows a weaker preference for ontologies and no clear evidence-regime effect; domain-level patterns are mixed and no configuration wins on both benchmarks. The authors conclude that the benefits of skill configurations are context-dependent and should be validated on target tasks. The paper is unusually explicit about its own confounds: the form contrast changes scope and skill count, the evidence contrast changes task selection and trace composition, and the label contrast changes both labels and mining instructions.","tokens_in":26029,"tokens_out":13289,"duration_ms":158368,"significance":"The study is a well-scoped, large measurement effort: 29,820 treatment runs across 994 tasks with paired task-level inference, five-fold held-out evaluation on external benchmarks, a fixed frozen meta-skill, full mining prompts and evidence-sampling details in the appendices, read-telemetry auditing, and an additional Random Blind arm simulating fully outcome-free mining. The central modest conclusion, that no single configuration performs best on both benchmarks and that gains are context-dependent, is supported by the reversal of point estimates across benchmarks and survives the acknowledged confounds because it refers to whole configurations rather than isolated components. The Goldilocks-versus-Blind pair is a particular strength, since traces are held fixed and the appendix validates the blind evidence as label-stripped in content and order, and the paper deserves credit for flagging post-hoc task explanations as hypotheses.","major_comments":[{"comment":"The aggregate workflow-versus-ontology contrast bundles the representation change together with mining scope (one global skill per fold versus one skill per task group), library size, per-skill evidence quantity, and routing burden, and Figure 4 shows that in neobank, car insurance, retail, and hotel the ontology arms often record no skill read or an off-scope read, so the +1.7 pp ThinkingBox-Bench lead in Table 3 is not identified as an effect of skill form alone. The body acknowledges this at sentence b161, but the abstract ('workflows score better than ontology by 1.7 pp') and Contribution 1 ('controlled comparisons of skill form') still present the comparison as isolating representation. Please either add a global-scope ontology arm (and ideally a per-task-group workflow arm) so the form contrast is estimated within a fixed scope, or systematically re-frame the abstract, the Introduction, and the contribution list so every claim is explicitly about the full form-plus-scope configuration.","section":"Section 3.2 / Table 3","authors_acknowledge":"The workflow and ontology configurations also differ in task grouping, skill count, and evidence quantity, so this comparison does not isolate representation alone (Section 3).","authors_acknowledge_p":0.87},{"comment":"The three headline aggregate contrasts quoted in the abstract (+1.7 pp workflow minus ontology, -2.4 pp Success minus Goldilocks, +3.1 pp Goldilocks minus Goldilocks Blind) are reported only with unadjusted 95% intervals. The Holm-adjusted p-values in A.5 are limited to within-domain baseline comparisons, where the headline domain effects are already borderline (neobank workflow Goldilocks 0.11, consulting workflow Success 0.075, car insurance Goldilocks Blind workflow 0.0501 by the sign-flip test, retail Goldilocks workflow 0.061). Please report multiple-comparison-adjusted inference for the six aggregate contrasts in Table 3 and state explicitly which of the abstract's three headline numbers survive that adjustment.","section":"Section 5.2 / Table 3 / A.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"For neobank, consulting, car insurance, and APEX-Agents, the no-skill controls are the same ten historical runs used to categorize each task in Equation (1), so for Success tasks the observed control rate is a selected sample (8-10 passes) and a no-effect arm would measure a negative lift by regression to the mean, while Goldilocks tasks are selected from a centered window; because the Success-versus-Goldilocks contrast compares disjoint task sets, this selection bias does not cancel and could contribute to the -2.4 pp aggregate. The effect is also visible in the two fresh-control domains (retail -4.5 and hotel -4.9 in Table 4), so the bias does not explain the finding away, but its share of the aggregate magnitude is unknown. Please quantify it, for example by re-estimating Success minus Goldilocks on the fresh-control domains alone, by bounding the bias from the observed baseline pass-count distribution, or by collecting fresh controls for a subset of the historical-control tasks.","section":"Sections 4.3-4.4 / Equations (1)-(2)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The claim that the Goldilocks-versus-Goldilocks-Blind comparison 'isolate[s] the effect of outcome-label visibility under a fixed experimental setup' is internally inconsistent with sentence b169, which correctly states that label access and the mining instructions change together. The comparison is valuable as a whole-condition contrast on fixed traces and paired tasks, but the word 'isolate' overstates the design; the contribution should be reworded to describe the labeled-versus-blind mining condition as a whole, matching the Section 5.2 wording.","section":"Section 1 / Contribution 2","authors_acknowledge":"The observed difference reflects the labeled versus blind mining conditions as a whole: traces are held fixed, but label access and mining instructions change together, so this comparison does not isolate label access alone.","authors_acknowledge_p":0.94}],"minor_comments":[{"comment":"The pilot drift of 25.8% versus 33.8% on 24 consulting tasks is acknowledged, but since the baseline lifts in Tables 1-2 are computed against historical controls and the headline gains are on the order of 2-4 pp, please add a short sensitivity statement quantifying how many aggregate lift intervals would change under a drift correction of that magnitude.","section":"Appendix A.4 / b107","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract's phrase 'Goldilocks blind simulating skills learnt without outcomes' is too strong, because task and run selection in that arm still uses outcome-based categories; the true outcome-free production simulation is the Random Blind extension of A.7, which shows no significant ThinkingBox-Bench gains. Please qualify the phrase or cross-reference A.7 where the 3.1 pp number is first quoted.","section":"Abstract / Section 3.3 / A.7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'matched regression of 3.1 pp' is not defined anywhere in the paper; the same quantity is called a paired difference in Section 5.2 and Table 3, so the Introduction should use the same term.","section":"Introduction / b42","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"'Variance analysis showed that tasks with multi-entity interactions resolved better with ontology' is the only occurrence of the term 'variance analysis'; either point to the task-level analysis in A.6 that supports this statement or drop the phrase.","section":"Introduction / b40","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"ThinkingBox-Bench intervals use a normal approximation treating domains as independent while APEX-Agents intervals use task bootstraps; please justify the normal approximation or compute task-bootstrap intervals for ThinkingBox-Bench as well so the two benchmarks report the same inference procedure.","section":"Tables 1 and 3 / A.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The useful thing here is the design: six arms crossing two skill forms against three evidence levels, run on two enterprise benchmarks with held-out tasks, five-fold mining, and paired-differences throughout. The matched Goldilocks/Blind setup is a real step forward—same traces, labels withheld—and the Random-Blind extension honestly simulates the production constraint of not knowing outcomes. The paper also ships unusually candid confound statements, which makes it trustworthy to read even where its claims exceed its design.\n\nWhat it actually shows: no skill configuration wins across domains, and the direction of the preference reverses between ThinkingBox-Bench and APEX-Agents. That is a real, if modest, empirical contribution, and it aligns with the growing \"skills don't always help\" literature. The domain-level tables and the audit of skill-read rates are genuinely useful for anyone deploying mined skills in enterprise agents.\n\nBut the soft spots are the headline numbers. Workflow versus ontology bundles scope, skill count, and evidence quantity (b161); success versus Goldilocks bundles task selection and trace composition (b165); even the clean-looking label contrast changes mining instructions along with labels (b169). The paper acknowledges each of these, yet the abstract and contribution list still present the 1.7pp, 2.4pp, and 3.1pp differences as controlled comparisons. They are not. The reader's stress-test note is right that the workflow-versus-ontology claim is not identified as a representation effect; a global-ontology arm would settle it. Also, no code or data is shipped, so the reproducibility burden rests entirely on the prose description, which is detailed but not a substitute for the artifacts.\n\nThe central conclusion—context-dependence, no universal winner—is still supported by the reversal in point estimates and the domain-level unevenness. The task-level explanations are explicitly post hoc, and the paper says so, which is fine. I would not reject it on the confound grounds; I would send it to review with a demand to reframe the sensitive claims and to release the skills and evaluation harness. As it stands, it is a solid, honest measurement study with a weaker-than-advertised contrast.\n\nFor peer review: yes, send it out. It deserves referee time. I would cite it for the Random-Blind setup and the paired-difference methodology, and I would bring it to a reading group focused on agent skill evaluation.","headline":"An honest, carefully designed measurement study whose headline skill-form and evidence contrasts are confounded in ways the authors themselves acknowledge; the modest context-dependence conclusion holds, but the paper overstates the control it actually achieved.","tokens_in":26471,"tokens_out":1512,"duration_ms":11982},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Mining agent skills from production traces works sometimes: neither the form of the skill nor the evidence it is mined from has a universally best configuration, and on one benchmark hiding the run outcomes costs 3.1 percentage points in…","keywords":["agent skills","skill mining","execution traces","outcome labels","workflow plan","operational ontology","enterprise agents","task-level paired differences"],"falsifier":"Run the six-arm comparison a second time on a fresh enterprise domain while holding the number of skills and per-skill evidence quantity identical between the workflow and ontology arms (one global skill per fold in both, or one per task group in both); if the workflow advantage over ontologies vanishes, the claimed representation effect does not survive. A second check: take the paper's task-structure prediction—ordered multi-step tasks favor workflows, entity- and policy-rich tasks favor ontologies—and test it on a held-out benchmark, where a task family predicted to favor one form that instead favors the other would break the task-sensitivity claim.","tokens_in":26994,"feed_emoji":"🤖","tokens_out":6690,"duration_ms":37555,"temperature":0.7,"pith_summary":"This paper asks whether an agent skill mined from production execution traces reliably helps future runs, and whether the answer depends on which traces are mined, whether the miner knows which runs succeeded, and the shape the skill takes. Holding the mining pipeline fixed, it crosses three evidence conditions with two skill forms and measures paired task-level success against no-skill baselines on two enterprise benchmarks. The headline result is that no configuration wins everywhere: on ThinkingBox-Bench workflow plans beat ontologies by 1.7 percentage points and labelled contrastive evidence beats success-only by 2.4 points, while hiding outcome labels costs 3.1 points, but on APEX-Agents ontologies are weakly preferred and no evidence regime clearly helps. The paper's practical claim is that skill design should be matched to the demands of the target tasks rather than assumed to generalize.","feed_headline":"Enterprise agent skills: best form depends on the task","feed_subtitle":"Workflows beat ontologies by 1.7 points on one benchmark; hiding outcome labels costs up to 3.1 elsewhere.","key_machinery":"The experimental design is the mechanism: a single frozen meta-skill (a distillation instruction) drives a miner that turns execution traces into skills, and the study crosses three evidence levels—success-only, Goldilocks (three passing and two failing labeled runs per task), and Goldilocks-blind (the same tasks, runs, and order with labels withheld)—with two skill forms, an ordered workflow plan and a declarative operational ontology, into six arms evaluated on tasks never mined. The isolations are the paired per-task pass-rate differences $\\Delta = \\frac{1}{N}\\sum_{x\\in D} d_x$ against no-skill baselines, and the matched Goldilocks/Blind pair, which fixes the traces and varies only label access, so the blind arm isolates the value of outcome information rather than the inclusion of failures.","core_discovery":"The paper claims that both the form of a mined skill and the evidence it is mined from change downstream agent success, but in a context-dependent way: on ThinkingBox-Bench, workflow plans beat ontologies by 1.7 points, contrastive labeled evidence (successes and failures together) beats success-only evidence by 2.4 points, and withholding the outcome labels costs 3.1 points, while APEX-Agents shows a weak preference for ontologies and no clear evidence-regime winner. The discovery is that there is no universally best configuration. It also reports that outcome labels are a real input: when the same matched traces are mined without labels, performance drops on the operational benchmark but not on the knowledge-intensive one, and within each benchmark task structure drives uneven gains and regressions. The paper's takeaway is to validate skill configurations against held-out target tasks rather than assume a one-size-fits-all design.","pith_inferences":["A matched-scope experiment—holding the number of skills and the evidence quantity equal between the two forms—would tell whether the 1.7 point workflow advantage is really about representation or about the extra skill count and broader evidence the workflow arms enjoyed; the paper leaves this open.","A testable extension would build a task-structure predictor (step count, entity count, policy density) to forecast which form and evidence regime wins per domain, then check it on a held-out domain; the paper's exploratory task-family analysis suggests such a predictor but does not construct one.","Because the Random Blind runs show unlabeled random traces still help in some domains, an inference is that mining instructions that honestly declare missing labels (extract recurring structure, no causal claims) may partially recover label value—worth testing directly against Goldilocks-blind.","Skills measurably hurt in some domains, so adding skills to an already-capable agent can backfire; a practical corollary is that skill deployment should gate on paired validation against the no-skill baseline per task family rather than on aggregate gains."],"forward_implications":["Deployment teams should tune the skill form and mining evidence to their target tasks on representative held-out examples, because the best configuration differs between benchmarks and even between task families within a domain.","Outcome-label availability is not a free choice: where privacy or compliance blocks label access, the paper measures a real cost (about 3.1 points on ThinkingBox-Bench), so label-free skill mining should be treated as its own regime rather than a drop-in substitute.","Task structure offers a usable prior: ordered, multi-step operational tasks favor workflow plans, while entity- and policy-rich tasks favor ontologies, so a domain's task mix can guide skill-form choice before any mining.","Contrastive evidence—successes and failures shown together—helps resolve decision branches in operational tasks, meaning failed traces carry informational value even when they never appear in the final procedure."],"supporting_citations":[{"why":"Supplies ThinkingBox-Bench, the stateful business-workflow benchmark whose golden-state grading defines five of the eight evaluation domains.","marker":"Li et al., 2026b"},{"why":"Supplies APEX-Agents, the professional-services benchmark whose strict binary grader defines the other three domains and produces the contrasting result landscape.","marker":"Vidgen et al., 2026"},{"why":"Prior finding of a label benefit for pools of three successes and two failures, which motivates the Goldilocks design and supports the label-access conclusion.","marker":"Jiang et al., 2026"},{"why":"Defines the agent-skills standard of a versioned instruction set loaded into context, the deployment format the whole study assumes.","marker":"Anthropic, 2025"},{"why":"Documents skill-induced failures in LLM agents, motivating the measurement of regressions and supporting the negative-lift results reported here.","marker":"Dong et al., 2026"},{"why":"Establishes paired enterprise trials under fixed execution and grading as a way to value skills, the evaluation protocol this study approximates.","marker":"Kevin et al., 2026"}],"fun_headline_variants":["Agent skill mining: best evidence and form depend on the domain","No one-size-fits-all recipe for mining agent skills from traces","Workflows beat ontologies by 1.7 points, but only on one benchmark","Hiding success labels costs up to 3.1 points on operational tasks","Match mined skill form and evidence to the target task, not to fashion"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The central claim that skill form matters rests on comparing workflow plans against ontologies where the two arms also differ in mining scope, skill count, and evidence quantity—if those differences, rather than the form itself, drive the 1.7 point gap, the representation conclusion is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Agent skill mining: best evidence and form depend on the domain","No one-size-fits-all recipe for mining agent skills from traces","Workflows beat ontologies by 1.7 points, but only on one benchmark","Hiding success labels costs up to 3.1 points on operational tasks","Match mined skill form and evidence to the target task, not to fashion"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"cost":0.00025872,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00025872,"upstream_inference_prompt_cost":0.00016492,"upstream_inference_completions_cost":9.38e-05},"total_tokens":1513,"prompt_tokens":1178,"completion_tokens":335,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":252}},"tokens_in":1178,"tokens_out":335,"cost_source":"openrouter_usage","duration_ms":5721,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":252,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-06T23:31:10.243406+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the six-arm comparison a second time on a fresh enterprise domain while holding the number of skills and per-skill evidence quantity identical between the workflow and ontology arms (one global skill per fold in both, or one per task group in both); if the workflow advantage over ontologies vanishes, the claimed representation effect does not survive. A second check: take the paper's task-structure prediction—ordered multi-step tasks favor workflows, entity- and policy-rich tasks favor ontologies—and test it on a held-out benchmark, where a task family predicted to favor one form that instead favors the other would break the task-sensitivity claim.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}