{"id":"d02d8b63-58cb-4c40-806b-612b765acd05","arxiv_id":"2608.11079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SkillZip is an evaluation-free skill compressor that finds a shortest covering explanation of an agent skill's typed contract, preserving rare rules by construction.","lead":"SkillZip compresses a growing agent skill by extracting its internal structure (rules, workflow, contracts) and reusing repeated parts once, instead of evaluating the compressed skill on test tasks. It reports 27-37% length reduction with roughly unchanged downstream performance and no rollout cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Parser coverage is the load-bearing assumption, and the paper itself flags it but provides no accuracy measurement; Eq. (3) protects only the extracted contract.","rationale":"The reader's weakest_assumption is exactly the parser's complete recovery of the typed contract, and Section III-C indeed names the parser as the main source of semantic uncertainty. My concern is the same load-bearing point, and I agree that the paper neither measures parser accuracy nor validates the renderer's semantic preservation independent of the parser. I considered several candidate concerns: (a) whether the compression rates are overstated because the length model is not the deployment tokenizer; the paper says L(·) is the rendered token cost, which plausibly matches the deployment tokenizer, and the appendix counts tokens, so this is not the weakest point. (b) Whether the theoretical preservation guarantee is circular; it is not — it is a conditional theorem and the paper explicitly scopes it to the parsed contract in Appendix D. (c) Whether SkillReducer comparisons are unfair; SkillReducer is given the same evolved skill and uses rollouts, so if anything the comparison favors the baseline; not the weakest point. (d) Whether the empirical improvement (0.577 vs 0.570) is within noise; this is real but secondary because the claim is about preservation, and small benchmark differences cannot detect silent rare-rule deletion. The parser-coverage issue is the one that, if it fails, makes the central fidelity claim inapplicable to exactly the rare rules the method promises to protect. A missing requirement is invisible to Eq. (3) because Eq. (3) quantifies only over A_req(parser). The concrete test I propose directly measures the empirical coverage of the parser, which is the missing evidence the paper itself calls for when it says 'contract recovery can be evaluated against human annotations' in Section V-A.2. This keeps the verdict CONDITIONAL: the architecture and formalization are sound conditional on the parser, but the central empirical claim of evaluation-free fidelity is not yet demonstrated for the risky part of the pipeline.","tokens_in":17878,"tokens_out":2027,"duration_ms":17285,"concrete_test":"Independent human annotation study: sample N skills (e.g., the full evolved skills for the nine model–benchmark pairs plus the appendix example), have two annotators mark every span that expresses a normative requirement (trigger, guard, modality, tool argument, workflow edge, output field, exception). Run SkillZip's parser (extract.py) on the same skills and compare A_req(parser) against A_req(human) using recall at the unit level after span-alignment. Report recall for each unit type and for low-confidence vs high-confidence parser output. If unit-level recall is below, say, 95% on any type, the fidelity guarantee in Eq. (3) does not cover the missed units and the central claim is weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that compression may change how a requirement is written but not whether it remains represented (Section IV-A). This is secured by the hard coverage constraint Eq. (3), which is defined over the parser-extracted contract A_req(S). Everything hinges on the parser recovering every normatively relevant span as a typed unit or locked residual. Section III-C explicitly calls the parser 'the main source of semantic uncertainty', and Appendix D reiterates that Prop. IV.1 protects only the contract produced by the parser. Yet the paper reports no parser precision/recall against human annotations, and the renderer's preservation is only optionally audited by reparsing with the same parser family. If the parser misses a requirement or misclassifies a guard/modality, that requirement is absent from A_req(S), so Eq. (3) is trivially satisfied for a compact representation that never mentions it. The locked-residual fallback mitigates low-confidence spans but depends on the parser's own confidence estimates being calibrated; a confidently wrong parse is not protected. This is not a question of internal consistency: the formal claim is conditional on correct extraction, and the paper states that boundary clearly. The load-bearing empirical question is whether the extraction step actually achieves complete coverage on evolved skills, which can contain implicit, cross-referenced, or highly condensed requirements (e.g., the appendix example shows duplicated rules that a parser must recognize as duplicates rather than as distinct requirements). Table I's macro-average 0.577 vs 0.570 could be achieved even with silent deletion of a rule that happens not to be exercised by the small benchmark test sets, exactly the limitation evaluation-free compression claims to avoid for rare rules.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SkillZip, a compression method for self-evolving agent skills. It treats a skill as a typed contract (interface, workflow, tool protocol, scoped rules, output contract, and evidence), extracts this contract with a schema-constrained LLM while placing uncertain spans into a locked residual, and then selects the shortest representation of the contract under a hard coverage constraint. The method is evaluation-free in that compression never observes tasks, rewards, rollouts, or behavioral verifiers. Two modes are presented: one-shot compression and Zip-on-Write continual compression. Experiments on three benchmarks and three backbone models report 27.1–36.9% compression with roughly preserved task performance, a 3.5× speedup over SkillReducer, cross-model transfer, and bounded continual growth. The formal preservation guarantee (Prop. IV.1) is explicitly conditional on the parser-extracted contract, a boundary the paper acknowledges in Sections III.C and Appendix D.","tokens_in":18150,"tokens_out":4750,"duration_ms":46919,"significance":"The paper contributes a clean and well-documented formulation: an MDL-style objective over typed contract content, a hard coverage constraint, and deterministic optimization after a single structured extraction call. The evaluation-free design is a genuine point of distinction from SkillReducer's task-based feedback loop, and the conservative locked-residual mechanism is principled. The appendices are unusually detailed for reproducibility, including schemas, prompts, a CLI, cache keys, and an atomic transaction protocol. If parser coverage can be demonstrated empirically, the method would be practically useful for maintaining evolved agent skills. However, the central fidelity claim currently rests on unmeasured parser coverage and on point estimates without variance, so the empirical core is thinner than the paper's stated conclusions.","major_comments":[{"comment":"The hard coverage constraint in Eq. (3) is defined over A_req(S), the contract produced by the parser, and Section V.A.2 states that contract recovery 'can be evaluated against human annotations,' yet no parser precision, recall, or coverage numbers are reported anywhere; Appendix D.C repeats the promise without results. The central claim in Section IV.A—that compression changes how a requirement is written but not whether it remains represented—therefore holds only relative to an extraction step whose error rate is unknown. The optional structural audit in Section V.A.5 and Algorithm 1 is not an independent check, since it reparses the rendered skill with the same parser family; a systematic parser blind spot (e.g., for implicit, cross-referenced, or highly condensed requirements such as the duplicated rules in the appendix example) would be invisible to it. Please report per-type extraction coverage against human annotations on evolved skills, including recall for rare rules and calibration of the confidence scores that route spans to the locked residual, and report renderer-level preservation using an independent re-annotation or manual audit.","section":"Section III-C, Eq. (3); Section V.A.2; Appendix D.C"},{"comment":"All nine task scores are single point estimates with no error bars, number of seeds, or significance tests. In three cells SkillZip is numerically below the Evolved Skill reference (Qwen3.7-Max BFCL-V4 0.863 vs 0.869 and Spreadsheet 0.519 vs 0.525; Kimi K2.6 BFCL-V4 0.747 vs 0.772), so the statement that SkillZip 'matches or improves the evolved skill in five of nine settings' is not established as a preservation claim. The macro-average 0.577 vs 0.570 could easily be within evaluation noise, and the held-out test-set sizes are not reported. Please provide multiple evaluation runs or bootstrap confidence intervals, report test-set sizes, and state whether each task score is a single pass or an aggregate.","section":"Table I, Section VI.C (RQ2)"},{"comment":"The four sharing mechanisms (equivalent requirements, scope lifting, workflow reuse, and guarded variants) are presented as unified under the objective in Eq. (4), but no ablation isolates any of them. The reported compression rate and fidelity could come mostly from one mechanism, or from simple deduplication, and the claim that the typed coverage constraint rather than token pruning is responsible for preserving behavior is untested. Add ablations that disable each of the four mechanisms and that replace the coverage constraint with a length-only objective, and report compression rate and task scores for each variant.","section":"Section IV.B; Appendix A.B; Appendix B.E"},{"comment":"Corollary IV.2 is presented as 'the key theoretical benefit of evaluation-free compression,' but it is a direct logical consequence of the hard coverage constraint over the extracted contract; it is a specification of the method rather than an empirical discovery. This is not an internal inconsistency, since the paper states that the guarantee is limited to the parsed contract, but the framing should be adjusted so that the rare-rule guarantee is not read as evidence of parser fidelity. The empirical question of whether rare requirements are actually extracted from evolved skills is exactly what needs to be measured.","section":"Section IV.C, Prop. IV.1 and Corollary IV.2"}],"minor_comments":[{"comment":"The Figure 4 caption names both SkillOpt and Memento-Skills, but Section VI.B only describes skill construction with SkillOpt; please clarify whether RQ1 includes Memento-Skills and how those skills are obtained, or remove the extra name from the caption.","section":"Figure 4, Section VI.B"},{"comment":"Appendix D.D promises that idempotence 'will be tested empirically by applying one-shot compression twice and reporting token and contract differences,' but no such experiment appears in the paper; either add the promised results or mark the property as a conjecture to be tested in future work.","section":"Appendix D.D"},{"comment":"The 'LLM calls' column counts only compressor-model calls, while SkillReducer's rollout column implicitly includes additional agent calls; this asymmetry should be stated in the table caption for a fairer cost comparison.","section":"Table II, Section VI.D"},{"comment":"The conclusion contains missing spaces ('SkillZiptreats', 'SkillZipuses'), and the appendix example asserts that 'every distinct output contract [is preserved] verbatim in meaning' without verification; mark example-level statements as illustrative rather than measured.","section":"Section VII and Appendix C.E"},{"comment":"The takeaway 'preserving or improving their overall performance' overstates what point estimates without error bars can support; please soften this to be consistent with the requested uncertainty reporting.","section":"Section VI.C"}],"recommendation":"major_revision","confidential_remarks":"The formal part is sound as far as it goes, and the reproducibility appendices are a genuine strength. The main risk is evidentiary: the evaluation-free claim places all weight on an unmeasured parser and on point estimates without variance. These are fixable within the manuscript's scope, so I would not reject, but the revision must add parser/renderer fidelity measurements and uncertainty-aware evaluation before the central claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. It addresses a real problem: self-evolving agents append edits to a skill document, and after enough rounds the same rule is restated in several branches while action sequences are copied rather than reused. The authors compress evolved skills by extracting a typed contract and finding the shortest representation that covers every extracted requirement — no rollouts, rewards, or task observations during compression. That is genuinely distinct from SkillReducer and from generic prompt compression, and it is the paper's strongest contribution. The formalization is clean, the cost tests in Appendix A are sensible, and the prose is notably honest: the paper explicitly calls the structural parser \"the main source of semantic uncertainty\" and Appendix D states plainly that Proposition IV.1 protects only the contract the parser produced.\n\nThe soft spot is exactly where the authors point. Equation (3)'s hard coverage constraint protects A_req(S), the parser-extracted contract. If the parser misses a requirement or misreads a guard, that requirement is absent from A_req(S), and the coverage constraint is satisfied by a compact skill that never mentions it. The locked-residual fallback helps only when parser confidence is calibrated; a confidently wrong parse is not protected. The paper never reports parser precision/recall against human annotations, and the renderer's fidelity is only optionally audited by reparsing with the same parser family — a weak check. This is a fixable omission, but it is the load-bearing empirical question, and the stress-test note lands.\n\nThe rest of the empirical core is thin but not broken. Table I shows single point estimates with no error bars; the macro-average 0.577 vs 0.570 is within noise, and four of nine cells degrade. The 27–37% compression rates are the more solid result. There are no ablations isolating the four sharing mechanisms, and the continual-mode results cover one benchmark with no variance. To its credit, the paper does not oversell: the preservation guarantee is presented as a definitional consequence of the constraint, which it is, and the local-update equivalence is explicitly conditional.\n\nMy verdict is conditional. The method plausibly works, the formal part is correct as a definition, and the paper's boundary statement is accurate. What is missing is evidence that the extraction step actually achieves complete coverage on evolved skills. Parser precision/recall on a small annotated set, error bars on Table I, and one ablation would answer most of my concerns.\n\nThis paper is for researchers working on agent skills, self-evolving agents, and skill maintenance. It deserves a serious referee. Send it to review; the right revision demands parser validation and variance estimates, not a redesign.","headline":"SkillZip is a clean, honest formalization of a real problem whose load-bearing empirical assumption — parser coverage of every normative span — is clearly flagged in the text but never actually measured.","tokens_in":18736,"tokens_out":5218,"would_cite":true,"duration_ms":43620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillZip claims that an evolved agent skill can be compressed by finding repeated structure inside the skill itself, with no tasks, rollouts, rewards, or verifiers, shrinking skills by 27–37% while preserving behavior.","keywords":["self-evolving agents","skill compression","minimum description length","typed contract","rare-rule preservation","Zip-on-Write","evaluation-free compression","LLM agents"],"falsifier":"Take evolved skills whose normative spans have been annotated by humans, run SkillZip's extraction, and measure recall of the typed units against those annotations; then delete the units the parser missed and test the rare branches those units govern. If any deleted unit changes behavior on tasks that activate its guard, the evaluation-free fidelity guarantee is falsified.","tokens_in":17688,"feed_emoji":"🗜️","tokens_out":7001,"duration_ms":62989,"temperature":0.7,"pith_summary":"Self-evolving agents improve by appending successful procedures and failure fixes, so the same rule gets restated in many branches and the same action sequence gets copied; skills grow faster than genuinely new knowledge. SkillZip's claim is that this redundancy can be removed using only the structure already inside the skill text: no downstream tasks, trajectories, rewards, or behavioral verifiers are needed. The method reads a skill as a typed contract of interface, workflow, tool protocol, scoped rules, output contract, and evidence, then selects the shortest representation that still covers every extracted requirement, with any uncertain span locked verbatim. In experiments on evolved skills, it cut token cost by 27.1–36.9% (about 31% on average) while the compressed skill's macro-average score of 0.577 matched or slightly exceeded the uncompressed skill's 0.570, and a continual Zip-on-Write mode kept skills near 1.6–1.9× seed length across 16 self-evolution rounds. A sympathetic reader cares because this makes skill maintenance a deterministic, evaluation-free cleanup instead of another rollout-hungry training loop.","feed_headline":"SkillZip shrinks evolved agent skills 31% with zero task rollouts","feed_subtitle":"Compression finds the shortest faithful explanation, protecting rare requirements without ever running tasks.","key_machinery":"The engine is the typed contract $\\mathcal{C}(S)=\\langle I,G,T,C,O,E\\rangle$ extracted from a skill: interface entries, workflow nodes and edges, tool protocol, scoped rules with modality and guard, output contract, and supporting evidence. Around this contract SkillZip builds a deterministic pipeline: a scanner that turns Markdown structure into numbered blocks, a schema-constrained extractor that returns typed units with source citations, type-compatible reuse proposal using hash matches plus a frozen relation checker, a min-cost cover optimizer that solves scope placement by dynamic programming and workflow reuse by weighted set packing, and fixed-template rendering with an optional structural audit. The objective it minimizes is $L(K)+L(R|K)$ subject to $\\forall a\\in A_{\\mathrm{req}}(S),\\ a\\preceq (K,R)$, and the cost model charges for definitions, references, and scope notation so that a shared abstraction must actually pay for itself. The same objective drives both one-shot compression and the Zip-on-Write updater, which classifies each incoming patch as absorb, refine, extend, or refactor, then periodically repacks when cross-scope reuse accrues.","core_discovery":"The central discovery is formalized as a typed minimum-description-length objective: choose a library $K$ of reusable contract elements and a residual $R$ of unique or uncertain content that minimizes $L(K)+L(R|K)$ subject to a hard coverage constraint for every extracted trigger, workflow node and edge, tool requirement, scoped rule, and output field. The paper argues that this single objective unifies four forms of reuse—collapsing equivalent paraphrases, lifting a repeated rule to its nearest common scope, factoring a repeated action sequence into a shared procedure, and writing guarded variants as one common rule plus explicit exceptions—and that each is accepted only when the shared form is literally shorter after paying for definitions, references, and scope notation. Because coverage is enforced for the parsed contract rather than for sampled tasks, preservation of a rare guard, tool argument, exception, or output field does not depend on how often any compression-time task distribution activates it (Corollary IV.2). The guarantee is deliberately limited to what the parser extracts: uncertain spans are locked verbatim, and an optional structural audit reparses the rendered skill and restores any missing span. The paper's headline claim is that compression may change how a requirement is written, but not whether it remains represented.","pith_inferences":["We infer that the practical reach of the fidelity guarantee is set by parser recall, which the paper leaves unmeasured; a human-annotated span-recall study would pin down how much of the guarantee is real in deployment.","We infer that the explain-once-reference-many objective is not tied to agent skills and could be applied to other evolving text artifacts such as memory banks, runbooks, or documentation, though the paper does not make that claim.","We infer that repeated application of the structural audit across many evolution rounds could serve as a drift detector, flagging when a restored span keeps being re-inserted by new patches; the paper does not report such longitudinal audit statistics."],"forward_implications":["Compression no longer depends on a sampled evaluation set, so a rare guard or output field survives simply because it is part of the extracted contract.","Running compression requires no rollouts, cutting measured end-to-end cost by about 3.5× compared with evaluation-guided skill compression.","Activating compression from round one of self-evolution keeps skill length near 1.6–1.9× its seed size instead of growing to 2.5–3.7×, with no test-accuracy loss in the reported runs.","Compressed skills transfer across agent backbones with about 0.97 retention on one benchmark, suggesting that explicit rules, guards, and output contracts travel better than raw evolved prose.","The structural audit can detect and restore any missing trigger, guard, workflow edge, tool argument, or output field, making under-compression the intended failure mode rather than silent deletion."],"supporting_citations":[{"why":"Primary baseline and closest prior work; its structure-aware rewriting and task-feedback loop define the evaluation-guided regime that SkillZip deliberately removes.","marker":"[1]"},{"why":"Generates the evolved skills used as compression inputs across all experiments and benchmarks.","marker":"[2]"},{"why":"Supplies the minimum description length principle that the shortest-faithful-explanation objective instantiates.","marker":"[23]"},{"why":"Provides the BFCL-v4 Web Search benchmark used to evaluate multi-step tool-use procedures.","marker":"[30]"},{"why":"Provides the LiveMathematicianBench benchmark used to evaluate precise reasoning and skill transfer.","marker":"[32]"},{"why":"Provides SpreadsheetBench, the third benchmark for measuring compression fidelity on spreadsheet manipulation.","marker":"[33]"},{"why":"Used alongside SkillOpt in RQ1 to document the skill-growth tendency in self-evolving agents.","marker":"[34]"},{"why":"Qwen3.7-Max serves as an evaluation backbone and as the fixed compressor model for one-shot SkillZip.","marker":"[27]"}],"fun_headline_variants":["SkillZip cuts agent skill size 31% without any task runs","Evaluation-free SkillZip shrinks skills 31%, keeps rare rules","SkillZip: shortest faithful skill compression, no rollouts","Zip skills 31% with no eval using SkillZip's structural MDL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the method's structural parser recovers the complete set of requirements hidden in natural-language skill text; if any requirement is missed, the hard coverage constraint protects the wrong set and the fidelity guarantee collapses.","fun_headline_variants_meta":{"raw":{"variants":["SkillZip cuts agent skill size 31% without any task runs","Evaluation-free SkillZip shrinks skills 31%, keeps rare rules","SkillZip: shortest faithful skill compression, no rollouts","Zip skills 31% with no eval using SkillZip's structural MDL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1532,"prompt_tokens":1095,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":711,"completion_tokens_details":{"reasoning_tokens":362}},"tokens_in":711,"tokens_out":437,"duration_ms":4222,"temperature":1.0,"reasoning_tokens":362,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:42:28.902616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take evolved skills whose normative spans have been annotated by humans, run SkillZip's extraction, and measure recall of the typed units against those annotations; then delete the units the parser missed and test the rare branches those units govern. If any deleted unit changes behavior on tasks that activate its guard, the evaluation-free fidelity guarantee is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the minimum description length principle that the shortest-faithful-explanation objective instantiates."},{"cited_title":"The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models,","cited_arxiv_id":null,"evidence_quote":"Provides the BFCL-v4 Web Search benchmark used to evaluate multi-step tool-use procedures."},{"cited_title":"Spreadsheetbench: Towards challenging real world spreadsheet manipulation,","cited_arxiv_id":null,"evidence_quote":"Provides SpreadsheetBench, the third benchmark for measuring compression fidelity on spreadsheet manipulation."},{"cited_title":"Qwen3.7: The agent frontier,","cited_arxiv_id":null,"evidence_quote":"Qwen3.7-Max serves as an evaluation backbone and as the fixed compressor model for one-shot SkillZip."}],"review_version":1}