{"id":"e4b854e1-81f6-4f49-bb6b-8f99b0ec4abb","arxiv_id":"2608.01772","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"FRAMES evolves an LLM agent's skill files from policy and feedback using consensus-filtered proposals, a per-category non-regression gate, and Pareto selection over accuracy and cost; it reports leading accuracy-cost results on FinDAS and tau-bench.","lead":"This paper describes FRAMES, a system that turns a company's policy documents and past audit feedback into improved instructions, called skills, for LLM agents while guarding against accuracy drops in any risk category and keeping inference cost flat. It reports the best accuracy-cost results on an internal financial document audit system and says the trend reproduces on the public tau-bench. Why read it: it is a concrete recipe for improving enterprise LLM agents without ret","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Held-out FinDAS test set is generated by the same feedback-to-case converter and from the same feedback pool that drives evolution (A.4.4, A.7.1); disjoint sampling removes identical cases but not shared failure-mode/rubric content, so the central accuracy claim is not yet independently measured.","rationale":"The reader's conditional verdict already identifies this as the weakest assumption, and my reading agrees. I considered alternative concerns -- the anti-regression proof in A.9.3 depends on assumptions A1-A3, the tau-bench cells are small, and no code is released -- but each is either secondary or already folded into the conditional verdict. The most load-bearing issue is that the internal test set is generated by the same pipeline that creates the training signal, so disjointness is insufficient for construct validity. This does not make the paper's claim false; FRAMES may genuinely improve production audits. But the empirical claim cannot be credited at face value until the measurement instrument is validated independently. Because the reader already conditioned acceptance on external validation and artifact release, my read does not move the verdict; it sharpens the reason. The concrete test above, an independent expert-authored rubric set, is the decisive check.","tokens_in":27650,"tokens_out":9328,"duration_ms":104834,"concrete_test":"Have two domain-expert auditors, blind to FRAMES outputs and not using the A.4.4 converter, independently author a fresh 210-case held-out FinDAS test set (70 per category) directly from raw production documents, policy clauses, and reviewer notes, with rubrics and expected outputs stated before seeing any FRAMES skill edits or converter outputs. Re-run the Table 1 protocol (same B1-B5 and FRAMES, k=3, paired McNemar) on this externally authored set. If FRAMES's total pass@k lead over B5 is no longer significant (p>=0.05) or the per-category margins shrink materially below the reported 0.07-0.11, the FinDAS accuracy claim is an artifact of shared rubric generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FinDAS is the primary quantitative support for the central claim. For Table 1 to establish the best accuracy-cost trade-off, the 210-case held-out set must measure production audit correctness independently of the signal FRAMES was evolved on. It does not clearly do so. Section 3.2 and A.4.4 show that every structured case, including held-out ones, is produced by the same feedback-to-case converter from the same operational feedback backlog; the converter prompt requires the rubric to encode the corrected/amended behavior precisely enough to score a fresh agent run pass/fail. The evolution loop's diagnosers, consolidator, and skill rewriter are optimized to satisfy exactly those rubrics. A.7.1's disjointness guarantees only that no case appears in both sets; it does not prevent a held-out case from sharing the same failure mode, policy clause, or converter-authored rubric structure as evolution cases. The comparison therefore estimates agreement with the converter's interpretation of reviewer feedback, not necessarily correctness under the policy corpus, unless the converter is assumed to be an unbiased gold-truth generator. The paper itself flags this class of problem in A.7.1: a method scored on signal it synthesized for itself cannot demonstrate that real corrections generalize into reusable rules. The tau-bench reproduction only partially relieves the concern: it uses public cases and manually written feedback, but the held-out cells are tiny (retail: 40 cases, with FRAMES tying B4 on total accuracy; airline: 25 cases, per-category n=5-7). Thus the FinDAS held-out result is the load-bearing evidence, and its construct validity is the principal risk. This is a test-design concern, not an allegation of misconduct; it can be settled empirically.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FRAMES, a closed-loop framework for evolving LLM agent skills in policy-governed enterprise workflows. A cold-start phase converts a policy corpus into a skill bank and turns operational feedback into structured evaluation cases; an evolution loop then mutates skills through parallel diagnosers, consensus consolidation, Pareto selection over pass rate and inference cost, and a per-category non-regression gate backed by a persistent regression pool. The main empirical claim is that on the internal FinDAS production system, FRAMES achieves the best held-out pass rate among five baselines (B1--B5) at comparable or lower inference cost, and that the same pattern is reproduced on the public tau-bench retail and airline domains. The paper further provides hyperparameter ablations, paired statistical tests for the FinDAS comparison, and an analysis of regression-pool detection under pool growth.","tokens_in":28027,"tokens_out":4975,"duration_ms":62773,"significance":"If the central claim holds, the contribution is practically significant: FRAMES improves agent skills without model retraining, keeps changes as reviewable natural-language diffs, explicitly co-optimizes cost, and enforces per-category safety floors. The paper has real strengths: the paired bootstrap and McNemar analysis in A.7.2 is a correct way to compare two methods on the same 210 held-out cases; the ablations in Table 2 isolate the most sensitive knob (batch size) and honestly report that no variant beats the default; the cost reporting distinguishes inference cost from one-time evolution cost; and the Limitations section is candid about feedback-quality dependence and evaluation cost. The main weakness is construct validity of the FinDAS held-out set: cases, rubrics, and expected outputs are generated by the same feedback-to-case LLM pipeline that drives evolution, so the central accuracy claim is not yet measured against an independently authored gold standard. The tau-bench reproduction is suggestive but per-category cells are far too small to support the abstract's wording. These issues are fixable within the manuscript's scope, but they currently block acceptance.","major_comments":[{"comment":"The central FinDAS accuracy claim is not yet an independent measurement. The converter prompt (A.4.4) requires that the rubric 'encode the corrected / amended behavior precisely enough to score a fresh agent run pass/fail', and the evolution loop is optimized against exactly those rubrics. A.7.1 states that both the evolution set and the held-out test set are drawn from the FinDAS case pool, which is itself converted from operational feedback by the same converter. Disjoint sampling (A.7.1) prevents identical cases but not shared failure modes, policy clauses, or rubric structure. The paper's own sentence in A.7.1 -- 'a method scored on signal it synthesized for itself cannot demonstrate that real corrections generalize into reusable rules' -- applies here. This is a construct-validity concern, not a statistical flaw in the McNemar test. To support the abstract's 'best accuracy-cost trad","section":"§3.2, §A.4.4, §A.7.1"},{"comment":"The tau-bench reproduction is much weaker than the abstract implies. Retail held-out special has n=6 and hallucination n=5; airline hallucination has n=7 and special n=6. With cells this small, FRAMES's numerical lead on hallucination in airline (1.00 vs. 0.86) is only a 1-case difference, and in retail FRAMES ties B4 on total (0.98) while scoring numerically below B3 and B4 on special (0.83 vs. 1.00). No paired significance test is reported for either domain. The public evidence therefore supports 'no cost penalty and competitive accuracy in a near-saturated regime', not 'the same gains reproduced on tau-bench'. Please either soften the abstract/conclusion wording or provide exact per-domain significance tests and confidence intervals for the reported differences.","section":"§A.8, Tables 4a/4b, Abstract"},{"comment":"The lifetime anti-regression claim is overstated. The proof that random-layer detection probability does not degrade with pool size relies on Assumption A3: lim inf_{|R|->inf} p_j > 0 for every actively maintained skill. The manuscript gives no mechanism that guarantees this; A2 (|S|=o(|R|)) only prevents the average share 1/|S| from vanishing, not an individual skill's share. If feedback shifts away from a skill that is later edited, p_j -> 0 and Eq. (5)'s detection probability collapses. The sentence 'Detection probability is therefore stable regardless of pool scale' should be replaced by a weaker statement, or supported by empirical cumulative-coverage measurements over the FinDAS evolution runs showing that every edited skill retained positive pool share.","section":"§A.9.3, Eqs. (4)--(5)"}],"minor_comments":[{"comment":"The default consensus mode is described as 'per-skill voting( section)' with an unresolved placeholder; this should be a numbered cross-reference to the actual section or a short definition.","section":"§3.4"},{"comment":"The text says 'FRAMES comes out ahead: it ties for the best overall quality (0.98)' -- 'ahead' and 'ties' are inconsistent. Please clarify that the advantage is on the cost/frontier dimension, not on total accuracy.","section":"§A.8.1"},{"comment":"The comparison reports only inference cost; B4 and B5 are evolution methods whose one-time evolution cost is not tabulated. The paper's claim about the 'accuracy-cost trade-off' would be more complete if Table 1 included the per-run evolution cost for B4, B5, and FRAMES, as FRAMES's own ablation does.","section":"§4.2 / Table 1"},{"comment":"The statement 'with n=k=3 each case yields a binary pass@3' is clear in context but could confuse readers who expect pass@k to use k samples per case; please define the relationship between n and k explicitly near Eq. (1) or in A.7.2.","section":"§A.7.2"},{"comment":"The word 'FRAMES' is bolded in the abstract but typeset inconsistently in the body; also, the abstract uses 'guarded' while the body uses 'per-category non-regression guard'. Minor wording consistency would help.","section":"Abstract / §1"}],"recommendation":"major_revision","confidential_remarks":"The evaluator concern about shared provenance of the held-out FinDAS set is real and load-bearing; the paper's own A.7.1 sentence concedes the principle. I would be willing to accept after the authors either provide an independent expert-scored sample or visibly weaken the central claim to 'measured against FRAMES's own converter-generated cases'. The tau-bench wording also needs to be pulled back to match the small per-category cells. The paper is otherwise well-engineered and the statistical analysis of the FinDAS comparison is a model of how such internal evaluations should be reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: this is one of the better industrial systems papers I've seen on evolving LLM-agent skills under policy. The package—cold start from policy, consensus-filtered mutation, per-category non-regression floor, cost-Pareto frontier, lifetime regression memory—is genuinely new relative to GEPA, GRASP, and SAGE, and the authors position it carefully against that concurrent work. The internal FinDAS evaluation is unusually careful: paired bootstrap on 210 held-out cases, McNemar p=0.0026, per-category reporting, and cost in the same band. That is real evidence.\n\nThe paper also does well on the systems side: prompts are in the appendix, the worked skill examples are useful, the ablation study is honest (no variant beats the default), and the limitations section is candid about benchmark scarcity and feedback-quality dependence.\n\nThe soft spot is exactly where the stress-test note lands. The FinDAS test set is built by the same feedback-to-case converter, from the same operational feedback backlog, that drives evolution. Disjoint sampling prevents identical cases but not shared failure-mode or rubric content. The paper itself concedes in A.7.1 that \"a method scored on signal it synthesized for itself cannot demonstrate that real corrections generalize into reusable rules.\" That sentence basically admits the point. So Table 1 measures agreement with the converter's interpretation of reviewer feedback, not independent correctness under the policy corpus. The tau-bench reproduction only partially relieves this because the per-category cells are tiny (n=5–7 in airline) and some results tie baselines. Also, the abstract's phrase \"anti-regression guarantee\" outruns the appendix's probabilistic, assumption-bound guard, which requires Assumptions A1–A3 (e.g., positive skill share forever). That is a wording overclaim, not a methodological sin.\n\nWho this is for: anyone building or studying self-improving agents for regulated enterprises, especially in industry. A researcher will get more from the mechanism design than from the numbers. It deserves a serious referee: the framework is specified clearly enough to reimplement on public benchmarks if the authors release the tau-bench artifacts and rubrics. My own verdict is skeptical on the magnitude of the claimed improvement, but not on the existence of the framework or the honesty of the write-up.\n\nIf I were the editor, I would send it to review with a request to release the tau-bench adaptation and to soften \"guarantee\" to \"guard.\" I would also ask the authors to either provide an external validation or explicitly separate the FinDAS-specific claim from the general one.","headline":"A serious industrial systems paper with a genuinely new combination of skill-evolution mechanisms; the internal evaluation is statistically careful but the construct-validity risk around the self-built test set means the central accuracy claim is plausible, not proven.","tokens_in":28605,"tokens_out":2119,"would_cite":true,"duration_ms":23910,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FRAMES claims that an agent's skill files—not its weights—are the right unit of continuous improvement, and that a guarded, cost-aware evolution loop can lift audit pass rate from 0.71 to 0.80 on held-out cases without retraining or extra s","keywords":["skill evolution","LLM agents","document auditing","non-regression guard","Pareto multi-objective optimization","policy compliance","enterprise workflows","feedback-driven learning"],"falsifier":"Concrete test: have a team of auditors who never saw the evolution loop write rubrics and expected outputs from the policy corpus alone for a fresh batch of real FinDAS production cases, run FRAMES' final skill set and the strongest baseline on that independently labeled batch, and check whether the per-category margins (0.07-0.11) and the paired-significance verdict (p=0.0026) survive; if they shrink to within sampling error, the reported trade-off does not extend to independently grounded truth.","tokens_in":27543,"feed_emoji":"🤖","tokens_out":7147,"duration_ms":74048,"temperature":0.7,"pith_summary":"This paper claims that an LLM agent's competences can be improved continuously, without retraining the model, by a closed loop that rewrites the agent's skill files from operational feedback. FRAMES cold-starts a deployable skill bank from enterprise policy documents alone, then evolves it: sparse, unlabeled case outcomes and reviewer notes are turned into scorable cases; a population of diagnosers proposes skill edits; their proposals are consolidated by agreement; and each candidate skill set is admitted only if it passes a per-category non-regression gate and survives Pareto competition on accuracy versus token cost. On a held-out set of 210 financial-document audit cases, FRAMES reaches the best pass rate (0.80), beating the strongest skill-evolution baseline in every category by 0.07-0.11 with paired McNemar p=0.0026, at inference cost in the same band and the lowest output cost; the qualitative pattern reproduces on the public tau-bench retail and airline domains. If right, this makes continuous, auditable skill improvement practical for regulated deployments where every change must stay reviewable and no category may silently regress.","feed_headline":"Skill-evolution loop lifts audit pass rate 9 points without retraining","feed_subtitle":"FRAMES turns scattered audit mistakes into scored cases and rewrites skill files under a per-category no-regression guard.","key_machinery":"The carrying mechanism is the guarded evolution loop. At its center is a per-category non-regression gate (Eq. 2): a candidate skill set is admitted only if its pass rate in every operational category stays within a small tolerance epsilon of the baseline P0's per-category pass rate, evaluated on the current case set plus a stratified sample of the regression pool (accumulated previously resolved cases). Edits are produced by consensus mutation—N diagnosers each analyze a reshuffled disjoint batch of failure records, a vote filter drops proposals supported by fewer than a threshold fraction of diagnosers, and a consolidator merges the survivors into one edit plan; the rewrite prompt is condi","core_discovery":"FRAMES' central claim is that guarded, dual-objective evolution of natural-language skill procedures is a viable and auditable way to improve LLM agents in production. Rather than fine-tuning weights or tuning a flat prompt, it treats the agent's skill bank as the unit of improvement: skills are rewritten from policy and feedback, and each edit is a versioned natural-language diff that a compliance officer can review. The evolution loop is protected by a per-category non-regression gate anchored at the start-of-run baseline P0, so a gain in one risk category cannot mask a loss in another, and by a Pareto frontier over pass rate and token cost that yields a deployment menu rather than a singl","pith_inferences":["The same dual-objective machinery could be applied per-case rather than per-skill-set: if each frontier operating point is paired with a cheap classifier that routes cases by risk level, a deployment could serve high-risk cases with the accurate (more expensive) skill set and bulk cases with the cheap one, generalizing the frontier menu one level down.","Consensus mutation is a text-space analog of ensemble bagging; one testable extension is to measure how diagnosis diversity (batch reshuffling, model variation, temperature) trades against the vote threshold, and whether a threshold calibrated per category outperforms the fixed 40% default.","Because the gate and the Pareto competition both read the evolution set and regression pool, the framework's lifetime guarantee is only as clean as the case-conversion pipeline; a natural extension is to maintain a small, human-curated 'gold' case set that is never regenerated by the converter, as a permanently independent guard on the held-out estimate.","The tau-bench generalization claim rests on small held-out pools (40 and 25 cases), so a public re-run on a larger failure-annotated benchmark—or a second enterprise domain—would be the most direct way to test portability beyond FinDAS."],"forward_implications":["Evolution of skills, not weights, is a practical unit of continuous improvement in regulated agent deployments: every change remains a reviewable, versioned diff and does not require rebuilding the model or re-labeling data.","The per-category gate prevents the masking effect—a flat total can hide a regression in one risk category offset by a gain in another—so gains are broad rather than concentrated; the observed 0.07-0.11 per-category lead over B5 is presented as direct evidence (paired bootstrap and McNemar p=0.0026).","Cost is a first-class objective: because output tokens dominate per-call latency, FRAMES claims the lowest output cost while matching baseline input cost, so the accuracy gain does not come from spending more at serving time.","Lifetime anti-regression memory is sustainable: previously resolved cases become permanent regression tests, and the two-layer sampling (skill-targeted plus stratified-random) keeps regression-detection probability independent of pool size under Assumptions A1-A3.","The cold start requires only a policy corpus (feedback optional), so the framework can be applied in other policy-governed domains, with tau-bench retail and airline providing generalization evidence in a near-saturated regime and a small pool respectively."],"supporting_citations":[{"why":"AutoSkill, the strongest baseline (B5) whose skill-evolution mechanism FRAMES must beat; supplies the seed skill bank, feedback, and policy agent for the comparison.","marker":"Yang et al., 2026"},{"why":"tau-bench, the public tool-agent benchmark whose retail and airline domains are used as a generalization probe and whose harness FRAMES builds on.","marker":"Yao et al., 2025"},{"why":"The unbiased pass@k estimator that defines the quality metric PR(S) for all methods.","marker":"Chen et al., 2021a"},{"why":"DeepAgents, the execution and evaluation harness in which the policy agent runs.","marker":"LangChain, 2024"},{"why":"Claude Sonnet 4.6 with thinking enabled, the frozen model used across all FRAMES modules and FinDAS evaluation.","marker":"Anthropic, 2026"},{"why":"GEPA, a concurrent reflective prompt-evolution method contrasted in A.10 for lacking cold start, cost objective, and per-category gates.","marker":"Agrawal et al., 2026"},{"why":"GRASP, a regression-aware skill proposer whose global gate is contrasted with FRAMES' per-category floor.","marker":"Moll et al., 2026"},{"why":"SkillsBench evidence that one-pass prompting a model to write its own skills barely helps, motivating iterative skill-file refinement.","marker":"Li et al., 2026"},{"why":"LAVA, the prior financial document-auditing framework whose policy-bound workflow FinDAS extends.","marker":"Shu et al., 2025"}],"fun_headline_variants":["Guarded skill evolution yields best accuracy-cost trade-off","No-regression Pareto loop lifts audit accuracy 9 points","Proven skill-evolution loop: +9 pass rate, zero retraining","Anti-regression skill edits: 9-point audit gain at same cost","Skill bank Pareto-mutated: better audits, no cost inflation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The accuracy-cost claim rests on the held-out FinDAS test set measuring production audit correctness without being contaminated: those cases, rubrics, and expected outputs were generated by the same LLM feedback-to-case conversion pipeline (and manual expert verification) from the same operational feedback pool that drives the evolution loop, so disjoint sampling does not by itself guarantee independence of content.","fun_headline_variants_meta":{"raw":{"variants":["Guarded skill evolution yields best accuracy-cost trade-off","No-regression Pareto loop lifts audit accuracy 9 points","Proven skill-evolution loop: +9 pass rate, zero retraining","Anti-regression skill edits: 9-point audit gain at same cost","Skill bank Pareto-mutated: better audits, no cost inflation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001348,"raw_usage":{"total_tokens":5268,"prompt_tokens":659,"completion_tokens":4609,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":4530}},"tokens_in":403,"tokens_out":4609,"duration_ms":39157,"temperature":1.0,"reasoning_tokens":4530,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:11:27.935247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Concrete test: have a team of auditors who never saw the evolution loop write rubrics and expected outputs from the policy corpus alone for a fresh batch of real FinDAS production cases, run FRAMES' final skill set and the strongest baseline on that independently labeled batch, and check whether the per-category margins (0.07-0.11) and the paired-significance verdict (p=0.0026) survive; if they shrink to within sampling error, the reported trade-off does not extend to independently grounded truth.","supporting_citations":[{"cited_title":"Proceedings of The 10th Workshop on Financial Technology and Natural Language Processing , month = nov, year =","cited_arxiv_id":null,"evidence_quote":"LAVA, the prior financial document-auditing framework whose policy-bound workflow FinDAS extends."}],"review_version":1}