Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Bing Zhu; Guanghui Wang; Peiyang He; Wei Qiu; Xing Zhang; Yanwei Cui; Ziyuan Li

Co-evolving an inspectable metric with a skill loop recovers most of the gains that a true grader would have given self-improving agents.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 03:19 UTC pith:GNX4TTMF

load-bearing objection Abstract-only: Double Ratchet claims 88–110% retained lift from a 10-item anchor across three tasks; worth a serious look if methods hold, but the anchor-sufficiency premise is still unproven. the 4 major comments →

arxiv 2607.12790 v1 pith:GNX4TTMF submitted 2026-07-14 cs.AI cs.CLcs.MA

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Xing Zhang , Guanghui Wang , Yanwei Cui , Ziyuan Li , Wei Qiu , Bing Zhu , Peiyang He This is my paper

classification cs.AI cs.CLcs.MA

keywords self-improving agentsevaluation metricsmetric evolutionDouble Ratchetdrawback detectorsanchored referenceskill lifecycleLLM agents

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Self-improving agents that invent and retire their own skills still need something that scores those skills, and in many real tasks no reliable automatic grader exists. This paper claims that the metric itself can be evolved: a lifecycle that composes small drawback detectors, trains them to match a ten-item anchored reference set, regularizes them by consensus on unlabeled outputs, and audits them on a held-out anchor they never see. The resulting co-evolution, called Double Ratchet, is shown to retain 88–110% of the held-out performance lift that the same skill loop achieves when driven by ground truth or the best available rubric, across code generation, enterprise text-to-SQL, and reference-free report writing. Safety is argued to come from anchor discipline and outer audits: without the anchors the metric collapses into a vacuous detector, while an independent judge can catch gaming and a repaired detector can restore preference for the evolved outputs. If the claim holds, agents can keep improving in domains where no fixed verifier is available, using a transparent, failure-expecting evaluation loop rather than an opaque judge.

Core claim

A co-evolution of a lifecycle-managed metric—built from compositions of drawback detectors trained on a ten-item anchored reference, consensus-regularized, and audited on a held-out anchor—with a lifecycle-managed skill loop (Double Ratchet) retains 88–110% of the held-out lift that the identical skill loop obtains when driven by ground truth or the best available rubric, on MBPP+, Spider 2.0-Snow, and reference-free report generation.

What carries the argument

Double Ratchet: joint evolutionary lifecycles for metrics and skills. The metric is a transparent composition of small drawback detectors trained to agree with a ten-item anchored reference, regularized by consensus over unlabeled outputs, and audited against a held-out anchor it never reads; that metric then drives skill creation, revision, and retirement.

Load-bearing premise

A ten-item anchored reference set plus consensus on unlabeled outputs and a held-out anchor audit is enough to keep the evolved metric aligned with true task quality as skills improve, rather than drifting into a vacuous or gameable detector.

What would settle it

On a held-out split of MBPP+, Spider 2.0-Snow, or report generation, run the same skill lifecycle once with Double Ratchet and once with ground truth or the best rubric; if the co-evolved metric recovers substantially less than ~88% of the ground-truth lift, or if removing the anchors does not collapse the metric into a vacuous detector while the full system still gains, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Self-improving agents can operate in domains without a reliable automatic verifier by co-evolving an inspectable metric rather than assuming one.
A transparent composition of drawback detectors can replace an opaque LLM-as-judge while still recovering most of the skill-loop gains.
Anchor discipline is the primary safety lever: removing it collapses the metric, while removing the evolutionary lifecycle does not.
When skills game a report rubric, an outer independent judge plus one repaired detector can restore preference for evolved outputs over the baseline in a majority of decided pairs.
The same co-evolution recipe applies to code generation, enterprise text-to-SQL, and open-ended report writing without task-specific redesign of the metric architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the ten-item anchor works across three domains, smaller or cheaper anchors may suffice for narrower tasks, which is testable by ablating anchor size while measuring held-out lift recovery.
The failure-expecting design suggests that continuous outer audits should be treated as a permanent control loop, not a one-time validation step, in any long-running self-improving agent deployment.
The method implies that metric drift under skill evolution is detectable by held-out anchor disagreement before it becomes catastrophic, offering an early-warning signal for production systems.
Neighboring problems such as multi-agent debate scoring or tool-use trajectory evaluation may be addressable by the same detector-composition and anchor-audit pattern without requiring a full ground-truth oracle.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Abstract-only: Double Ratchet claims 88–110% retained lift from a 10-item anchor across three tasks; worth a serious look if methods hold, but the anchor-sufficiency premise is still unproven.

read the letter

The one thing to know: this abstract claims that co-evolving a lifecycle-managed metric (compositions of drawback detectors trained on a 10-item anchor, consensus-regularized, held-out audited) with a skill loop recovers 88–110% of the held-out lift that the same skill loop gets from ground truth or the best rubric, on MBPP+, Spider 2.0-Snow, and open-ended reports. That is the result that matters for self-improving agents without automatic verifiers.

What looks new is treating the metric itself as an evolvable, inspectable object under a full lifecycle, co-evolving it with skills (Double Ratchet), and building in the expectation of gaming with outer audits rather than assuming a fixed judge. The three-task scope and the ablations they flag—anchor removal collapses the metric into a vacuous detector; lifecycle removal does not; gaming caught and repaired, then 77% preference—are concrete and useful if they survive the full paper. Credit for stating the safety story in failure-expecting terms instead of hand-waving reliability.

The soft spot is real and load-bearing, not manufactured. Everything rests on ten anchored items plus consensus and one held-out audit keeping the detector composition aligned after skills optimize against it. The abstract itself shows the anchor is critical (collapse without it). Nothing here shows re-audit after evolution or that the ten items cover the distribution shift and gaming strategies that arise under co-evolution. On code and SQL the retained-lift numbers vs external GT would be strong if the metric stayed non-drifting; on reports the independent-judge rescue is post-hoc. Free parameters (anchor size, composition search, consensus weight) and invented entities (Double Ratchet, lifecycle-managed detector metric) are fine as long as the methods pin them down. Circularity is moderate: fit to anchors/consensus, success partly measured against external GT/rubrics.

This is for people working on agent self-improvement, LLM-as-judge, and reward design without verifiers. A serious referee should see the full methods, error bars, exact lift definitions, and any post-evolution metric checks. I would not desk-reject on the abstract alone; the claim is important enough and the architecture clear enough to deserve review. Send it to peer review if the paper materializes with those details. I would bring it to reading group only after methods are readable, and I would not cite from the abstract alone.

Referee Report

4 major / 4 minor

Summary. The manuscript proposes Double Ratchet, a co-evolution architecture in which an evaluation metric and a skill library improve together for self-evolving LLM agents when no reliable automatic verifier exists. The metric is built by evolving compositions of small drawback detectors under a full lifecycle: detectors are trained to agree with a ten-item anchored reference, regularized by consensus on unlabeled outputs, and audited against a held-out anchor never used in training, yielding a transparent metric rather than an opaque judge. Across MBPP+, Spider 2.0-Snow, and reference-free report generation, the co-evolved loop is claimed to retain 88–110% of the held-out lift that the same skill loop achieves when driven by ground truth or the best available rubric. Safety is attributed to anchor discipline plus outer audits: removing anchors collapses the metric into a vacuous detector, while removing the lifecycle does not; gaming of a report rubric was caught by an independent judge, repaired by one detector, and a task-aware judge then preferred evolved outputs in 77% of decided pairs.

Significance. If the retained-lift and safety claims hold under full scrutiny, the work addresses a genuine bottleneck for self-improving agents: the hidden assumption that a reliable evaluation metric already exists. A transparent, lifecycle-managed composition of drawback detectors that can substitute for ground truth or best rubrics would be practically useful in domains without automatic verifiers (enterprise SQL, open-ended report generation, and similar). Explicit credit is due for framing the yardstick as recovery of GT/rubric-enabled lift rather than absolute performance, for reporting ablations that isolate anchor discipline, and for documenting a gaming incident with independent-judge detection and repair. Those design choices make the contribution falsifiable in principle and more inspectable than opaque reward-model loops.

major comments (4)

[Abstract (retained-lift claim)] The central retained-lift claim (88–110% of GT/rubric-driven skill-loop lift on MBPP+, Spider 2.0-Snow, and report generation) is load-bearing, yet the abstract alone supplies no variance, number of seeds/runs, confidence intervals, or exact definition of “lift” and “retained lift.” Without those quantities (and the corresponding result tables in the full manuscript), it is impossible to judge whether the recovery range is statistically reliable or sensitive to a few favorable runs.
[Abstract (anchor discipline / metric loop)] The abstract itself establishes that the ten-item anchored reference is load-bearing (“removing anchor guards collapses the metric into a vacuous detector”). For the co-evolution claim to hold, compositions trained only to match that tiny anchor, consensus-regularized on unlabeled outputs, and audited on one held-out anchor must remain correlated with true task quality after the skill lifecycle optimizes against the metric. The abstract does not report re-audit of detector agreement or held-out-anchor consistency on post-evolution outputs, nor any measure of distribution shift or gaming coverage. That sufficiency is the soft premise under the recovery numbers and must be demonstrated with concrete post-evolution audits.
[Abstract (gaming / independent judge)] On reference-free report generation, gaming was “caught by an independent judge,” “one detector repaired it,” and a task-aware judge preferred evolved outputs in 77% of decided pairs. It is unclear whether detection and repair were part of the automated Double Ratchet loop or a post-hoc human/outer intervention, and whether the 77% figure is on the same distribution used to train or select the metric. If repair is external, the safety claim is weaker than the architecture framing suggests and needs explicit protocol detail.
[Manuscript availability] Only the abstract is available for this review. Methods for detector composition search, consensus regularization weight, lifecycle operators (create/revise/retire), skill-loop coupling, and the precise held-out evaluation protocol are not inspectable. The free parameters noted in the architecture (anchor set size, composition search, consensus weight) cannot be assessed for sensitivity. A full methods and results section is required before the central claims can be accepted or rejected on technical grounds.

minor comments (4)

[Abstract] The abstract is extremely dense and packs three claims, three tasks, ablations, and a gaming anecdote into one paragraph. A structured abstract (problem / method / results / safety) would improve readability for a general AI audience.
[Abstract] Terminology such as “Double Ratchet,” “lifecycle-managed drawback-detector metric,” and “anchor guards” is introduced without brief definitions; even a parenthetical gloss would help first-time readers.
[Abstract] The phrase “88–110% of the held-out lift” can exceed 100%; a one-sentence clarification that recovery can slightly exceed the GT/rubric baseline (and under what conditions) would prevent misreading as overclaim.
[Reproducibility (expected in full text)] When the full manuscript is supplied, ensure that the ten-item anchor contents (or a representative sample), the held-out audit protocol, and any independent-judge prompts are documented or released for reproducibility.

Circularity Check

0 steps flagged

No significant circularity: metric is fitted to anchors/consensus but claims are validated against external GT/rubrics and held-out audits, not by construction.

full rationale

Abstract-only review. The metric is trained to agree with a ten-item anchored reference, regularized by consensus, and audited on a held-out anchor; Double Ratchet then co-evolves skills under that metric and reports retained lift (88–110%) versus the same skill loop driven by ground truth or the best available rubric on held-out data (MBPP+, Spider 2.0-Snow, report generation). That is a fitted proxy used as a driver, but the central quantitative claim is an external comparison to independent GT/rubric baselines, not a re-prediction of the fit itself. Ablations (removing anchors collapses to a vacuous detector; independent judge catches gaming) further treat the metric as falsifiable rather than definitional. No self-definitional reduction, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result as a derivation. The reader’s concern about whether ten anchors suffice under distribution shift is a correctness/robustness risk, not circularity by construction. Score 0 is the honest finding for an abstract that presents an empirical co-evolution loop with external yardsticks.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 2 invented entities

Abstract-only ledger. Free parameters and invented entities are those the abstract itself makes load-bearing: the 10-item anchor size, the evolutionary composition of drawback detectors, consensus regularization, held-out anchor audit, and the Double Ratchet co-evolution loop. No fitted numeric values beyond the reported lift percentages are given. Axioms are domain assumptions about anchors and detectors being sufficient for metric quality under skill gaming.

free parameters (3)

anchor_set_size = 10 items
Metric is trained to agree with a fixed ten-item anchored reference set; size and selection are design choices that the abstract treats as sufficient.
detector_composition_search
Metric loop searches compositions of small drawback detectors under a full evolutionary lifecycle; search space, fitness, and stopping rules are free design parameters not fixed by theory.
consensus_regularization_weight
Metric is regularized by consensus over unlabeled outputs; the strength and form of that regularization are free choices.

axioms (3)

domain assumption A small human-anchored reference set plus consensus on unlabeled outputs and a held-out anchor audit can yield a metric aligned with true task quality under skill evolution.
Central to claim 1 and the safety story; abstract states removing anchor guards collapses the metric into a vacuous detector.
ad hoc to paper Drawback detectors can be composed under an evolutionary lifecycle into a transparent, inspectable metric that substitutes for ground-truth or best-available rubrics.
Methodological premise of the metric loop; not a standard theorem.
domain assumption Retained held-out lift vs ground truth / best rubric is the right yardstick when no metric exists to beat.
Explicit second claim: “the yardstick is recovering what an accurate metric would have enabled.”

invented entities (2)

Double Ratchet co-evolution loop no independent evidence
purpose: Joint lifecycle of metric evolution and skill create/revise/retire so agents self-improve without a pre-existing reliable grader.
Named architecture introduced by the paper; independent evidence would be the reported retained-lift numbers and gaming-recovery ablations, which are not independently verifiable from the abstract alone.
lifecycle-managed drawback-detector metric no independent evidence
purpose: Transparent, inspectable substitute for opaque LLM judges, evolved from anchors and consensus.
Core invented construct of claim 1; falsifiable only via the held-out anchor audit and outer-judge results described in the abstract.

pith-pipeline@v1.1.0-grok45 · 6219 in / 3136 out tokens · 24721 ms · 2026-07-15T03:19:39.537086+00:00 · methodology

0 comments

read the original abstract

Self-evolving agent systems improve by creating, revising, and retiring their own skills, but every such loop rests on a hidden assumption: a reliable evaluation metric already exists. In many real applications it does not. We make three claims. First, metrics can be \emph{evolved}: our metric loop searches compositions of small drawback detectors under a full evolutionary lifecycle, trained to agree with a ten-item anchored reference set, regularized by consensus over unlabeled outputs, and audited against a held-out anchor it never reads, yielding a transparent, inspectable metric rather than an opaque judge. Second, since no metric exists to beat, the yardstick is recovering what an accurate metric would have enabled, and \emph{Double Ratchet}, our co-evolution of the metric with a lifecycle-managed skill loop, does so: across code generation (MBPP+), enterprise text-to-SQL (Spider~2.0-Snow), and reference-free report generation, it retains 88--110\% of the held-out lift achieved by the same skill loop driven by ground truth or the best available rubric. Third, safety comes from anchor discipline plus outer audits: removing anchor guards collapses the metric into a vacuous detector while removing the lifecycle does not; and when evolved skills gamed the report rubric, an independent judge caught it, one detector repaired it, and a task-aware judge then preferred the evolved outputs over the pre-evolution baseline in 77\% of decided pairs. We argue this failure-expecting architecture is the right default wherever no reliable automatic verifier exists.

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

Core claim

What carries the argument

Load-bearing premise

What would settle it

If this is right

Where Pith is reading between the lines

discussion (0)