Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Reward-Free Evolving Agents via Pairwise Validator

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A frozen pairwise validator can replace the per-step reward in self-evolving agent loops, matching or beating reward-gated baselines on most evaluated tasks without the labeling cost.

desk verdict The pairwise-gate idea is real and the prompt-substrate evidence is suggestive, but the code-substrate 'reward-free' claim is not supported: the only fully reward-free variant loses 6 of 10 cells, and the winning code variants keep a scalar reward at parent selection. read the letter →

arxiv 2607.14408 v1 pith:FZDMR376 submitted 2026-07-15 cs.AI

Minghao Liu , Yu Wang , Jiayun Wang , Wei Wei This is my paper
classification cs.AI
keywords pairwisevalidatorself-evolvingagentsreward-freeevolutionLLM-as-judgeaccept/rejectgatepromptoptimizationcodeEloratings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the expensive per-iteration reward in self-evolving agent loops can be replaced by a pairwise validator—a frozen language model that judges whether a proposed child candidate is better than its parent. Across prompt and code evolution, and across several agent model families, the binary-verdict gate matches or exceeds the full-reward baseline on the majority of evaluated cells, and the pattern persists when the validator is swapped for a different model family. The claim matters because reward design—clean labels, domain expertise, score calibration—is often the costliest part of building an agent that improves itself; if a frozen judge can substitute, self-evolution becomes much cheaper. The paper offers two variants: one keeps the existing validation rewards for parent selection, and the fully reward-free 'Soft Elo' variant drops validation rewards entirely, driving parent selection with Elo ratings updated by validator verdicts.

What carries the argument

The carrying object is the pairwise validator gate: a frozen LLM that takes parent and child outputs plus a small context and returns a binary 'better/worse' verdict under greedy decoding; the child is accepted when it wins at least as many train-minibatch verdicts as the parent. Two configurations hang off this gate: Adaptive Focus periodically refreshes a rubric (accuracy, soundness, or holistic) that focuses the validator, while Soft Elo replaces reward-based parent selection with an Elo rating system updated by the validator's verdicts, weighted by the agent's token confidence. The gate is what does the work; parent selection is the controlled axis.

What would settle it

Take a task where the validator's preferences can be measured against ground truth by collecting human or programmatic labels on a sample of parent-child pairs, and check whether gate acceptances correlate with true task-score improvements. If, on a task class where the validator systematically disagrees with the reward, the reward-free loop's test score drops below the full-reward baseline by more than the paper's reported margins, the central claim is falsified. Concretely: run the validator-swap setup on a task with high style-over-substance risk and measure whether the gate's accept/reject

Watch

Extended reading notes

Core claim

The central discovery is that the accept/reject decision in a self-evolving agent loop does not need a scalar task reward. A frozen LLM queried with the parent and child candidates and asked which is better—one binary verdict per example, greedy decoding—can serve as the gate. On the paper's evaluations this pairwise gate matches or exceeds the reward-gated baseline on most cells, and the fully reward-free version (Soft Elo) also drops validation rewards without collapsing performance. The authors attribute part of the gain to the contrastive nature of pairwise judgment: it avoids scale calibration and, they conjecture, carries generalization signal that scalar validation rewards do not. The

Load-bearing premise

The binary verdicts of the frozen validator must track true task quality on the mutated children it is asked to judge; if the validator prefers style over correctness or struggles on a task type, the gate adds noise and the claimed match to the reward baseline breaks down.

Editorial extensions

If this is right

  • Self-evolving agents can be built with no per-step reward design: a frozen judge plus a few in-context examples replaces labeled scalar rewards at the gate.
  • The fully reward-free configuration (Soft Elo) shows that even the validation-set reward used for parent selection can be dropped, which lowers the labeling burden further.
  • The pattern transferring across a validator swap suggests the method is not an artifact of a model judging itself; it should hold with an external judge.
  • On cells where validation saturates early, the pairwise gate shows a smaller validation-to-test gap than the reward baseline, implying better generalization in low-data regimes.
  • The same gate applies to both prompt artifacts and code artifacts, so reward-free evolution is not substrate-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension: measure agreement between validator verdicts and true task reward on rejected children; if disagreement concentrates on style-over-substance decisions, adding a cheap objective check to the gate could recover the cells where the validator mismatches the task.
  • If pairwise comparison indeed carries generalization signal, the approach may transfer to settings where no reward exists at all, such as open-ended exploration or quality-diversity search, where 'better' is inherently relative.
  • The Elo formulation suggests a deeper connection: the loop is performing preference-based policy improvement without a learned reward model; one could compare it directly against a trained reward model at equal total LLM-call budgets.
  • The paper's cost analysis implies the reward-free gate costs about 1.7x the baseline in LLM calls (and 2.4x for Elo variants); a reader should weigh that compute premium against labeling cost, since the wall-clock and token-spend ratios are flatter than the call-count ratio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing the per-step scalar task reward at the accept/reject gate of self-evolving agent loops with a frozen LLM pairwise validator. Two variants are introduced: Adaptive Focus, which keeps the engine's validation-reward parent selection, and Soft Elo, which replaces parent selection with Elo ratings driven by validator verdicts to become fully reward-free. The method is integrated into GEPA, ADRS, and ShinkaEvolve and evaluated on prompt and code evolution tasks. The central claim is that the pairwise gate can substitute for per-step reward design at competitive task accuracy and without labeling cost.

Significance. If established, the contribution is practically valuable: pairwise LLM judgment is cheaper and more stable than scalar reward design, and a training-free gate would lower the barrier to self-evolving agents. The paper has clear strengths: it integrates the gate into three published engines, includes a cross-family validator swap to address self-pairing concerns, provides ablations of the gate context and Elo step size, and is unusually candid about saturation, small-N, and validator-capability limitations. The analytical cost decomposition in Appendix H is also a useful addition. The main weakness is that the strongest code-substrate evidence is not actually reward-free: the fully reward-free Direct variant loses most cells, and the wins are carried by variants that retain a scalar reward at parent selection. The headline 'reward-free across two artifact substrates' is therefore not yet supported.

major comments (4)
  1. [§5.4, Table 5; Abstract] The cross-substrate claim in the abstract ('two artifact substrates ... without the labeling cost') is not supported by the code experiments. In Table 5, the four validator-gated variants differ only in what reward signal feeds parent selection. The only fully reward-free variant, Direct, wins 4 of 10 cells against the full-reward baseline; the paper itself states that 'Direct + full reward carries the cross-substrate claim' (5/10) and that adding 'Direct + easy reward' brings Direct-gate variants to 6/10. Both of those winning configurations keep a scalar reward at parent selection. Thus the evidence shows that a pairwise gate can be added on top of an existing reward-based selector, not that the scalar reward can be removed in code evolution. This is load-bearing for the paper's central reward-free claim and must be addressed, either by reporting fully reward-free code-evolution result
  2. [§5.2, Table 2] The headline that the method 'matches or exceeds full-reward GEPA on all 20 rows' relies on taking the per-row best across the six/seven validator variants in the slate. This is not a single drop-in replacement. Individual variants often underperform the baseline substantially; e.g., Table 2, Qwen3-8B HotpotQA: Soft Elo scores 59.3 and Plain Elo 64.3 versus the 77.0 full-reward baseline. If the intended claim is about the validator-gate family as a whole, that should be stated explicitly and aggregated appropriately; if the claim is about a specific configuration, the paper needs to identify that configuration and support it with multiple seeds. The abstract's 'our method' phrasing overstates what the table shows.
  3. [§5.2, Table 2; Appendix A, Table 6] Main tables contain no error bars or seed information, yet Appendix A reports large seed-to-seed variation for Soft Elo. For example, Soft Elo on HotpotQA with k=32 gives 57.6, 68.4, and 61.2 across three seeds, a spread of more than 10 points, while Table 2 reports 59.3 for the same configuration. The appendix value is not among the three seeds reported in Table 6, so the provenance of the main-table number is unclear. Without a clear seed policy and variance reporting, per-cell deltas of a few points — which are central to the 'majority of settings' and 'matches or exceeds' claims — are not interpretable. The authors should report all seeds or justify why single-seed reporting is adequate for these conclusions.
  4. [§5.2, Figure 2 and discussion] The generalization claim that the pairwise formulation 'carries generalization signal that scalar validation or training rewards do not' is presented as a conclusion from three illustrative cells. The evidence is indirect: the validation-to-test gap is compared at the best-validation checkpoint on small test sets (e.g., 15 problems for AIME, 30 for LiveBench-Math), where single problems move scores by 3.3–6.7%. Moreover, Adaptive Focus still selects parents using validation rewards, so the statement that 'our pairwise gate does not depend on V at decision time' cannot explain that variant's behavior. This claim should be softened to a hypothesis or supported with a controlled comparison that isolates the gate's effect on generalization.
minor comments (5)
  1. [Abstract] The abstract says 'our method matches or exceeds the full-reward baseline on the majority of settings.' Given that the results are reported as a per-row best across a slate of variants, the abstract should say 'a variant of our method' or 'at least one validator-gated variant' to avoid overclaiming.
  2. [§5.4] The scoring rule changes between substrates: prompt evolution uses test score at the best-validation checkpoint, while code evolution uses test score at the final iteration. This makes cross-substrate comparisons and the 'same pattern' narrative harder to interpret. Please state why the code substrate could not use a validation-based checkpoint or report both checkpoint rules.
  3. [Table 5] The txn_scheduling rows are described in the text as 'full-reward baseline retains its lead,' but the reported numbers are ambiguous about whether higher or lower is better. Please clarify the metric direction in the table caption or text, since it directly affects whether Direct's 3663 vs. 3937 is a win or a loss.
  4. [§4.3, Eq. (4)] The confidence weight c_t is defined as the mean per-token probability of the agent's previous response. This requires access to token-level logprobs. The fallback Plain Elo is described, but it would help to state explicitly in the main text which of the evaluated models expose per-token logprobs and which required the Plain Elo fallback.
  5. [Appendix A] The Soft Elo k sweep reports only three seeds and no aggregate statistics. Since the main text presents k=32 as the default, the sweep would be more useful with a per-k mean and standard deviation across seeds, and with a statement of how the main-table Soft Elo values were selected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the comparison is empirical against independent external baselines, with minor reporting caveats that are not circular.

full rationale

The paper's chain is an empirical evaluation, not a derivation: Eq. (2) defines the baseline gate on reward r, and Eq. (3) substitutes a frozen validator V_phi's binary verdicts. There is no fitted parameter that is later renamed as a prediction, no equation that reduces to an earlier equation, and no self-citation chain: none of the load-bearing references are authored by the present paper's authors. The validator is frozen and training-free; Adaptive Focus and Soft Elo are evaluated against each engine's published full-reward baseline on held-out test splits, which is independent evidence outside the paper's own fitted values. The only notable caveats are reporting choices: headline prompt-evolution rows use per-row best across a slate of six variants, and in the code substrate the strongest variant (Direct + full reward, Table 5) retains the engine's full scalar reward at parent selection, so the code rows support gate replacement but not a fully reward-free loop. The paper discloses both facts (Tables 2/5 and §5.4). These are scope and interpretation issues, not circular steps; no prediction is equivalent to its input by construction.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of LLM pairwise judgments as a reward proxy, plus several hand-chosen hyperparameters (minibatch size, Elo step, exemplar count, rubric refresh period, softmax temperature). No new entities are introduced. The validation checkpointing and reward-labeled few-shot variants soften the 'reward-free' framing.

free parameters (9)
  • Elo base step k = 32 (Soft Elo); 64 (Plain Elo)
    Eq. (4); swept in Appendix A over {16,32,64,128} with no clear winner; main results use k=32.
  • Elo archive comparisons E = 3
    Appendix F/H: each iteration matches child vs parent plus 3 randomly sampled archive members; chosen by hand.
  • Train minibatch size b=|Tt| = 3
    §4.1 and Appendix H: validator queries 3 examples per gate decision; chosen by hand.
  • In-context exemplar count N (few-shot) = 3
    §4.1/Appendix B: swept {1,3,5,8}, no monotone trend; default N=3.
  • Rubric refresh period = 10 iterations
    §4.2: meta-LLM refreshes rubric every 10 iterations; chosen by hand.
  • Rubric set = Accuracy, Soundness, Holistic
    §4.2: three author-defined rubrics; the refresh mechanism picks among them.
  • Acceptance threshold = majority of 3 verdicts (child wins >= 2)
    §4.1: gate accepts if child wins at least as many per-example verdicts as parent.
  • Parent softmax temperature = rating / 200
    Appendix F: next parent sampled from softmax over ratings divided by 200.
  • Confidence weight ct = mean per-token probability of agent's previous response
    §4.3 Eq. (4): per-iteration Elo update weight; design choice with no theoretical grounding; fallback Plain Elo when logprobs unavailable.
assumptions (5)
  • domain assumption Pairwise LLM comparison is more reliable and stable than absolute LLM scoring
    Invoked in §1/§2 citing RLHF and Chatbot Arena; underpins the whole gate design.
  • domain assumption Each engine's mutator M is fixed and proposes children whose quality ordering by reward and by validator are correlated
    §3 Eq. (1); the result is measured only under the three published engines; if the mutator produced mostly trivial children the gate would be uninformative.
  • domain assumption The small validation pool V is a usable proxy for test performance for checkpoint selection
    §3 and §5.1; the paper deliberately uses |V| in [12,30], and reports test at best-validation checkpoint; with small V the checkpoint may overfit V.
  • ad hoc to paper Per-token logprobs of the agent's output are available and their mean is a useful per-iteration confidence weight ct
    §4.3 Eq. (4); no theoretical grounding for using mean per-token probability as Elo update weight; fallback Plain Elo exists when logprobs unavailable.
  • domain assumption The objective task rewards (exact match, sum of radii, makespan) are correct and comparable across methods
    §5.1; if the evaluator itself is noisy, the comparison is meaningless.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reward-Free Evolving Agents via Pairwise Validator." pith.science (2026). https://pith.science/paper/FZDMR376

@misc{pith2026260714408,
  author       = {Pith},
  title        = {Pith review of: Reward-Free Evolving Agents via Pairwise Validator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZDMR376}},
  note         = {Machine review of arXiv:2607.14408}
}
read the original abstract

A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal. Designing that signal is often the costly part of the project: a reliable scalar reward requires domain expertise and labeled examples that are themselves as expensive to assemble as the agent's underlying task. We propose replacing the scalar at the accept/reject gate with a pairwise validator: a frozen LLM that, given the parent and child candidate, returns a binary verdict on which is better. Pairwise judgment is generally easier and more stable than absolute scoring, due to its contrastive nature, which mitigates the need for strict scale calibration. The validator also requires no training of its own. We integrate the validator into three published self-evolving engines (GEPA, ADRS, ShinkaEvolve) and report two flavors: Adaptive Focus, which retains the engine's existing val-set parent selection, and Soft Elo, which lets the validator's verdicts drive parent selection so that val-set rewards drop as well. Across multiple agents and two artifact substrates (prompt and code), our method matches or exceeds the full-reward baseline on the majority of settings we evaluate, and the pattern survives a cross-family validator swap. The pairwise gate is thus a drop-in replacement for per-step reward design at competitive task accuracy without the labeling cost.

Figures

Figures reproduced from arXiv: 2607.14408 by the authors.

Figure 1
Figure 1. Overview of our framework. A self-evolving agentic loop iterates over propose, accept-or-reject (the gate), and select. The gate traditionally uses a per-step task reward that compares parent and child prompt candidate scores. We replace that reward with a pairwise validator (LLM) that returns a binary verdict over parent and child. From left to right, the full-reward baseline (left) keeps both train and val rewards… view at source ↗
Figure 2
Figure 2. Per-iteration GEPA trajectories where the validation-to-test gap is most visible. Panels: HoVer (Gemma￾4), AIME (Qwen3-8B (thinking)), IFBench (Qwen3-8B). Solid line = running validation-best trajectory; right-edge marker = per-variant best test score (matches cells in Tables 2 and 3). Color: red full-reward GEPA; green train-reward￾free family (Direct, Few-shot (with rewards), Adaptive Focus); blue fully reward-fre… view at source ↗
Figure 3
Figure 3. shows one full iteration of the loop, top to bottom. Each level is one LLM-call site; the right-side annotation shows the symbolic per-iteration cost. The reader can sum down the diagram to obtain the per-iteration total for any condition; Appendix H carries the formal per-condition formulas and the cost ladder. Level 1: Reflection mutation Level 2: Parent subsample eval Level 3: Child subsample eval improved? Level… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Soft Elo / Plain Elo gate. Left: per iteration the child plays (1+E) = 4 pairwise matches (parent + E = 3 random archive members), each consisting of b = 3 validator queries on Tt — 12 validator calls per iteration. Right: every verdict triggers the Elo update of Eq. (…
Figure 5
Figure 5. Figure 5: Per-cell LLM-call breakdown by condition (medians across the six headline tasks). Each segment maps to one level in [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 9 linked inside Pith

  1. [1]

    GEPA: Reflective prompt evolution can outperform reinforcement learning

    Lakshya A Agrawal et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457, 2025

    arXiv 2025

  2. [2]

    Constitutional AI: Harmlessness from AI feedback

    Y untao Bai et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  3. [3]

    Quality-diversity through AI feedback

    Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through AI feedback. arXiv preprint arXiv:2310.13032, 2023

    arXiv 2023

  4. [4]

    Barbarians at the gate: How AI is upending systems research

    Audrey Cheng et al. Barbarians at the gate: How AI is upending systems research. arXiv preprint arXiv:2510.06189, 2025

    arXiv 2025

  5. [5]

    Jordan, Joseph E

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In ICML, 2024

    2024

  6. [6]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS, 2017

    2017

  7. [7]

    Meyarivan

    Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation , 6(2):182–197, 2002

    2002

  8. [8]

    Quality diversity through human feedback: Towards open-ended diversity-driven optimization

    Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback: Towards open-ended diversity-driven optimization. In ICML, 2024

    2024

Show all 53 references
  1. [9]

    Open questions in creating safe open-ended AI: Tensions between control and creativity

    Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended AI: Tensions between control and creativity. In ALIFE, 2020

    2020

  2. [10]

    Arpad E. Elo. The Rating of Chessplayers, Past and Present . Arco Publishing, New Y ork, 1978

    1978

  3. [11]

    A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence

    Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, et al. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence. arXiv preprint arXiv:2507.21046, 2026. 9 Reward-Free Evolving ...

    2026 arXiv

  4. [12]

    AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms, May 2025

    Google DeepMind. AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms, May 2025

    2025

  5. [13]

    Direct preference-based evolutionary multi-objective optimization with dueling bandits

    Tian Huang, Shengbo Wang, and Ke Li. Direct preference-based evolutionary multi-objective optimization with dueling bandits. In NeurIPS, 2024

    2024

  6. [14]

    HoV er: A dataset for many-hop fact extraction and claim verification

    Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. HoV er: A dataset for many-hop fact extraction and claim verification. In Findings of EMNLP, 2020

    2020

  7. [15]

    ShinkaEvolve: Towards open-ended and sample- efficient program evolution

    Robert Tjarko Lange, Y uki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards open-ended and sample- efficient program evolution. arXiv preprint arXiv:2509.19349, 2025

    2025 arXiv

  8. [16]

    Feedback descent: Open-ended text optimization via pairwise comparison

    Y oonho Lee, Joseph Boen, and Chelsea Finn. Feedback descent: Open-ended text optimization via pairwise comparison. arXiv preprint arXiv:2511.07919, 2025

    2025

  9. [17]

    Do llms understand code preference? training code preference models via synthetic code evolution

    Jiawei Liu, Thanh Nguyen, Mingyue Shang, Hantian Ding, Xiaopeng Li, Y u Y u, V arun Kumar, and Zijian Wang. Do llms understand code preference? training code preference models via synthetic code evolution. In DL4C Workshop, ICLR, 2025

    2025

  10. [18]

    American Invitational Mathematics Examination (AIME)

    Mathematical Association of America. American Invitational Mathematics Examination (AIME). Annual com- petition problem set, 2024

    2024

  11. [19]

    OpenEvolve: An open-source evolutionary coding agent

    OpenEvolve Contributors. OpenEvolve: An open-source evolutionary coding agent. GitHub repository, 2025

    2025

  12. [20]

    Generalizing verifiable instruction following

    V alentina Pyatkin et al. Generalizing verifiable instruction following. In NeurIPS Datasets and Benchmarks Track, 2025

    2025

  13. [21]

    PAPILLON: Pri- vacy preservation from internet-based and local language model ensembles

    Li Siyan, V ethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Y u Wu. PAPILLON: Pri- vacy preservation from internet-based and local language model ensembles. In arXiv preprint arXiv:2410.17127, 2024

    2024 arXiv

  14. [22]

    ScoreFlow: Mastering LLM agent workflows via score-based preference optimization

    Yinjie Wang et al. ScoreFlow: Mastering LLM agent workflows via score-based preference optimization. arXiv preprint arXiv:2502.04306, 2025

    2025 arXiv

  15. [23]

    LiveBench: A challenging, contamination-limited LLM benchmark

    Colin White, Samuel Dooley, et al. LiveBench: A challenging, contamination-limited LLM benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  16. [24]

    Cohen, Ruslan Salakhutdinov, and Christo- pher D

    Zhilin Y ang, Peng Qi, Saizheng Zhang, Y oshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018

    2018

  17. [25]

    ExPairT-LLM: Exact learning for LLM code selection by pairwise queries

    Tom Y uviler and Dana Drachsler-Cohen. ExPairT-LLM: Exact learning for LLM code selection by pairwise queries. In AAAI, 2026

    2026

  18. [26]

    Darwin Gödel machine: Open-ended evolution of self-improving agents

    Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel machine: Open-ended evolution of self-improving agents. In ICLR, 2026

    2026

  19. [27]

    AFlow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Y u, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Y uyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024

    2024 arXiv

  20. [28]

    MOEA/D: A multiobjective evolutionary algorithm based on decomposition

    Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6):712–731, 2007

    2007

  21. [29]

    Evolution without an oracle: Driving effective evolution with LLM judges

    Zhe Zhao, Y uheng Y ang, Haibin Wen, Xiaojie Qiu, Zaixi Zhang, and Qingfu Zhang. Evolution without an oracle: Driving effective evolution with LLM judges. arXiv preprint arXiv:2511.19489, 2025

    2025

  22. [30]

    X was born in Y

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Y onghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P . Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. 10 Reward-Free Evolvi...

    2023

  23. [31]

    A conceptual or philosophical question

  24. [32]

    Specific formatting requirements (e.g., dialogue script, word limit, required keywords)

  25. [33]

    Constraints on ambiguity, interpretation, or thematic elements ## Task Requirements - Generate a response that directly addresses the question - Ensure the response is concise and adheres to formatting constraints - Incorporate required keywords or thematic elements - Maintain...

  26. [34]

    **Interpret the question and constraints carefully**: Understand the core of the question and the specific formatting, thematic, and linguistic requirements.,→

  27. [35]

    **Structure the response to meet formatting and content requirements**: Ensure that the response follows the specified format (e.g., poem, dialogue, list) and includes all required elements.,→

  28. [36]

    **Balance creativity with clarity and logical consistency**: Use imaginative language and metaphors where appropriate, but ensure that the response remains coherent and logically sound.,→

  29. [37]

    **Use ambiguity where appropriate without sacrificing coherence**: Introduce open-ended interpretations or multiple layers of meaning, but ensure the response remains grounded and understandable.,→

  30. [38]

    **Ensure all required elements are included**: Verify that all keywords, formatting rules, and thematic constraints are met.,→

  31. [39]

    Pay careful attention to all constraints and formatting requirements

    **Maintain a natural, engaging tone while adhering to constraints**: Use language that is expressive and engaging, while still following the structural and content guidelines.,→ ## Domain-Specific Knowledge - Responses should reflect an understanding of philosophical or concep...

  32. [40]

    **Break down the claim** into its component parts to understand what is being asserted

  33. [41]

    **Analyze each piece of evidence** to determine if it directly or indirectly supports, contradicts, or is neutral with respect to the claim.,→

  34. [42]

    **Use logical reasoning** to assess the overall support level of the claim

  35. [43]

    Scary Movie 5

    **Provide a clear, structured, and well-reasoned response** that includes: - A **breakdown of the claim** and its components. - An **analysis of the evidence** and its relevance. - A **conclusion** indicating whether the claim is **SUPPORTED**, **NOT_SUPPORTED**, or **PARTIALL...

    1920

  36. [44]

    **Break down the claim** into its individual assertions

  37. [45]

    **Analyze each piece of evidence** to see if it supports, contradicts, or is neutral with respect to the claim

  38. [46]

    **Use external knowledge** when necessary to verify factual claims, but clearly state that you are using such knowledge.,→

  39. [47]

    **Determine the overall support level** based on whether all parts of the claim are supported or if any part is not

  40. [48]

    Scary Movie 5

    **State your final verdict** clearly as either **SUPPORTED**, **NOT_SUPPORTED**, or **PARTIALLY_SUPPORTED**, and provide a **concise and well-reasoned justification**.,→ ## Task Inputs Format - The **claim** is a statement that you must evaluate. - The **evidence** is a list o...

    1920

  41. [49]

    a user,"

    **Identify and Remove PII:** - Scrutinize the input for names, locations, specific identifiers, or sensitive information. - Replace or generalize these elements with neutral, non-specific terms (e.g., "a user," "a team member," "a character," "a community," "a region").,→ - En...

  42. [50]

    - Ensure that the rephrased content conveys the same ideas and nuances as the original

    **Preserve Core Meaning and Intent:** - Maintain the original message 's purpose, tone, and structure. - Ensure that the rephrased content conveys the same ideas and nuances as the original. - Avoid introducing new interpretations or altering the message 's context

  43. [51]

    **Generalize Specific Terms:** - Replace specific technical terms, brand names, or proprietary features with neutral alternatives where necessary. - If the input includes code, formulas, or game mechanics, rephrase them in a way that retains their function without exposing sen...

  44. [52]

    **Maintain Clarity and Professionalism:** - Ensure the output is well-structured, clear, and easy to understand. - Use formal or semi-formal language depending on the context (e.g., marketing materials, internal reports, social media).,→ - Avoid ambiguity or misinterpretation ...

  45. [53]

    Names, locations, and specific dates were removed to protect privacy. Cultural and historical references were generalized to maintain the emotional tone while ensuring neutrality

    **Include a Note on Changes:** - Add a concise note at the end of the response explaining the changes made for privacy, clarity, or generalization. - Clearly state what was removed or altered and why. - Ensure the note is clear, concise, and directly tied to the changes made. ...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.