DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space

Chenxi Zhou; Da Zhu; Guanjun Jiang; Haihan Shi; Hai Wan; Haiyan Wu; Hanqing Li; Jiangwang Chen; Junlin Liu; Shuaiyu Zhou

REVIEW 2 major objections 3 minor 18 references

This paper claims that when open-ended LLM tasks are optimized in text space, the evaluation rubric should be co-evolved with the solver skill under objectives decoupled from aggregate solver score, and that this yields the best official-ev

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:48 UTC pith:IWNAEJSP

load-bearing objection DecoEvo is a serious, well-controlled paper whose main empirical claim survives scrutiny; the unvalidated same-backbone audit preferences and absent code keep the mechanism story conditional. the 2 major comments →

arxiv 2607.25675 v1 pith:IWNAEJSP submitted 2026-07-28 cs.AI

DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space

Jiangwang Chen , Zixin Song , Junlin Liu , Shuaiyu Zhou , Haiyan Wu , Haihan Shi , Chenxi Zhou , Hanqing Li

show 5 more authors

Xiao Yang Da Zhu Guanjun Jiang Hai Wan Xibin Zhao

This is my paper

classification cs.AI

keywords text-space optimizationblack-box LLM adaptationrubric generationco-evolutionscore decouplingLLM-as-judgeskill optimizationopen-ended evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DecoEvo claims that text-space optimization of LLMs stalls when the evaluation rubric is fixed, because once the solver satisfies the measured criteria, remaining weaknesses fall in dimensions the rubric omits. Co-evolving the rubric along with the solver is dangerous if generator updates are rewarded by the solver's aggregate score, since the rubric can drift toward easy-to-satisfy criteria. The paper's solution is to decouple the two update loops: the solver skill improves from criterion-level rubric feedback, while the rubric-generator skill improves through two audits that do not use aggregate solver score—a structural audit of requirement coverage and a rubric-blind contrastive audit of near-tied responses. Under official gold-rubric evaluation, DecoEvo reports the highest mean score in all 15 backbone–benchmark pairs, with 2.8–5.0% relative gains over the strongest fixed-rubric comparator. A matched score-coupled baseline falls below that fixed-rubric comparator in 13 of 15 cases, supporting the claim that decoupling is what prevents rubric drift.

Core claim

On the paper's own terms, the central discovery is that score-coupling is the mechanism that corrupts solver–rubric co-evolution, and that a score-decoupled update can avoid it. DecoEvo defines a generator update as score-decoupled when neither its proposal nor its acceptance uses the solver's aggregate score under generated rubrics. Instead, the generator is revised through a task-conditioned structural audit, which checks whether the rubric covers the publicly stated task requirements, and a near-tie contrastive audit, which asks a rubric-blind judge to prefer one of two near-tied responses and then explains why the current rubric misses that distinction. Recurring audit findings are disti

What carries the argument

The load-bearing mechanism is the dual-view audit loop that selects generator-skill revisions without consulting aggregate solver score. The structural audit uses only the public task description and evaluation instructions as a task-level prior; it scores each generated rubric's adequacy, cites evidence of omissions, and proposes repair principles. The contrastive audit locates response pairs that the current rubric scores as near-tied, obtains a rubric-blind preference for one response over the other, and then diagnoses why the rubric fails to distinguish them. A rewriter distills these diagnoses into reusable rubric-generation principles, and a Pareto acceptance rule on held-out audit rec

Load-bearing premise

The method trusts the same frozen model's unaided preference about which of two near-tied answers is better, and no human validation of those audit preferences is reported; if that model's taste mirrors the solver's blind spots, the rubric can evolve toward the model's bias rather than toward true task quality.

What would settle it

Sample a set of near-tied answer pairs from the probe set, have the same backbone give rubric-blind preferences, and have independent human judges rate the same pairs; if human-model agreement is low, the discrimination audit is not tracking task quality. A cleaner causal test: run the full system with human labels substituted for the backbone's audit labels and check whether the official-gold gains persist.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

On open-ended tasks, a fixed evaluation rubric becomes a bottleneck after the solver has addressed the criteria it measures; co-evolving the rubric under decoupled objectives yields further official-score gains.
Selecting rubric updates by the solver's aggregate score is actively harmful: the paper's matched score-coupled baseline often falls below both the fixed-rubric baseline and zero-shot, even while its internal proxy stays high.
The optimized solver skill transfers within a domain to unseen benchmarks without any further updates, indicating that decoupling produces reusable strategies rather than question-specific patches.
The evolved rubric-generator skill shows higher criterion-level overlap with held-out gold rubrics, suggesting that the gains come from better evaluation coverage rather than a more permissive rubric.
Only the final solver skill is kept at deployment; the rubric generator and audit machinery are development-time costs, not inference-time overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial extension: a decisive test would replace the backbone's rubric-blind audit labels with independent human labels on the same near-tie pairs; if agreement is low, the discrimination audit may be tracking model-specific taste rather than task quality.
Editorial extension: the score-decoupling principle likely generalizes beyond text-space skills to any setting where a learned evaluator is co-adapted with a policy—for example, reward-model updates in weight-space RL should be selected by criteria other than the policy's current reward.
Editorial extension: one can construct a direct probe of the structural audit's value by initializing the generator with a deliberately incomplete rubric that omits a known safety-relevant dimension (such as contraindication warnings) and checking whether the audit recovers that dimension over evolution; the paper does not report such a targeted test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

DecoEvo is a serious, well-controlled paper whose main empirical claim survives scrutiny; the unvalidated same-backbone audit preferences and absent code keep the mechanism story conditional.

read the letter

Quick take: this is one of the more carefully evaluated text-space optimization papers I've seen recently. The core idea—decouple the rubric-generator's update from aggregate solver score, using task-conditioned structural audits and rubric-blind near-tie comparisons instead—is genuinely new relative to SkillOpt, OpenRS, and the weight-space co-evolution papers. The authors build a matched score-coupled baseline that does worse than SkillOpt, which makes the decoupling point concretely.

What's good: five benchmarks, three backbones, five matched seeds, paired CIs with Holm correction, ablations that remove each mechanism, and controls for extra compute, task priors, and direct audit routing. The cross-benchmark transfer results are a nice addition. Rubric-overlap analysis with a separately validated GPT-5.5 matcher supports the coverage mechanism. The paper is honest about its limits in the conclusion.

Soft spots, in proportion: no code or data is released, so the 15-setting superiority isn't reproducible from the paper alone. More importantly, the near-tie contrastive audit labels come from the same frozen backbone used everywhere else. There's no human validation of those labels—only the final GPT-5.5 rubric matcher gets that. If the backbone systematically prefers fluency over safety, or verbose over concise, the generator can co-adapt to that bias rather than to an external standard. The paper's ablation shows blindness matters, but not that the blind labels are correct. This doesn't sink the empirical result, because final evaluation uses official graders and the gains are real under that metric. It does weaken the mechanistic claim that score decoupling produces a genuinely better evaluation signal rather than a better proxy for the backbone's tastes.

Bottom line: worth a serious referee, and would cite if I worked in this area. For acceptance, I'd want code/data and at least a small human study on audit-label agreement.

Referee Report

2 major / 3 minor

Summary. DecoEvo proposes score-decoupled co-evolution of two text-space skills: a solver skill updated from criterion-level rubric feedback, and a rubric-generator skill updated via structural and near-tie contrastive audits whose acceptance rule (Eq. 10) excludes aggregate solver score. The paper reports experiments on five benchmarks and three LLM backbones, claiming the highest mean official-gold score in all 15 backbone–benchmark combinations, with 2.8–5.0% average relative gains over SkillOpt, supported by ablations, alternative-explanation controls, cost-matched comparison, and rubric-alignment analysis.

Significance. If the empirical claims hold, the paper provides a practical answer to a known failure mode in co-evolving evaluation artifacts: by selecting generator revisions on held-out audit objectives rather than solver score, the solver and rubric generator do not co-adapt to an easier proxy. The evaluation is unusually thorough: 5 benchmarks × 3 backbones, 5 matched seeds, paired confidence intervals with Holm correction, ablations, alternative-explanation controls, a cost-matched comparison, and final scoring by external official graders. The rubric-alignment analysis uses a human-validated matcher. These strengths make the headline result credible. The main weakness is that the contrastive audit labels—the sole acceptance signal for generator updates—are produced by the same frozen backbone and are not human-validated, leaving a correctness risk in the mechanism claim.

major comments (2)

[§3.3, Eq. (8)–(10)] The rubric-blind near-tie audit labels are produced by the same frozen backbone as the solver and judge (§4.1) and are the only acceptance signal for generator updates: Φctr in Eq. (8) is defined from these labels, and the acceptance rule (10) uses Φctr when the contrastive set is available. No human validation of audit labels is reported; only the GPT-5.5 rubric matcher in §4.7 is human-validated. If the backbone's blind preferences share the solver's blind spots (e.g., favoring fluency over safety), the generator can co-adapt to model-specific bias rather than to external task quality. The 'w/o Rubric Blindness' ablation shows blindness matters empirically but does not establish label correctness. The conclusion's limitation statement concedes the shared-backbone setting. Please provide human agreement on a sample of audit labels, or a cross-backbone auditor condition, or otherwise bou
[Reproducibility (all sections)] No code, data, prompts, or audit specifications are released. The pipeline involves multiple role-specific prompts, audit schedules, and thresholds (ε, δ, η, τ_tie, H, γ), so the empirical claims cannot be independently checked or re-run without these artifacts. Please release code and data, or at minimum the exact prompts, audit specifications, and hyperparameter values from the supplementary material.

minor comments (3)

[§4.2, Table 1] Paired 95% CIs are reported for backbone-level averages, but not for individual benchmark–backbone pairs. Since the headline says 'all 15 settings,' clarify how many per-benchmark differences are individually significant, or explicitly label the per-setting claim as descriptive.
[§3.3, Eq. (6)–(8)] The values of τ_tie, H, γ, δ, η, ε are said to be chosen by pilot search, but no sensitivity analysis or even the chosen values appear in the main text. Reporting these values and a small sensitivity sweep would help assess robustness.
[Definition 1 and §3.3] The term 'score-decoupled' is defined only with respect to aggregate solver score; response-level rubric scores still drive near-tie pair selection and the contrastive margin in Eq. (7). This should be stated explicitly to avoid the impression that rubric scores play no role in generator updates.

Circularity Check

0 steps flagged

No significant circularity; main results rest on held-out official gold-rubric evaluation and disjoint audit gates.

full rationale

Walking the derivation chain: solver candidates are accepted on a held-out solver gate under the current rubric (Eq. 4); generator candidates are proposed from probe audit records and accepted only if they improve the structural or contrastive audit objective on a disjoint generator gate (Eqs. 5, 8-10). The contrastive audit uses rubric-blind preferences from the same frozen backbone, so Φctr rewards making future rubrics reflect that backbone's preferences; this is an optimization objective, not a prediction whose target was fitted. The paper's headline claims are measured by each benchmark's official gold-rubric grader, which is never used for proposal or selection, so the main result is not forced by construction. Self-citations (Liu et al. 2026a,b,c) appear only in related-work surveys and are not load-bearing; no uniqueness theorem or ansatz is imported via self-citation. The shared-backbone setting is a real external-validity limitation, acknowledged in the Conclusion ('Our evidence is limited to settings where optimization roles share a backbone...'), but that is a risk about the audit signal's independence from model bias, not a reduction of a claimed result to its own inputs. No equation in the paper makes a reported improvement identical to an input by definition. Therefore no circular step is present.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method's load-bearing inputs are the pilot-selected optimization thresholds and the assumption that a single frozen LLM can provide unbiased audits of its own solver's outputs. These are not derived from first principles and are only partially validated.

free parameters (5)

Solver acceptance margin ε
Candidate solver skills are accepted only if validation proxy improves by more than ε (Eq. 4); value selected in pilot, not derived.
Generator acceptance margins δ and η
Generator candidate accepted if at least one audit objective improves by δ and none regresses beyond η (Eq. 10); pilot-selected.
Near-tie threshold τ_tie and max pairs H
Define which rollout pairs are considered near-tied and how many are retained for contrastive audit (Eq. 6); pilot-selected.
Contrastive resolution margin γ
Required rubric-induced margin for a resolved distinction in Φ_ctr (Eq. 8); pilot-selected.
Audit trigger frequency K and stall threshold M
Outer loop runs every K inner steps or after M consecutive rejections; pilot-selected schedules.

axioms (4)

domain assumption The same frozen backbone can act as solver, judge, auditors, and rewriters via role-specific prompts without introducing correlated bias
Section 3 states all roles use the same frozen backbone; if the auditor shares solver biases, the audit signal is not independent.
domain assumption Rubric-blind pairwise preferences of the LLM auditor approximate true task quality
Section 3.3 near-tie contrastive audit relies on y_qij labels; no human validation of these labels is reported.
domain assumption Public task descriptions and evaluation instructions provide sufficient priors for structural audits without gold rubrics
Section 3.3 structural audit uses π_tau_str built only from public instructions; leakage of gold criteria into these specs is not analyzed.
ad hoc to paper Rubric scores are linear weighted sums of criterion scores with positive weights for requirements
Section 3.1 Eq. 2; this convention is stated to be independent of gold scoring but constrains rubric structure.

pith-pipeline@v1.3.0-alltime-deepseek · 12524 in / 14437 out tokens · 144814 ms · 2026-08-01T01:48:39.879501+00:00 · methodology

0 comments

read the original abstract

Text-space optimization adapts large language models (LLMs) by editing external natural-language artifacts rather than model weights, so the optimized artifacts remain inspectable and the model can be treated as a black box. However, most existing text-space methods keep evaluation fixed. On open-ended tasks, this can become a bottleneck: once the solver improves on the criteria a rubric measures, omitted dimensions remain invisible to the optimization signal. Simply evolving the rubric is also unreliable when updates are selected by the current solver's score, because apparent progress can come from making the rubric easier to satisfy. We introduce DecoEvo (Decoupled Co-Evolution), which co-evolves a solver skill and a rubric-generator skill under decoupled objectives without using gold rubrics during optimization. The solver skill is updated using criterion-level feedback, while the rubric-generator skill is revised through complementary audits of requirement coverage and response discrimination that are independent of aggregate solver score. This separation focuses generator updates on newly exposed solver weaknesses, reducing repeated emphasis on criteria the solver already satisfies. Under each benchmark's official evaluation, DecoEvo outperforms all compared methods across five benchmarks and three LLM backbones, yielding 2.8--5.0\% relative gains over SkillOpt in the five-benchmark average.

discussion (0)

Reference graph

Works this paper leans on

18 extracted references · 3 canonical work pages · 1 internal anchor

[1]

Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775,

Pith/arXiv arXiv
[7]

General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks

Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Yehao Lin, Xinxuan Lv, Xuanlin Wang, Xiaoyu Li, Ziwen Wang, Xuezhi Cao, and Xunliang Cai. AMO-bench: Large language models still struggle in high school math competitions. InFindings of the Association for Computational Linguistics: ACL 2026, pp. 2120– 2137, San Diego, California, United States, July 2026a. As...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.11778 2026
[8]

GPT-4o system card.arXiv preprint arXiv:2410.21276,

OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276,

Pith/arXiv arXiv
[9]

URL https://doi.org/10.48550/arXiv.2410.21276

doi: 10.48550/arXiv.2410.21276. URL https://doi.org/10.48550/arXiv.2410.21276. OpenAI. Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, April

Pith/arXiv arXiv doi:10.48550/arxiv.2410.21276
[11]

Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang

URLhttps://arxiv.org/abs/2606.21262. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A comprehensive benchmark for generative writing. InAdvances in Neural Information Processing Systems (NeurIPS), vol- ume 38,

Pith/arXiv arXiv
[12]

Datasets and Benchmarks Track

URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 4aedf0cba303537fcb6cf948bb41b2df-Abstract-Datasets_and_Benchmarks_Track.html. Datasets and Benchmarks Track. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

Pith/arXiv arXiv 2025
[13]

URLhttps://doi.org/10.48550/arXiv.2505.09388

doi: 10.48550/ arXiv.2505.09388. URLhttps://doi.org/10.48550/arXiv.2505.09388. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. InThe Twelfth International Conference on Learning Repre- sentations (ICLR),

Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388
[14]

URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 3339f19c5fcee3ad74502947a32be9e6-Abstract-Conference.html. 12 DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong ...

Pith/arXiv arXiv 2024
[15]

URLhttps://doi.org/ 10.1162/TACL.a.732

doi: 10.1162/TACL.a.732. URLhttps://doi.org/ 10.1162/TACL.a.732. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative AI by backpropagating language model feedback.Nature, 639:609–616,

work page doi:10.1162/tacl.a.732
[16]

URLhttps://doi.org/10.1038/s41586-025-08661-4

doi: 10.1038/s41586-025-08661-4. URLhttps://doi.org/10.1038/s41586-025-08661-4. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. LLMEval-Med: A real-world clinical benchmark for medical LLMs with physicia...

doi:10.1038/s41586-025-08661-4 2025
[17]

doi: 10.18653/v1/2025.findings-emnlp.263

Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.263. URLhttps: //aclanthology.org/2025.findings-emnlp.263/. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language mod- els are scalable judges. InThe Thirteenth International Conference on Learning Representa- tions (ICLR),

doi:10.18653/v1/2025.findings-emnlp.263 2025
[18]

URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 7f8f73134e253845a8f82983219a8452-Abstract-Conference.html. 13

2025
[1960]

Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao, Zheng Li, Jinyang Zhang, Zhijing Wu, Junfeng Zhao, and Yasha Wang

doi: 10.1177/001316446002000104. Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao, Zheng Li, Jinyang Zhang, Zhijing Wu, Junfeng Zhao, and Yasha Wang. EvoRubrics: Dynamic rubrics as rewards via adversarial co-evolution for LLM reinforcement learning.arXiv preprint arXiv:2606.23038,

Pith/arXiv arXiv
[2012]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D

doi: 10.1007/978-3-540-92910-9_31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 53728–53741. Curran Associates, Inc.,

work page doi:10.1007/978-3-540-92910-9_31
[2023]

Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, and Jiuxin Cao

URL https://proceedings.mlr.press/v202/ gao23h.html. Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, and Jiuxin Cao. EvoRubric: Self-evolving rubric-driven RL for open-ended generation.arXiv preprint arXiv:2605.29847,

Pith/arXiv arXiv
[2024]

Shuyue Stella Li, Rui Xin, Teng Xiao, Yike Wang, Rulin Shao, Zoey Hao, Melanie Sclar, Sewoong Oh, Faeze Brahman, Pang Wei Koh, and Yulia Tsvetkov

URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 803485352e61e3ebf41221e4776c9fd4-Abstract-Conference.html. Shuyue Stella Li, Rui Xin, Teng Xiao, Yike Wang, Rulin Shao, Zoey Hao, Melanie Sclar, Sewoong Oh, Faeze Brahman, Pang Wei Koh, and Yulia Tsvetkov. EvoLM: Self-evolving language models through co-evolved discriminative rubrics.arXiv prepr...

Pith/arXiv arXiv 2024
[2025]

URLhttps://doi.org/10.48550/arXiv

doi: 10.48550/arXiv.2505.08775. URLhttps://doi.org/10.48550/arXiv. 2505.08775. Jacob Cohen. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1): 37–46,

Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775
[2026]

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T

URLhttps://arxiv.org/abs/2602.14069. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In Proceedings of the T...

arXiv

[1] [1]

Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775,

Pith/arXiv arXiv

[2] [7]

General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks

Junlin Liu, Shengnan An, Shuang Zhou, Dan Ma, Yehao Lin, Xinxuan Lv, Xuanlin Wang, Xiaoyu Li, Ziwen Wang, Xuezhi Cao, and Xunliang Cai. AMO-bench: Large language models still struggle in high school math competitions. InFindings of the Association for Computational Linguistics: ACL 2026, pp. 2120– 2137, San Diego, California, United States, July 2026a. As...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.11778 2026

[3] [8]

GPT-4o system card.arXiv preprint arXiv:2410.21276,

OpenAI. GPT-4o system card.arXiv preprint arXiv:2410.21276,

Pith/arXiv arXiv

[4] [9]

URL https://doi.org/10.48550/arXiv.2410.21276

doi: 10.48550/arXiv.2410.21276. URL https://doi.org/10.48550/arXiv.2410.21276. OpenAI. Introducing GPT-5.5.https://openai.com/index/introducing-gpt-5-5/, April

Pith/arXiv arXiv doi:10.48550/arxiv.2410.21276

[5] [11]

Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang

URLhttps://arxiv.org/abs/2606.21262. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A comprehensive benchmark for generative writing. InAdvances in Neural Information Processing Systems (NeurIPS), vol- ume 38,

Pith/arXiv arXiv

[6] [12]

Datasets and Benchmarks Track

URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 4aedf0cba303537fcb6cf948bb41b2df-Abstract-Datasets_and_Benchmarks_Track.html. Datasets and Benchmarks Track. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

Pith/arXiv arXiv 2025

[7] [13]

URLhttps://doi.org/10.48550/arXiv.2505.09388

doi: 10.48550/ arXiv.2505.09388. URLhttps://doi.org/10.48550/arXiv.2505.09388. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. InThe Twelfth International Conference on Learning Repre- sentations (ICLR),

Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388

[8] [14]

URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 3339f19c5fcee3ad74502947a32be9e6-Abstract-Conference.html. 12 DecoEvo: Score-Decoupled Co-Evolution of Solver and Rubric-Generator Skills in Text Space Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong ...

Pith/arXiv arXiv 2024

[9] [15]

URLhttps://doi.org/ 10.1162/TACL.a.732

doi: 10.1162/TACL.a.732. URLhttps://doi.org/ 10.1162/TACL.a.732. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative AI by backpropagating language model feedback.Nature, 639:609–616,

work page doi:10.1162/tacl.a.732

[10] [16]

URLhttps://doi.org/10.1038/s41586-025-08661-4

doi: 10.1038/s41586-025-08661-4. URLhttps://doi.org/10.1038/s41586-025-08661-4. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. LLMEval-Med: A real-world clinical benchmark for medical LLMs with physicia...

doi:10.1038/s41586-025-08661-4 2025

[11] [17]

doi: 10.18653/v1/2025.findings-emnlp.263

Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.263. URLhttps: //aclanthology.org/2025.findings-emnlp.263/. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language mod- els are scalable judges. InThe Thirteenth International Conference on Learning Representa- tions (ICLR),

doi:10.18653/v1/2025.findings-emnlp.263 2025

[12] [18]

URL https://proceedings.iclr.cc/paper_files/paper/2025/hash/ 7f8f73134e253845a8f82983219a8452-Abstract-Conference.html. 13

2025

[13] [1960]

Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao, Zheng Li, Jinyang Zhang, Zhijing Wu, Junfeng Zhao, and Yasha Wang

doi: 10.1177/001316446002000104. Hongxin Ding, Baixiang Huang, Yue Fang, Weibin Liao, Zheng Li, Jinyang Zhang, Zhijing Wu, Junfeng Zhao, and Yasha Wang. EvoRubrics: Dynamic rubrics as rewards via adversarial co-evolution for LLM reinforcement learning.arXiv preprint arXiv:2606.23038,

Pith/arXiv arXiv

[14] [2012]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D

doi: 10.1007/978-3-540-92910-9_31. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InAdvances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 53728–53741. Curran Associates, Inc.,

work page doi:10.1007/978-3-540-92910-9_31

[15] [2023]

Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, and Jiuxin Cao

URL https://proceedings.mlr.press/v202/ gao23h.html. Xin Guan, Xiaomeng Hu, Shen Huang, Zhenyi Wang, Bo Zhang, Zijian Li, Pengjun Xie, Bo Liu, and Jiuxin Cao. EvoRubric: Self-evolving rubric-driven RL for open-ended generation.arXiv preprint arXiv:2605.29847,

Pith/arXiv arXiv

[16] [2024]

Shuyue Stella Li, Rui Xin, Teng Xiao, Yike Wang, Rulin Shao, Zoey Hao, Melanie Sclar, Sewoong Oh, Faeze Brahman, Pang Wei Koh, and Yulia Tsvetkov

URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/ 803485352e61e3ebf41221e4776c9fd4-Abstract-Conference.html. Shuyue Stella Li, Rui Xin, Teng Xiao, Yike Wang, Rulin Shao, Zoey Hao, Melanie Sclar, Sewoong Oh, Faeze Brahman, Pang Wei Koh, and Yulia Tsvetkov. EvoLM: Self-evolving language models through co-evolved discriminative rubrics.arXiv prepr...

Pith/arXiv arXiv 2024

[17] [2025]

URLhttps://doi.org/10.48550/arXiv

doi: 10.48550/arXiv.2505.08775. URLhttps://doi.org/10.48550/arXiv. 2505.08775. Jacob Cohen. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1): 37–46,

Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775

[18] [2026]

Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T

URLhttps://arxiv.org/abs/2602.14069. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. In Proceedings of the T...

arXiv