ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

Andreas Krause; Barna P\'asztor; Davit Melikidze; Ido Hakimi; Jessica Lam; Marian Schneider; Martin Wertich

REVIEW 2 major objections 5 minor 24 references

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning

T0 review · 2 major / 5 minor · reviewed 2026-07-15 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Active pair selection that targets quality gaps can match or beat static preference data with one-sixth the annotations.

desk verdict Solid modular active-preference pipeline with two new delta-oriented selectors that beat static and classic bandit baselines under a fixed synthetic judge; the 6× annotation claim is real in that setting but untested with humans. read the letter →

arxiv 2603.09692 v2 pith:WPPVX4YI submitted 2026-03-10 cs.LG cs.AIcs.CL

Davit Melikidze , Marian Schneider , Jessica Lam , Martin Wertich , Ido Hakimi , Barna Pásztor , Andreas Krause This is my paper · ORCID

classification cs.LGcs.AIcs.CL

keywords activelearningpreferencedataRLHFduelingbanditsDeltaDRTSUCBsampleefficiency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Preference data for aligning language models is expensive, especially when experts must label it. This paper argues that you do not need to label every generated response pair: if you keep a running probabilistic model of response quality and deliberately pick the pairs that look most informative, you can build better datasets with far fewer labels. The authors package this idea as ActiveUltraFeedback, a modular loop that generates many candidate answers, estimates rewards and uncertainty, chooses a pair, obtains a preference label, and updates the reward model. Two new selection rules, Double Reverse Thompson Sampling and DeltaUCB, specifically chase large predicted quality gaps rather than only minimizing regret or uncertainty. Across several prompt collections and preference-tuning algorithms, those rules produce datasets that match or beat static baselines (including the original UltraFeedback pairs) while using as little as one-sixth of the annotations and only one comparison per prompt.

What carries the argument

Active delta learning via DRTS and DeltaUCB: selection rules that use ensemble-based upper and lower confidence bounds on reward to deliberately pair high- and low-quality responses rather than merely minimize dueling-bandit regret.

What would settle it

Collect a modest human preference set on the same actively selected pairs and re-run the reward-model and DPO/IPO/SimPO evaluations; if human labels erase the sample-efficiency advantage over random or UltraFeedback-style pairs, the central efficiency claim fails under the intended annotation regime.

Watch

Extended reading notes

Core claim

ActiveUltraFeedback with the new DRTS and DeltaUCB selectors produces preference datasets that yield comparable or superior reward-model and downstream fine-tuning performance using as little as one-sixth of the annotated pairs relative to static baselines, while needing only a single pairwise comparison per prompt and remaining usable across prompt sources and optimization algorithms.

Load-bearing premise

The pipeline treats an LLM judge’s aspect-wise scores as a faithful enough stand-in for the human preference signal that the method is meant to serve; if that proxy is systematically biased on the hard pairs that active selection chooses, the reported annotation savings may not transfer to real human budgets.

Editorial extensions

If this is right

When annotation budget is the bottleneck, DRTS or DeltaUCB become strong default pair selectors for preference data collection.
Selecting the single best response is not the same as selecting the most informative preference pair; large quality gaps give a cleaner training signal than near-ties between strong answers.
The same actively collected datasets can support both reward modeling and multiple direct preference optimizers without being locked to one model family.
Domains that lack large static preference corpora become more feasible once only a fraction of the usual labels are required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If response-generation cost later becomes the bottleneck, the same delta objective could be pushed upstream: choose which models to query rather than which already-generated answers to label.
Judge–human disagreement is likely largest precisely on the high-delta pairs that the new methods prefer, so calibration studies on those pairs would be the highest-leverage next measurement.
The modular loop invites testing whether simpler uncertainty estimators or smaller response pools can preserve most of the efficiency gain at lower compute.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful takeaway is simple: if you care about preference-data quality under a fixed prompt pool and a strong LLM judge, DRTS and DeltaUCB give clearer chosen–rejected gaps than random, UltraFeedback-style, or classic dueling-bandit selectors, and the resulting DPO/IPO/SimPO and reward-model curves look better at 5–10k pairs than the static baselines do at ~60k. That is the paper’s actual contribution.

What is new is not “active RLHF” in the abstract—DTS, InfoMax, MaxMinLCB, and ENN-based selection already exist—but a single modular loop that runs the same selectors on both reward modeling (RewardBench 2) and three preference optimizers, plus two deliberately delta-oriented rules (DRTS: Thompson best vs reversed-Thompson worst; DeltaUCB: max optimistic preference probability). The diagnostics in the appendix are honest: regret-minimizing methods correctly find strong answers but pair them with other strong answers, so the supervision signal is weak; the delta methods keep high chosen scores and low rejected scores. Multi-prompt-source ablations (UltraFeedback, Skywork, Combined, Tulu-3) and open code/data make the comparison reproducible. Stability numbers are reported and look tight enough for the claims they make.

Soft spots, in proportion. The central efficiency claim is measured entirely under Qwen3-235B aspect-wise probabilistic scoring. The methods are designed to surface large-gap pairs; if that judge’s ranking of those pairs diverges from humans, the 1/6 figure need not transfer to real annotation budgets. Appendix E gives proxy agreement numbers, not agreement on the actively selected pairs that drive Figures 3–5. Second, they pre-compute the full response pool and all judge scores for ablations, so the reported gains are annotation efficiency, not end-to-end compute; they say this themselves. Third, classic bandit objectives are shown to be misaligned with preference-data quality—that is a useful negative result, not a flaw. Hyperparameters (β, ensemble size, ζ decay, etc.) are free but swept and documented.

This is for people who build preference datasets or run active RLHF loops. It is not a theory paper. I would send it to peer review: the experimental design is careful, the negative result on regret methods is clean, and the open artifacts let others stress-test the judge-transfer question. Engage with it if you collect preference data; treat the human-label transfer as the open item, not as a reason to ignore the rest.

Referee Report

2 major / 5 minor

Summary. The paper introduces ACTIVEULTRAFEEDBACK, a modular active-learning pipeline for preference-data generation framed as a contextual dueling bandit. For each prompt, responses are drawn from a diverse 30-model pool, an Epistemic Neural Network (ENN) supplies reward means and uncertainties, and a pair is selected for annotation; the ENN is then updated. The authors systematically compare passive heuristics (RANDOM, MAXMIN, UltraFeedback-style, DeltaQwen), standard dueling-bandit rules (InfoMax, DTS, MaxMinLCB), and two novel delta-oriented rules (DRTS, DeltaUCB) that target large predicted quality gaps. Across UltraFeedback, Skywork, Combined, and Tulu-3 prompts, and under DPO, IPO, and SimPO, DRTS and DeltaUCB produce datasets that match or beat static baselines on GSM8K, IFEval, TruthfulQA, AlpacaEval 2, and RewardBench 2, often with roughly one-sixth the annotated pairs and only one comparison per prompt. Code and datasets are released.

Significance. If the sample-efficiency claims hold under realistic annotation, the work would materially lower the cost of preference data for RLHF, especially outside the training domains of fixed model families. Strengths include a clean modular design, multi-prompt and multi-optimizer ablations, seed/stability analyses (Appendix F.2), dataset diagnostics that explain why regret-minimizing rules underperform (Table 23, Figure 6), and full open-sourcing of pipeline, datasets, and models. The introduction of DRTS and DeltaUCB as practical, delta-aware acquisition functions is a concrete contribution that other labs can adopt immediately.

major comments (2)

The central sample-efficiency claim (comparable/superior performance with ~5k–10k vs 60k pairs; Abstract, §5.3, Figures 3–5) is established entirely under Qwen3-235B-A22B aspect-wise probabilistic scoring (§4.4, Appendix E). DRTS/DeltaUCB deliberately select large-gap pairs (Table 23: rejected means ~1.97–2.11). Appendix E validates the judge only on AlpacaEval-style and RewardBench-2 proxies, not on the actively selected UltraFeedback-style pairs that drive the efficiency curves. Without a human-annotation pilot or judge–human agreement study on those high-delta pairs, transfer of the reported 1/6 annotation savings to real human budgets remains an untested assumption that is load-bearing for the abstract’s claim.
§5.2–5.3 and Table 2 show that DeltaQwen is competitive for DPO downstream scores yet collapses on RewardBench 2 and on IPO/SimPO (§5.5, Figure 5). The paper attributes this to limited diversity, which is plausible, but the practical recommendation (“use DRTS or DeltaUCB”) would be stronger if the authors quantified how much of the DPO gain of DeltaQwen is AlpacaEval-driven (they already note this in Appendix G.2) and stated more clearly when a static size-gap heuristic remains preferable. This is not a correctness error, but it affects how readers should interpret the “best method” claim.

minor comments (5)

Figure 1 and the abstract claim “one-sixth of the annotated data”; the precise comparison (5k–10k vs 60k on UltraFeedback prompts under DPO) should be stated once in the main text with the exact baseline so readers do not have to reverse-engineer it from the figures.
Notation for UCB/LCB probabilities (Eqs. 3–4) re-uses pϕ for both; a short clarifying sentence or distinct symbols would reduce ambiguity when reading Algorithms 1–5.
Table 2 and Figure 3 report mean deltas; adding standard errors from the seed studies already present in Appendix F.2 to the main tables would make significance claims easier to assess at a glance.
The compute discussion (§5.3, F.4) correctly notes that pre-computing all responses inflates upfront cost; a short forward-looking paragraph on model-pool pruning (already partially supported by Table 26) would help practitioners estimate end-to-end cost.
Minor typographical inconsistencies appear in author affiliations and in the repeated “ACTIVEULTRAFEEDBACK” capitalization; a final pass would improve polish.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical active-learning pipeline evaluated on held-out external benchmarks, not algebraically forced by its own acquisition objective or fits.

full rationale

ACTIVEULTRAFEEDBACK is a modular systems paper. Response-pair selection (RANDOM, UltraFeedback-style, DTS, MaxMinLCB, and the proposed DRTS/DELTAUCB) uses ENN reward means and UCB/LCB or Thompson samples (Eqs. 3–4, Algorithms 1–5) that are updated from Bradley–Terry NLL plus standard centering/anchor regularizers (Eq. 5). The central sample-efficiency claim is established by training separate reward models and DPO/IPO/SimPO policies on the resulting preference datasets (or subsets thereof) and measuring performance on external held-out suites (GSM8K, IFEval, TruthfulQA, AlpacaEval 2, RewardBench 2) that are not the acquisition objective. No equation equates a reported downstream delta to a fitted parameter by construction; no uniqueness theorem or ansatz is imported from overlapping authors to force the result; the Delta Learning Hypothesis citation (Geng et al., 2025) is external inspiration, not a self-citation chain. Judge-proxy and transfer concerns are validity issues, not circularity. The derivation chain is therefore self-contained and non-circular.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The paper’s claims rest on standard RLHF modeling choices (Bradley–Terry, DPO/IPO/SimPO), an ensemble ENN uncertainty model with several tunable scalars, and the operational premise that an LLM judge can stand in for human preference labels at experimental scale. No new physical entities are postulated; free parameters are ordinary ML hyperparameters whose values are swept and reported.

free parameters (5)

ENN uncertainty scale β
Multiplies ensemble standard deviation to form reward UCB/LCB; swept in {1,2} and chosen per acquisition method and task (Table 18).
ENN ensemble size K and MLP head width
Fixed at K=20 heads, 2-layer MLP width 128 (Table 16); controls uncertainty quality and is not derived from first principles.
ENN regularization ζ decay and replay factor ρ
Anchor-to-init strength and replay-buffer size relative to batch; swept and set differently for DPO vs reward modeling (Tables 17–18).
Preference-optimization hyperparameters (λ, learning rate, epochs, SimPO γ)
Chosen by grid search on UltraFeedback for each of DPO/IPO/SimPO (Table 19); downstream deltas depend on these fits.
Active-learning batch size b=64
Fixed iteration size for the acquisition loop; affects how often the ENN is updated relative to data volume.

assumptions (5)

domain assumption Bradley–Terry model: preference probability is a sigmoid of latent reward difference.
Used for ENN training loss and for defining UCB/LCB preference probabilities (Eqs. 1, 3–4).
domain assumption Epistemic uncertainty of an ensemble of shallow heads on a frozen LLM backbone is a useful signal for selecting informative preference pairs.
Inherited from prior ENN-RLHF work; underpins all active selectors (Section 4.2).
domain assumption Large predicted quality gaps between chosen and rejected responses provide stronger fine-tuning signal than near-ties (Delta Learning Hypothesis).
Motivates DRTS and DeltaUCB (Section 4.3); supported empirically here but not proved.
ad hoc to paper LLM-as-judge aspect scores are adequate labels for comparing acquisition functions at scale.
Explicit experimental choice (Section 4.4, Appendix E); human labels are not used for the main tables.
standard math Standard math of Thompson sampling / UCB over bounded reward intervals.
Algorithms 1–5 implement known or minor variants of dueling-bandit acquisition rules.

invented entities (3)

Double Reverse Thompson Sampling (DRTS)
purpose: Acquisition rule that samples one high and one low reward draw to force large quality gaps while retaining exploration.
New algorithmic object defined in Algorithm 4; evaluated only inside this pipeline.
DeltaUCB
purpose: Deterministic acquisition rule maximizing optimistic preference probability between two distinct responses.
New algorithmic object defined in Algorithm 5; no external validation outside the paper’s experiments.
ACTIVEULTRAFEEDBACK pipeline independent evidence
purpose: Modular loop coupling multi-model response generation, ENN reward prediction, pair selection, LLM judging, and reward-model updates.
Systems contribution; existence is the open-sourced codebase rather than a physical entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning." pith.science (2026). https://pith.science/paper/WPPVX4YI

@misc{pith2026260309692,
  author       = {Pith},
  title        = {Pith review of: ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPPVX4YI}},
  note         = {Machine review of arXiv:2603.09692}
}

read the original abstract

Reinforcement Learning from Human Feedback (RLHF) has become the standard for aligning Large Language Models (LLMs), yet its efficacy is bottlenecked by the high cost of acquiring preference data, especially in low-resource and expert domains. To address this, we introduce ACTIVEULTRAFEEDBACK, a modular active learning pipeline that leverages uncertainty estimates to dynamically identify the most informative responses for annotation. Our pipeline facilitates the systematic evaluation of standard response selection methods alongside DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB, two novel methods prioritizing response pairs with large predicted quality gaps, leveraging recent results showing that such pairs provide good signals for fine-tuning. Our experiments demonstrate that ACTIVEULTRAFEEDBACK yields high-quality datasets that lead to significant improvements in downstream performance, notably achieving comparable or superior results with as little as one-sixth of the annotated data relative to static baselines. Our pipeline is available at https://github.com/lasgroup/ActiveUltraFeedback and our preference datasets at https://huggingface.co/ActiveUltraFeedback.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 8 linked inside Pith

[1]

cc/paper_files/paper/2017/file/ d5e2c0adad503c91f91df240d0cd4e49- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ d5e2c0adad503c91f91df240d0cd4e49- Paper.pdf. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Cohere, T., Aakanksha, Ahmadian,...

arXiv doi:10.52202/079017-4025 2017
[2]

Ethayarajh, K., Choi, Y ., and Swayamdipta, S

URL https://proceedings.mlr.press/ v235/dwaracherla24a.html. Ethayarajh, K., Choi, Y ., and Swayamdipta, S. Un- derstanding dataset difficulty with V-usable informa- tion. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.),Proceed- ings of the 39th International Conference on Machine Learning, volume 162 ofProceeding...
[3]

Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P

URL https://proceedings.mlr.press/ v162/ethayarajh22a.html. Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P. W. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. InSecond Conference on Language Modeling, 2025. URL https: //openreview.net/forum?id=9rwtezthwo. Grattafiori, A., Dubey, A., Jauhri, ...

arXiv doi:10.52202/079017-1154 2025
[4]

URL https://proceedings.mlr.press/ v235/muldrew24a.html. 12 ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning Olmo, T., :, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, ...

arXiv 2025
[5]

cc/paper_files/paper/2023/file/ 07fbde96bee50f4e09303fd4f877c2f3- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ 07fbde96bee50f4e09303fd4f877c2f3- Paper-Conference.pdf. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., a...

2023
[6]

cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731- Paper-Conference.pdf. P´asztor, B., Kassraie, P., and Krause, A. Bandits with preference feedback: A stackelberg game perspec- tive. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances in Neural Informa...

arXiv doi:10.52202/079017-0383 2022
[7]

cc/paper_files/paper/2023/file/ a85b405ed65c6477a4fe8302b5e06ce7- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ a85b405ed65c6477a4fe8302b5e06ce7- Paper-Conference.pdf. Saha, A. Optimal algorithms for stochastic contextual preference bandits. In Ranzato, M., Beygelzimer, A., Dauphin, Y ., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 30050–30062...

2023
[8]

cc/paper_files/paper/2021/file/ fc3cf452d3da8402bebb765225ce8c0e- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ fc3cf452d3da8402bebb765225ce8c0e- Paper.pdf. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shen, Y ., Sun, H., and Ton, J.-F. Reviving the classics: Ac- tive reward modeling in large languag...

arXiv 2021

Show all 24 references

[9]

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., V oss, C., Radford, A., Amodei, D., and Chris- tiano, P

URL https://openreview.net/forum? id=ev1xpo9mbI. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., V oss, C., Radford, A., Amodei, D., and Chris- tiano, P. F. Learning to summarize with human feedback. In Larochelle, H., Ranzato, M., Had- sell, R., Balcan, M., and Lin, ...
[10]

cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951- Paper.pdf. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini ...

2020 arXiv
[11]

Walsh, E

URL https://openreview.net/forum? id=aKkAwZB6JV. Walsh, E. P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y ., Huang, S., Jordan, M., Lam- bert, N., Schwenk, D., Tafjord, O., Anderson, T., Atkin- son, D., Brahman, F., Clark, C., Dasigi, P., Dziri, N., Ett...

2025 doi
[12]

findings-emnlp.620/

URL https://aclanthology.org/2024. findings-emnlp.620/. Wang, Z., Dong, Y ., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer 2: Open-source dataset for training top-performing reward models. In Globerson, A., Mackey, L....

2024 doi
[13]

cc/paper_files/paper/2016/file/ 9de6d14fff9806d4bcd1ef555be766cd- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2016/file/ 9de6d14fff9806d4bcd1ef555be766cd- Paper.pdf. Xu, Z., Jiang, F., Niu, L., Deng, Y ., Poovendran, R., Choi, Y ., and Lin, B. Y . Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing....

2016 arXiv
[14]

URL https://www.sciencedirect.com/ science/article/pii/S0306457325004893

doi: https://doi.org/10.1016/j.ipm.2025.104548. URL https://www.sciencedirect.com/ science/article/pii/S0306457325004893. Zhang, R., Zhang, C., Zhang, X., Qiu, L., Jiang, H., Zhuang, Y ., Zhang, Q., Yun, H., Li, X., Yin, B., Zhao, T., and Zhang, C. DORM: Preference data weight...

2025 doi
[15]

helpfulness

URL https://aclanthology.org/2025. findings-emnlp.1237/. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y ., Zhou, D., and Hou, L. Instruction-following evalu- ation for large language models, 2023. URL https: //arxiv.org/abs/2311.07911. Zhu, B., Frick, E., Wu, T., ...

2025 arXiv
[16]

Weakeners: e.g., ’I guess,’ ’probably.’
[17]

•No uncertainty expression indicate confidence

Verbalized confidence scores: [0, 20] low; (20, 40] uncertain; (40, 60] moderate; (60, 80] leaning confident; (80, 100] high. •No uncertainty expression indicate confidence. •Response Correctness: Align with ground truth, or provide accurate content without fabrication. Scorin...
[18]

Contradictory with the World (Factual Error): Entities, locations, concepts, or events that conflict with established knowledge
[19]

Contradictory with Instruction and Input: Responses diverge, introducing new facts not aligned with instructions or inputs
[20]

Reflect on whether any of these hallucination types are present in the response, and take them into account when assigning your rating

Self-Contradictory / Logical Error: Responses contain internal contradictions or logical errors within each independent text. Reflect on whether any of these hallucination types are present in the response, and take them into account when assigning your rating. Scoring: Rate o...
[21]

Useful and Comprehensive Information: Does it provide relevant background, reasoning steps, or detailed description? 3.Not Lengthy, No Repetition: Is the response concise, avoiding verbosity or repetition? Score on a scale of 1 to 5 based on extent of helpfulness, regarding bo...
[22]

Severely Incorrect: Contains significant inaccuracies or fabricated content, even if comprehensive information is provided
[23]

3.Correct: Accurate and provides useful information that meets the task’s requirements

Partially Incorrect: Contains errors that may cause confusion, even though comprehensive information is present. 3.Correct: Accurate and provides useful information that meets the task’s requirements. 4.Highly Informative: Accurate and extensive, providing valuable insights an...
[24]

Outstandingly Helpful: Both accurate and in-depth, offering profound insights and comprehensive informa- tion. Here are the instructions and the AI assistant response: <INSTRUCTIONS>{prompt}</INSTRUCTIONS> <TEXT>{response}</TEXT> 31 ActiveUltraFeedback: Efficient Preference Da...

2025

Pith tools

Reviewed July 15, 2026 · model on record in the stance chip above.

[1] [1]

cc/paper_files/paper/2017/file/ d5e2c0adad503c91f91df240d0cd4e49- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ d5e2c0adad503c91f91df240d0cd4e49- Paper.pdf. Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Cohere, T., Aakanksha, Ahmadian,...

arXiv doi:10.52202/079017-4025 2017

[2] [2]

Ethayarajh, K., Choi, Y ., and Swayamdipta, S

URL https://proceedings.mlr.press/ v235/dwaracherla24a.html. Ethayarajh, K., Choi, Y ., and Swayamdipta, S. Un- derstanding dataset difficulty with V-usable informa- tion. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.),Proceed- ings of the 39th International Conference on Machine Learning, volume 162 ofProceeding...

[3] [3]

Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P

URL https://proceedings.mlr.press/ v162/ethayarajh22a.html. Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., and Koh, P. W. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. InSecond Conference on Language Modeling, 2025. URL https: //openreview.net/forum?id=9rwtezthwo. Grattafiori, A., Dubey, A., Jauhri, ...

arXiv doi:10.52202/079017-1154 2025

[4] [4]

URL https://proceedings.mlr.press/ v235/muldrew24a.html. 12 ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning Olmo, T., :, Ettinger, A., Bertsch, A., Kuehl, B., Graham, D., Heineman, D., Groeneveld, D., Brahman, F., Timbers, F., Ivison, H., Morrison, J., Poznanski, J., Lo, K., Soldaini, L., Jordan, M., Chen, M., Noukhovitch, ...

arXiv 2025

[5] [5]

cc/paper_files/paper/2023/file/ 07fbde96bee50f4e09303fd4f877c2f3- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ 07fbde96bee50f4e09303fd4f877c2f3- Paper-Conference.pdf. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., a...

2023

[6] [6]

cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731- Paper-Conference.pdf. P´asztor, B., Kassraie, P., and Krause, A. Bandits with preference feedback: A stackelberg game perspec- tive. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.),Advances in Neural Informa...

arXiv doi:10.52202/079017-0383 2022

[7] [7]

cc/paper_files/paper/2023/file/ a85b405ed65c6477a4fe8302b5e06ce7- Paper-Conference.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2023/file/ a85b405ed65c6477a4fe8302b5e06ce7- Paper-Conference.pdf. Saha, A. Optimal algorithms for stochastic contextual preference bandits. In Ranzato, M., Beygelzimer, A., Dauphin, Y ., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 30050–30062...

2023

[8] [8]

cc/paper_files/paper/2021/file/ fc3cf452d3da8402bebb765225ce8c0e- Paper.pdf

URL https://proceedings.neurips. cc/paper_files/paper/2021/file/ fc3cf452d3da8402bebb765225ce8c0e- Paper.pdf. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shen, Y ., Sun, H., and Ton, J.-F. Reviving the classics: Ac- tive reward modeling in large languag...

arXiv 2021