Granular feedback merits sophisticated aggregation

Anmol Kagrecha; Benjamin Van Roy; Henrik Marklund; Potsawee Manakul; Richard Zeckhauser

REVIEW 2 major objections 4 minor 26 references

Granular feedback merits sophisticated aggregation

T0 review · 2 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes that as feedback becomes more granular, a learned aggregation method increasingly beats regularized averaging, needing about half the raters on 5-point scales.

desk verdict A careful single-dataset study showing sophisticated aggregation beats regularized averaging as feedback gets more granular, but the headline savings may not hold for new raters beyond the fixed 20/19 split. read the letter →

arxiv 2507.12041 v1 pith:IJ6UUA7I submitted 2025-07-16 cs.LG

Anmol Kagrecha , Henrik Marklund , Potsawee Manakul , Richard Zeckhauser , Benjamin Van Roy This is my paper · ORCID

classification cs.LG

keywords humanfeedbackaggregationgranularitydistributionpredictionregularizedaveragingsupervisedlearningordinallabelsRLHFcrowdsourcing

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human feedback is increasingly collected on multi-point scales, and predicting a population's feedback distribution from a small sample is a common task. The paper argues that how you aggregate that feedback matters more as the scale gets finer: a learned aggregator (supervised learning) gains a growing edge over regularized averaging as feedback becomes more granular. On their social-attitudes dataset, the learned method needs about 44% fewer raters on a 5-point scale and 54% fewer on an 11-point scale to match regularized averaging, while on binary feedback the two are nearly tied. The reason offered is that regularized averaging silently throws away the information inside each rating by binarizing it, and that binarization loss grows with granularity. If true, the result changes how feedback collection budgets should be spent, especially in RLHF and survey pipelines that already collect granular ratings.

What carries the argument

The load-bearing object is the binarized form of each rating. RegAvg's prediction of an output CDF element $O_t(y)$ is a convex combination of a prior and the average of indicators $\mathbf{1}[Y_{t,k} \le y]$, so it only ever sees whether each rater fell below or above the threshold $y$; all magnitude information inside the rating is discarded. Against this, the paper sets supervised learning, an MLP whose output is constrained to valid CDFs and trained to predict the output-set empirical CDF from input-set ratings. The argument that the gap must widen with granularity rests on two assumptions: (Assumption 1) the extra information in the full rating $Y_t$ about $O_t(y)$ relative to the binarized version $\mathbf{B}^{(y)}_t$ increases with granularity, and (Assumption 2) SL and a binarized-input counterpart SLb optimally use all relevant information. Claims 1 through 6 then derive that the advantage ratio $K/K'$ of SL over RegAvg increases with granularity. The ordinal CDF log-loss in Equation (2) is what makes the comparison meaningful for ordered scales.

What would settle it

Run the same SL versus RegAvg comparison on a second, larger population of several hundred raters rating the same task units and evaluate against the full population CDF rather than a 19-person output set; if SL no longer matches RegAvg with roughly half the raters on 5-point feedback, the granularity mechanism does not transfer. A more targeted check is to measure the mutual information between full ratings and binarized ratings about the output CDF and see whether it actually grows with scale.

Watch

Extended reading notes

Core claim

The paper's central claim is that as feedback granularity increases, the advantage of sophisticated aggregation policies over regularized averaging increases. Concretely, supervised learning (SL), which is trained to predict a held-out group's empirical CDF from a small input group's ratings, matches regularized averaging (RegAvg) using roughly 44% fewer individuals with 5-point feedback and 54% fewer with 11-point feedback, whereas with binary feedback the two methods perform almost identically. The paper also shows that in loss functions typical of RLHF reward-model training, SL requires about 29% fewer raters with 5-point feedback and 36% fewer with 11-point feedback. The reason, the paper argues, is that RegAvg's prediction of any CDF element $O_t(y)$ depends only on the binarized indicators $\mathbf{1}[Y_{t,k} \le y]$, while SL can exploit the full rating; as the scale becomes finer, the information lost by binarization grows, so a method that uses all available information should improve relative to one that does not.

Load-bearing premise

The load-bearing premise is that the 19 held-out raters' empirical distribution faithfully represents the infinite population's distribution, and that granular ratings carry increasingly more predictive information than their binarized versions about that distribution; if either fails, the measured 44%–54% reductions will not transfer to new raters or settings.

Editorial extensions

If this is right

With 5-point or finer feedback, switching from regularized averaging to a learned aggregator roughly halves the number of human raters needed to achieve the same distribution-prediction accuracy.
With binary feedback, simple regularized averaging is nearly as good as learned aggregation, so sophistication buys little on two-point scales.
RLHF pipelines that collect granular ratings and then binarize or average them could match current aggregated-feedback quality with roughly 20% fewer total scores by training a learned aggregator on the original ratings, after accounting for the extra labels needed to train it.
The advantage grows monotonically with granularity on the scales tested (2, 5, and 11 points), implying that spending on finer rating scales and on better aggregation are complements rather than substitutes.
Because training SL needs output-set labels, the net benefit in applications depends on how many task units are used for training and how many are served; the paper's RLHF example shows a net saving for 10,000 task units.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extension: the same binarization-information mechanism should apply to other ordinal-label domains, such as recommender ratings, so a learned aggregator trained on held-out user ratings should beat the simple item mean by a margin that grows with rating-scale granularity; this is directly testable on existing rating datasets.
Caveat to test: the paper itself notes the advantage may stop growing beyond very fine scales because raters may not use the full scale; a natural test is to measure SL's advantage on 21-point or 101-point versions of the same task units.
Practical extension: for AI training, comparing reward models trained on SL-aggregated labels versus RegAvg-aggregated labels on a downstream task would test whether the aggregation gain survives the full RLHF pipeline, not just the preference-signal loss.
Design implication: data collection for distribution prediction should include more overlapping raters per task unit and finer scales when a learned aggregator will be used, since the value of that overlap increases with granularity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper's core claim—SL's edge over RegAvg grows with feedback granularity—is plausible, well-presented, and backed by a newly collected dataset, but the headline headcount savings (44% at 5 points, 54% at 11) are measured entirely inside a fixed 20/19 split of 39 Thai crowdworkers. The experiment does not test whether the learned aggregator transfers to novel raters, so I read the savings as in-pool rather than out-of-pool.

What is actually new: the dataset itself, plus the specific granularity-dependent gap, which I have not seen in prior work. The paper does several things well. It releases the data and code, uses a sensible ordinal loss, gives bootstrap confidence intervals, and performs 5-fold cross-validation over task units. It also separates the empirical result from an informal theory, labeling assumptions explicitly and listing limitations in Section 7. That transparency earns credit.

The soft spots are real but proportionate. The single-dataset problem is acknowledged by the authors, but the deeper issue is the missing worker-level out-of-sample check. Because SL is trained to predict the 19 output workers from the 20 input workers, and because rater idiosyncrasies are stable within this pool and shared by construction, SL can exploit patterns that a newly hired rater would not share. The stress-test note is right: the 5-fold CV is over task units, not workers, so it cannot establish transfer to new raters. Second, Assumption 1 in Section 5.4 basically restates the central claim, but since it is flagged as an assumption rather than smuggled in, it weakens the theory without invalidating the empirical finding. Third, the 5-point and binary datasets are derived by binning the 11-point scores rather than collected at those scales; acceptable for a first pass, but it slightly confounds granularity with the mapping. The informal proof also leans on strong optimality assumptions (Assumption 2) that are unlikely to hold exactly with 1020 training units.

Who is this for? Practitioners in RLHF, crowdsourcing, or survey aggregation who want cheaper feedback, and methodologists interested in distribution prediction. It deserves a serious referee: the question is important, the data are new and public, and the pattern is worth checking. A referee should push for an out-of-pool rater experiment, or at least demand that the abstract's 'half as many individuals' be qualified as conditional on the rater pool. I recommend accepting it into peer review with that expectation.

Referee Report

2 major / 4 minor

Summary. This paper studies the problem of predicting a population's distribution of ordered feedback from a small number of respondents. The authors claim that the advantage of sophisticated aggregation (exemplified by a supervised MLP, SL) over regularized averaging (RegAvg) grows with the granularity of the rating scale. To support this, they introduce a new dataset of 1,020 social-attitude task units rated on an 11-point scale by 39 Thai crowdworkers, and they construct binary and 5-point variants by deterministic or probabilistic binning. The experiment splits the 39 workers into 20 input and 19 output raters, compares RegAvg with SL across K using 5-fold cross-validation over task units, and reports that SL matches RegAvg with about 44% and 54% fewer individuals on 5-point and 11-point scales, while the advantage on binary feedback is small. Section 5.4 gives an informal argument based on two assumptions, and Section 6 extends the analysis to RLHF-style preference signals.

Significance. The question is practically important: if the claim holds, it implies substantial savings in human-annotation budgets and provides a simple design rule (use granular scales only if you also upgrade the aggregation method). The paper's strengths include a new, openly described dataset; a clean separation of training and evaluation environments; a sensible ordinal loss function; and an unusually candid limitations section. The empirical comparison is an independent data point rather than an artifact of the formal framework. The barrier to full acceptance is that the headline quantitative claim is measured within one fixed worker pool of 39 people, and the theoretical argument for generalization rests on an assumption that is close to the conclusion. These issues are fixable and do not invalidate the empirical comparison as a case study.

major comments (2)

[Section 5.1 and 5.3] The central claim that sophistication reduces headcount is established only within a single fixed 20/19 split of the same 39 crowdworkers. Every eval-environment draws input from the same 20 input-set workers and evaluates against the same 19 output-set workers; the 5-fold cross-validation described in Section 5.2 is over task units, not over workers. Because rater idiosyncrasies such as harshness, central tendency, and extreme responding are stable within this pool and are shared by construction between the input and output sets, SL can exploit them while RegAvg cannot. The practical reading of the headline result, hiring about 44% fewer individuals, requires transfer to new raters, which the current experiment does not test. Please add a worker-level holdout analysis (for example, repeated random splits into input/output workers or leave-one-worker-out evaluation), or clearly restrict the claim to within the studied worker pool; the same limitation applies to the RLHF-style savings reported in Section 6.
[Section 5.4, Assumption 1] Assumption 1 states that the information advantage of the original feedback Yt over the binarized feedback B(y)_t about the output-set CDF element Ot(y) increases with feedback granularity. This is essentially the paper's main claim restated for a single CDF element; the informal justification gives plausibility but does not derive the assumption from more basic conditions. Since Claims 2 through 6 and the argument that the result extends beyond the current experimental setup all depend on this assumption, the theoretical section does not provide independent evidence for the central claim beyond the experiment itself. Please either prove Assumption 1 under explicit generative assumptions, or test it directly with information-theoretic estimates on the collected data, or label it as a conjecture and soften the wording that the paper 'shows' the monotonic relationship.

minor comments (4)

[Section 5.2] The prior Q0 is described as the average of the empirical CDFs of 'OOS individuals'; the acronym OOS is not defined and appears to refer to the output-set workers. Please define it or use a consistent name throughout.
[Section 5.1] The binary dataset construction assigns neutral scores randomly to {-1, +1}; the results for binary feedback may depend on this random assignment. Reporting the variance over random assignment seeds, or at least stating the seed used, would help interpret the near-tie between SL and RegAvg on the binary scale.
[Section 5.3] The headcount savings in Figure 4 are obtained by interpolating between mean losses at integer values of K. Please state explicitly the interpolation method (a uniform grid with linear interpolation appears to be used) and note that the reported savings inherit any approximation error from this interpolation.
[Section 5.4] There are several typos in this section, including 'a individual' and 'an a score'; similar typos appear in the appendix tables ('feeedback' in Tables 5 and 6). A careful proofread would improve readability.

Circularity Check

1 steps flagged · score 5.0 of 10

The empirical headcount comparison is independent, but the Section 5.4 generalization argument restates the main claim as Assumption 1 and then derives it back out; partial circularity.

self definitional [Section 5.4, Assumption 1 and Claim 2, built on Claim 1 and Eq. (3); cf. Section 1 main claim]
"Assumption 1. Fix a score y ∈ [−1, 1). Consider the difference in the information that one gets from Yt about Ot(y) and the information one gets from B(y)_t about Ot(y). As feedback granularity increases, this difference increases. ... Claim 2. With the assumptions above, for any K, the advantage of SL over RegAvg increases with feedback granularity."

The paper's main claim is 'as feedback granularity increases, the advantage of sophisticated methods increases relative to regularized averaging.' Claim 1 (which is definitionally true from Eq. (3)) establishes that RegAvg predicts Ot(y) from the binarized vector B(y)_t only. Assumption 2 adds that SL optimally uses all of Yt while SLb optimally uses B(y)_t. Given these bridges, Claim 2 — the advantage of SL over RegAvg grows with granularity — is the same proposition as Assumption 1, that the information gap between Yt and B(y)_t about Ot(y) grows with granularity. Claims 4–6 only translate this information gap into loss differences via Eq. (2). Thus the Section 5.4 derivation imports its conclusion as an assumption; no independent evidence is added by the formal chain.

full rationale

The experimental part is self-contained and not circular: SL is trained on DE,K with 5-fold cross-validation over task units, RegAvg's γ is tuned on separate train-environments, and the 44%/54% reductions are measured within the fixed 20/19 worker split. The Section 7 limitation (one dataset, no worker-level out-of-sample check) concerns external validity, not circularity, and is candidly acknowledged. There is no load-bearing self-citation chain; the only author-overlap citation (Friedman et al. 2018, co-authored by Zeckhauser) supports related-work context only. The one circular element is the informal generalization argument in Section 5.4: Assumption 1 restates the main claim under the paper's own definitional bridges (Claim 1 and Assumption 2), so Claims 2–6 derive back the assumption. Because the empirical results are independent evidence but the theoretical generality argument reduces to its premise, the overall circularity score is 5.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new particles, forces, or theoretical constructs are introduced. The output-set CDF is an evaluation target, not an invented entity. The main burden is carried by domain assumptions about representativeness of the 19-worker output set and by the informal information-theoretic assumptions in Section 5.4.

free parameters (3)

RegAvg regularization gamma = chosen per K from {0.1, 0.2, ..., 1.0} on training environments; individual values not tabulated
Controls the strength of the prior Q0 in RegAvg; RegAvg's loss curves, and therefore the measured advantage of SL, depend on this choice.
SL hyperparameters per granularity = 2 hidden layers; sizes 25/25/50; batch 25/50/50; learning rate 0.01/0.01/0.005; weight decay 0.001/0.001/0.0005…
Chosen by stepwise tuning on training environments; SL's performance advantage is partly a result of these choices.
neutral-score random assignment seed for binary dataset = not reported
The binary dataset randomly assigns neutral 0 scores to -1 or +1; the authors do not state the seed or average over repetitions, so this stochastic choice can affect binary results.

assumptions (5)

domain assumption Exchangeability of the feedback sequence (Section 3)
Allows the population CDF Pt to be defined via de Finetti-style limits; assumed for all task units but not tested.
domain assumption The 19-person output-set empirical CDF approximates the infinite population CDF (Section 5.1)
The evaluation treats Ot as a faithful proxy for Pt; noise and representativeness of the fixed 19-worker split are not quantified in the claim.
ad hoc to paper Assumption 1 in Section 5.4: granular feedback carries increasingly more information than binarized feedback about Ot(y)
This is the core premise of the informal argument; it effectively asserts the conclusion in information-theoretic form and is not derived from data or first principles.
ad hoc to paper Assumption 2 in Section 5.4: SL and SLb optimally use all relevant information
Only true asymptotically with large data and large neural networks; in the finite experiments it is an idealization.
domain assumption Reward models trained on aggregated feedback of similar quality generalize similarly (Section 6)
Used to extrapolate from aggregation quality to end-to-end RLHF pipeline quality; the paper explicitly labels this as an assumption made for simplicity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Granular feedback merits sophisticated aggregation." pith.science (2026). https://pith.science/paper/IJ6UUA7I

@misc{pith2026250712041,
  author       = {Pith},
  title        = {Pith review of: Granular feedback merits sophisticated aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJ6UUA7I}},
  note         = {Machine review of arXiv:2507.12041}
}

read the original abstract

Human feedback is increasingly used across diverse applications like training AI models, developing recommender systems, and measuring public opinion -- with granular feedback often being preferred over binary feedback for its greater informativeness. While it is easy to accurately estimate a population's distribution of feedback given feedback from a large number of individuals, cost constraints typically necessitate using smaller groups. A simple method to approximate the population distribution is regularized averaging: compute the empirical distribution and regularize it toward a prior. Can we do better? As we will discuss, the answer to this question depends on feedback granularity. Suppose one wants to predict a population's distribution of feedback using feedback from a limited number of individuals. We show that, as feedback granularity increases, one can substantially improve upon predictions of regularized averaging by combining individuals' feedback in ways more sophisticated than regularized averaging. Our empirical analysis using questions on social attitudes confirms this pattern. In particular, with binary feedback, sophistication barely reduces the number of individuals required to attain a fixed level of performance. By contrast, with five-point feedback, sophisticated methods match the performance of regularized averaging with about half as many individuals.

Figures

Figures reproduced from arXiv: 2507.12041 by the authors.

**Figure 2.** An example of a task unit that we gave to the crowdworkers. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Losses of policies as a function of number of individuals for feedback of different granu [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Number of individuals needed to match RegAvg’s performance. Note here that a smaller [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Performance comparison for predicting choices of output-set individuals using loss func [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Examples of a task unit that we gave to the crowdworkers. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]

**Figure 7.** Figure 7: Prompt and system prompt for rating topics. The topic list is scraped from the website [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

**Figure 8.** Figure 8: Prompt and system prompt for generating task units. The idea is to generate task units [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]

**Figure 9.** Figure 9: Instructions for the crowdworkers. understanding how ‘noisy’ the crowdworkers were. If a crowdworker’s feedback changes a lot for the same task unit, then it is intuitive to call this crowdworker more noisy. While we did find some crowdworkers noisy, we did not exclude…

**Figure 10.** Figure 10: Losses of methods as a function of number of individuals for feedback of different [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]

**Figure 11.** Figure 11: Losses of methods as a function of number of individuals for feedback of different [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages

[1]

L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

arXiv
[9]

The crowdworkers were compensated for this task at a rate more than three times the minimum wage in Thailand

The task units were all translated into Thai for the crowdworkers. The crowdworkers were compensated for this task at a rate more than three times the minimum wage in Thailand. The crowdworkers were informed that their responses would be used for research purposes. The data collection was reviewed and approved by the management, and we followed internal e...

work page 2025
[10]

International Social Survey Programme

Accessed: 2025-06-06. International Social Survey Programme. International social survey programme (issp). https: //issp.org,

work page 2025
[11]

Ipeirotis, P

Accessed: 2025-06-06. Ipeirotis, P. AdultContent3 Dataset,

work page 2025
[14]

GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling

URL https://mattlease.com/data/ trec-rf10-crowd.tgz. Crowdsourced relevance judgments for the TREC 2010 Relevance Feed- back Track. Li, J., Ling, S., Wang, J., Li, Z., and Callet, P. L. GPM: A generic probabilistic model to recover annotator’s behavior and ground truth labeling. arXiv preprint arXiv:2003.00475 ,

work page Pith review arXiv 2010
[16]

Accessed: 2025-03-03

URL https://github.com/glgh/ awesome-llm-human-preference-datasets . Accessed: 2025-03-03. Liu, T., Venkatachalam, A., Sanjay Bongale, P., and Homan, C. Learning to predict population- level label distributions. In Companion Proceedings of The 2019 World Wide Web Conference , pages 1111–1120,

work page 2025
[18]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al

Accessed: 2025-06-06. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744,

work page 2025
[19]

PEW Research Center

Accessed: 2025-06-06. PEW Research Center. Research topics,

work page 2025

Show all 26 references

[20]

Accessed: 2025-03-07

URL https://www.pewresearch.org/topics/. Accessed: 2025-03-07. Ricci, F., Rokach, L., and Shapira, B. Introduction to recommender systems handbook. In Recom- mender systems handbook , pages 1–35. Springer,

2025
[21]

Llama 2: Open foundation and fine-tuned chat models

21 Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

arXiv
[22]

Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al

Accessed: 2025-06-06. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 ,

2025 arXiv
[23]

thumbs up

Yu, H., Aronson, R. M., Allen, K. H., and Short, E. S. From “thumbs up” to “10 out of 10”: Recon- sidering scalar feedback in interactive reinforcement learning. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4121–4128. IEEE,

2023
[24]

and some datasets (Lease, 2010; Zhou et al.,

2010
[1975]

Deepseek-v3 technical report

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 ,

arXiv
[2010]

Accessed: 2025-03-06

URL https://github.com/ipeirotis/ Get-Another-Label/tree/master/data/AdultContent3-HCOMP2010 . Accessed: 2025-03-06. Kaufmann, T., Weng, P., Bengs, V., and H¨ ullermeier, E. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925 , 10,

2025
[2011]

L., Zhou, K., Patel, K., Hashimoto, T., and Bernstein, M

Gordon, M. L., Zhou, K., Patel, K., Hashimoto, T., and Bernstein, M. S. The disagreement deconvolution: Bringing machine learning performance metrics in line with reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1–14,

2021
[2012]

Specifically, each task unit has fewer than six individuals giving feedback

used in crowdsourcing literature have too few individuals giving feedback for each question. Specifically, each task unit has fewer than six individuals giving feedback. Other datasets used in the crowdsourcing literature, like those by Ipeirotis (2010) and Halpern et al. (201...

2010
[2015]

Webgpt: Browser-assisted question-answering with human feedback

20 Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

arXiv
[2016]

Halpern, M., Zhu, Y., and Reddi, V. J. Mobile CPU’s rise to power: Quantifying the impact of generational mobile CPU design trends on performance, energy, and user satisfaction. In 2016 IEEE International Symposium on High Performance Computer Architecture (HPCA) , pages 64–76. IEEE,

2016
[2018]

I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

Gemini, T., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 ,

arXiv
[2020]

K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al

Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217 ,

arXiv
[2021]

L., Lam, M

Gordon, M. L., Lam, M. S., Park, J. S., Patel, K., Hancock, J., Hashimoto, T., and Bernstein, M. S. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , pages 1–19,

2022
[2022]

The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

arXiv
[2023]

and Teh, Y

Lakshminarayanan, B. and Teh, Y. W. Inferring ground truth from multi-annotator ordinal data: a probabilistic approach. arXiv preprint arXiv:1305.0015 ,

arXiv
[2024]

Stable benchmark dataset

URL https://grouplens.org/datasets/ movielens/32m/. Stable benchmark dataset. 32 million ratings and two million tag applications applied to 87,585 movies by 200,948 users. Collected 10/2023; Released 05/2024. Guo, X. and Kwok, J. T. Aggregating crowdsourced ordinal labels via...

2023
[2025]

Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al

Accessed: 2025-06-06. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 ,

2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Al- tenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

arXiv

[2] [9]

The crowdworkers were compensated for this task at a rate more than three times the minimum wage in Thailand

The task units were all translated into Thai for the crowdworkers. The crowdworkers were compensated for this task at a rate more than three times the minimum wage in Thailand. The crowdworkers were informed that their responses would be used for research purposes. The data collection was reviewed and approved by the management, and we followed internal e...

work page 2025

[3] [10]

International Social Survey Programme

Accessed: 2025-06-06. International Social Survey Programme. International social survey programme (issp). https: //issp.org,

work page 2025

[4] [11]

Ipeirotis, P

Accessed: 2025-06-06. Ipeirotis, P. AdultContent3 Dataset,

work page 2025

[5] [14]

GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling

URL https://mattlease.com/data/ trec-rf10-crowd.tgz. Crowdsourced relevance judgments for the TREC 2010 Relevance Feed- back Track. Li, J., Ling, S., Wang, J., Li, Z., and Callet, P. L. GPM: A generic probabilistic model to recover annotator’s behavior and ground truth labeling. arXiv preprint arXiv:2003.00475 ,

work page Pith review arXiv 2010

[6] [16]

Accessed: 2025-03-03

URL https://github.com/glgh/ awesome-llm-human-preference-datasets . Accessed: 2025-03-03. Liu, T., Venkatachalam, A., Sanjay Bongale, P., and Homan, C. Learning to predict population- level label distributions. In Companion Proceedings of The 2019 World Wide Web Conference , pages 1111–1120,

work page 2025

[7] [18]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al

Accessed: 2025-06-06. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730–27744,

work page 2025

[8] [19]

PEW Research Center

Accessed: 2025-06-06. PEW Research Center. Research topics,

work page 2025