{"id":"1e23a87e-59fc-456f-978a-55e6c67b7290","arxiv_id":"2507.12041","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"As feedback becomes more granular, sophisticated supervised aggregation outperforms regularized averaging, needing about 44% fewer raters at 5-point scales.","lead":"The paper reports that when humans rate items on fine-grained scales, a machine-learned aggregator needs roughly half as many raters as plain averaging to reach the same accuracy. This suggests practical cost savings in collecting feedback for AI training, surveys, and recommender systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Measured headcount savings are within a single fixed 20/19 split: SL may exploit stable rater idiosyncrasies shared by the pool, so the 44%/54% reductions may not transfer to new raters.","rationale":"Full-text review. The paper's central claim is well motivated and the informal theory is transparently informal, but the empirical support hangs on a single fixed split of 39 crowdworkers. The reader's weakest assumption—that the 19-worker output CDF stands in for the population—is close to my concern but I sharpen it: even if the 19-worker CDF were a perfect population proxy, the experiment never varies which workers provide input. SL is trained and evaluated on the same 20 known input workers, so the K axis counts how many of these known workers' responses are revealed, not how many novel raters are hired. Stable rater-specific scale usage within this single pool is a resource SL can mine and RegAvg cannot, making the measured 44%/54% headcount savings an upper bound on what would be achieved with new raters. The paper itself flags the single-dataset limitation (Section 7) and partially anticipates this by describing in Section 6 how SL would be deployed; the missing piece is a worker-level out-of-sample experiment. A leave-one-worker-out evaluation, or a fresh-rater collection on the same task units, would settle whether the advantage transfers. Given this unresolved but addressable threat, the CONDITIONAL verdict stands; the paper should be revised either to add such a test or to explicitly reframe the claim as within-pool performance.","tokens_in":24087,"tokens_out":5486,"duration_ms":67277,"concrete_test":"Run a leave-one-worker-out evaluation. For each of the 39 workers w, set the target for every task unit to w's own 11-point score (its degenerate CDF), train SL on the other 38 workers' feedback as input, and compare the K-required curve against RegAvg for 2-, 5-, and 11-point granularities. If the advantage over RegAvg at K=18 shrinks substantially or vanishes compared to the reported 44%/54% (i.e., SL no longer matches RegAvg's loss with far fewer 'individuals'), the headline result is driven by within-pool idiosyncrasy rather than granularity. Since the target here is an individual rather than a 19-worker average, also run a secondary check: recompute the main protocol with the input/output roles swapped (19 input, 20 output) and with 5 additional random 20/19 splits; if the 44%/54% figures are highly split-dependent, this corroborates the transfer concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline quantity—the 44%/54% reduction in individuals needed—is measured entirely inside a single fixed 20/19 split of the same 39 crowdworkers. Every environment and every K in Figures 3–5 draws input from the same 20 input-set workers and evaluates against the same 19 output-set workers; the 5-fold cross-validation in Section 5.2 is over task units, not over workers. SL is therefore trained to predict the 19 known output workers from the 20 known input workers, and nothing in the experiment tests whether the learned map transfers to novel raters. Since rater idiosyncrasies (harshness, central tendency, extreme responding) are stable within this pool and are shared by construction across the input and output sets, SL can exploit them in a way that regularized averaging cannot. The practical reading of the claim—that hiring about 44% fewer people suffices—requires out-of-pool transfer, and the current experiment cannot establish it. The paper's Section 7 candidly notes the single-dataset limitation, and Assumption 1 in Section 5.4 is close to the conclusion, but the experimental bottleneck is the missing worker-level out-of-sample check, not the informal theory.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the problem of predicting a population's distribution of ordered feedback from a small number of respondents. The authors claim that the advantage of sophisticated aggregation (exemplified by a supervised MLP, SL) over regularized averaging (RegAvg) grows with the granularity of the rating scale. To support this, they introduce a new dataset of 1,020 social-attitude task units rated on an 11-point scale by 39 Thai crowdworkers, and they construct binary and 5-point variants by deterministic or probabilistic binning. The experiment splits the 39 workers into 20 input and 19 output raters, compares RegAvg with SL across K using 5-fold cross-validation over task units, and reports that SL matches RegAvg with about 44% and 54% fewer individuals on 5-point and 11-point scales, while the advantage on binary feedback is small. Section 5.4 gives an informal argument based on two assumptions, and Section 6 extends the analysis to RLHF-style preference signals.","tokens_in":24381,"tokens_out":4731,"duration_ms":54685,"significance":"The question is practically important: if the claim holds, it implies substantial savings in human-annotation budgets and provides a simple design rule (use granular scales only if you also upgrade the aggregation method). The paper's strengths include a new, openly described dataset; a clean separation of training and evaluation environments; a sensible ordinal loss function; and an unusually candid limitations section. The empirical comparison is an independent data point rather than an artifact of the formal framework. The barrier to full acceptance is that the headline quantitative claim is measured within one fixed worker pool of 39 people, and the theoretical argument for generalization rests on an assumption that is close to the conclusion. These issues are fixable and do not invalidate the empirical comparison as a case study.","major_comments":[{"comment":"The central claim that sophistication reduces headcount is established only within a single fixed 20/19 split of the same 39 crowdworkers. Every eval-environment draws input from the same 20 input-set workers and evaluates against the same 19 output-set workers; the 5-fold cross-validation described in Section 5.2 is over task units, not over workers. Because rater idiosyncrasies such as harshness, central tendency, and extreme responding are stable within this pool and are shared by construction between the input and output sets, SL can exploit them while RegAvg cannot. The practical reading of the headline result, hiring about 44% fewer individuals, requires transfer to new raters, which the current experiment does not test. Please add a worker-level holdout analysis (for example, repeated random splits into input/output workers or leave-one-worker-out evaluation), or clearly restrict the claim to within the studied worker pool; the same limitation applies to the RLHF-style savings reported in Section 6.","section":"Section 5.1 and 5.3"},{"comment":"Assumption 1 states that the information advantage of the original feedback Yt over the binarized feedback B(y)_t about the output-set CDF element Ot(y) increases with feedback granularity. This is essentially the paper's main claim restated for a single CDF element; the informal justification gives plausibility but does not derive the assumption from more basic conditions. Since Claims 2 through 6 and the argument that the result extends beyond the current experimental setup all depend on this assumption, the theoretical section does not provide independent evidence for the central claim beyond the experiment itself. Please either prove Assumption 1 under explicit generative assumptions, or test it directly with information-theoretic estimates on the collected data, or label it as a conjecture and soften the wording that the paper 'shows' the monotonic relationship.","section":"Section 5.4, Assumption 1"}],"minor_comments":[{"comment":"The prior Q0 is described as the average of the empirical CDFs of 'OOS individuals'; the acronym OOS is not defined and appears to refer to the output-set workers. Please define it or use a consistent name throughout.","section":"Section 5.2"},{"comment":"The binary dataset construction assigns neutral scores randomly to {-1, +1}; the results for binary feedback may depend on this random assignment. Reporting the variance over random assignment seeds, or at least stating the seed used, would help interpret the near-tie between SL and RegAvg on the binary scale.","section":"Section 5.1"},{"comment":"The headcount savings in Figure 4 are obtained by interpolating between mean losses at integer values of K. Please state explicitly the interpolation method (a uniform grid with linear interpolation appears to be used) and note that the reported savings inherit any approximation error from this interpolation.","section":"Section 5.3"},{"comment":"There are several typos in this section, including 'a individual' and 'an a score'; similar typos appear in the appendix tables ('feeedback' in Tables 5 and 6). A careful proofread would improve readability.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations and the empirical setup is carefully described, but the headline 'fewer individuals' claim is not supported outside the single 39-worker pool. The most important fix is a worker-level out-of-sample evaluation; without it, the practical claim is not established. The Assumption 1 issue is also real: the informal theory does not yet provide independent support for the monotonicity claim. Both are fixable within the manuscript's scope, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's core claim—SL's edge over RegAvg grows with feedback granularity—is plausible, well-presented, and backed by a newly collected dataset, but the headline headcount savings (44% at 5 points, 54% at 11) are measured entirely inside a fixed 20/19 split of 39 Thai crowdworkers. The experiment does not test whether the learned aggregator transfers to novel raters, so I read the savings as in-pool rather than out-of-pool.\n\nWhat is actually new: the dataset itself, plus the specific granularity-dependent gap, which I have not seen in prior work. The paper does several things well. It releases the data and code, uses a sensible ordinal loss, gives bootstrap confidence intervals, and performs 5-fold cross-validation over task units. It also separates the empirical result from an informal theory, labeling assumptions explicitly and listing limitations in Section 7. That transparency earns credit.\n\nThe soft spots are real but proportionate. The single-dataset problem is acknowledged by the authors, but the deeper issue is the missing worker-level out-of-sample check. Because SL is trained to predict the 19 output workers from the 20 input workers, and because rater idiosyncrasies are stable within this pool and shared by construction, SL can exploit patterns that a newly hired rater would not share. The stress-test note is right: the 5-fold CV is over task units, not workers, so it cannot establish transfer to new raters. Second, Assumption 1 in Section 5.4 basically restates the central claim, but since it is flagged as an assumption rather than smuggled in, it weakens the theory without invalidating the empirical finding. Third, the 5-point and binary datasets are derived by binning the 11-point scores rather than collected at those scales; acceptable for a first pass, but it slightly confounds granularity with the mapping. The informal proof also leans on strong optimality assumptions (Assumption 2) that are unlikely to hold exactly with 1020 training units.\n\nWho is this for? Practitioners in RLHF, crowdsourcing, or survey aggregation who want cheaper feedback, and methodologists interested in distribution prediction. It deserves a serious referee: the question is important, the data are new and public, and the pattern is worth checking. A referee should push for an out-of-pool rater experiment, or at least demand that the abstract's 'half as many individuals' be qualified as conditional on the rater pool. I recommend accepting it into peer review with that expectation.","headline":"A careful single-dataset study showing sophisticated aggregation beats regularized averaging as feedback gets more granular, but the headline savings may not hold for new raters beyond the fixed 20/19 split.","tokens_in":24867,"tokens_out":2767,"would_cite":true,"duration_ms":33016,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that as feedback becomes more granular, a learned aggregation method increasingly beats regularized averaging, needing about half the raters on 5-point scales.","keywords":["human feedback aggregation","feedback granularity","distribution prediction","regularized averaging","supervised learning","ordinal labels","RLHF","crowdsourcing"],"falsifier":"Run the same SL versus RegAvg comparison on a second, larger population of several hundred raters rating the same task units and evaluate against the full population CDF rather than a 19-person output set; if SL no longer matches RegAvg with roughly half the raters on 5-point feedback, the granularity mechanism does not transfer. A more targeted check is to measure the mutual information between full ratings and binarized ratings about the output CDF and see whether it actually grows with scale.","tokens_in":23905,"feed_emoji":"📊","tokens_out":7610,"duration_ms":78811,"temperature":0.7,"pith_summary":"Human feedback is increasingly collected on multi-point scales, and predicting a population's feedback distribution from a small sample is a common task. The paper argues that how you aggregate that feedback matters more as the scale gets finer: a learned aggregator (supervised learning) gains a growing edge over regularized averaging as feedback becomes more granular. On their social-attitudes dataset, the learned method needs about 44% fewer raters on a 5-point scale and 54% fewer on an 11-point scale to match regularized averaging, while on binary feedback the two are nearly tied. The reason offered is that regularized averaging silently throws away the information inside each rating by binarizing it, and that binarization loss grows with granularity. If true, the result changes how feedback collection budgets should be spent, especially in RLHF and survey pipelines that already collect granular ratings.","feed_headline":"Smarter aggregation needs half the raters on 5-point feedback","feed_subtitle":"With binary ratings simple averaging is fine, but on 5-point or finer scales a learned aggregator matches it with about half the people.","key_machinery":"The load-bearing object is the binarized form of each rating. RegAvg's prediction of an output CDF element $O_t(y)$ is a convex combination of a prior and the average of indicators $\\mathbf{1}[Y_{t,k} \\le y]$, so it only ever sees whether each rater fell below or above the threshold $y$; all magnitude information inside the rating is discarded. Against this, the paper sets supervised learning, an MLP whose output is constrained to valid CDFs and trained to predict the output-set empirical CDF from input-set ratings. The argument that the gap must widen with granularity rests on two assumptions: (Assumption 1) the extra information in the full rating $Y_t$ about $O_t(y)$ relative to the binarized version $\\mathbf{B}^{(y)}_t$ increases with granularity, and (Assumption 2) SL and a binarized-input counterpart SLb optimally use all relevant information. Claims 1 through 6 then derive that the advantage ratio $K/K'$ of SL over RegAvg increases with granularity. The ordinal CDF log-loss in Equation (2) is what makes the comparison meaningful for ordered scales.","core_discovery":"The paper's central claim is that as feedback granularity increases, the advantage of sophisticated aggregation policies over regularized averaging increases. Concretely, supervised learning (SL), which is trained to predict a held-out group's empirical CDF from a small input group's ratings, matches regularized averaging (RegAvg) using roughly 44% fewer individuals with 5-point feedback and 54% fewer with 11-point feedback, whereas with binary feedback the two methods perform almost identically. The paper also shows that in loss functions typical of RLHF reward-model training, SL requires about 29% fewer raters with 5-point feedback and 36% fewer with 11-point feedback. The reason, the paper argues, is that RegAvg's prediction of any CDF element $O_t(y)$ depends only on the binarized indicators $\\mathbf{1}[Y_{t,k} \\le y]$, while SL can exploit the full rating; as the scale becomes finer, the information lost by binarization grows, so a method that uses all available information should improve relative to one that does not.","pith_inferences":["Extension: the same binarization-information mechanism should apply to other ordinal-label domains, such as recommender ratings, so a learned aggregator trained on held-out user ratings should beat the simple item mean by a margin that grows with rating-scale granularity; this is directly testable on existing rating datasets.","Caveat to test: the paper itself notes the advantage may stop growing beyond very fine scales because raters may not use the full scale; a natural test is to measure SL's advantage on 21-point or 101-point versions of the same task units.","Practical extension: for AI training, comparing reward models trained on SL-aggregated labels versus RegAvg-aggregated labels on a downstream task would test whether the aggregation gain survives the full RLHF pipeline, not just the preference-signal loss.","Design implication: data collection for distribution prediction should include more overlapping raters per task unit and finer scales when a learned aggregator will be used, since the value of that overlap increases with granularity."],"forward_implications":["With 5-point or finer feedback, switching from regularized averaging to a learned aggregator roughly halves the number of human raters needed to achieve the same distribution-prediction accuracy.","With binary feedback, simple regularized averaging is nearly as good as learned aggregation, so sophistication buys little on two-point scales.","RLHF pipelines that collect granular ratings and then binarize or average them could match current aggregated-feedback quality with roughly 20% fewer total scores by training a learned aggregator on the original ratings, after accounting for the extra labels needed to train it.","The advantage grows monotonically with granularity on the scales tested (2, 5, and 11 points), implying that spending on finer rating scales and on better aggregation are complements rather than substitutes.","Because training SL needs output-set labels, the net benefit in applications depends on how many task units are used for training and how many are served; the paper's RLHF example shows a net saving for 10,000 task units."],"supporting_citations":[{"why":"Supplies the canonical RLHF reward-model loss and the observation that granular human scores are binarized and averaged before training.","marker":"Stiennon et al. (2020)"},{"why":"Documents a concrete RLHF scoring scale and thresholded loss, used as evidence that typical pipelines convert granular feedback to binary or tertiary form.","marker":"Bai et al. (2022)"},{"why":"Exemplifies the reward-model loss family in modern LLM training, motivating the Section 6 claim about binarization in typical pipelines.","marker":"Touvron et al. (2023)"},{"why":"Continues the same reward-model loss family in Llama 3, supporting the claim that the binarize-then-average pattern persists in current practice.","marker":"Grattafiori et al. (2024)"},{"why":"Provides the ordinal CDF log-loss formulation used as the paper's main evaluation metric.","marker":"Niu et al. (2016)"},{"why":"Supplies the exchangeability background invoked to define the population CDF as a limit of empirical CDFs.","marker":"Barber et al. (2024)"}],"fun_headline_variants":["Granular feedback boosts smarter aggregation gains","Smarter aggregation wins with finer feedback scales","Half the raters: how granular feedback unlocks better aggregation","With fine-grained feedback, smart aggregation halves rater needs","Granularity unlocks aggregation gains: binary not so much"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 19 held-out raters' empirical distribution faithfully represents the infinite population's distribution, and that granular ratings carry increasingly more predictive information than their binarized versions about that distribution; if either fails, the measured 44%–54% reductions will not transfer to new raters or settings.","fun_headline_variants_meta":{"raw":{"variants":["Granular feedback boosts smarter aggregation gains","Smarter aggregation wins with finer feedback scales","Half the raters: how granular feedback unlocks better aggregation","With fine-grained feedback, smart aggregation halves rater needs","Granularity unlocks aggregation gains: binary not so much"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000721,"raw_usage":{"total_tokens":3247,"prompt_tokens":965,"completion_tokens":2282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":2208}},"tokens_in":581,"tokens_out":2282,"duration_ms":18741,"temperature":1.0,"reasoning_tokens":2208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:55:35.954281+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same SL versus RegAvg comparison on a second, larger population of several hundred raters rating the same task units and evaluate against the full population CDF rather than a 19-person output set; if SL no longer matches RegAvg with roughly half the raters on 5-point feedback, the granularity mechanism does not transfer. A more targeted check is to measure the mutual information between full ratings and binarized ratings about the output CDF and see whether it actually grows with scale.","supporting_citations":[],"review_version":1}