{"id":"83e62fa9-7e1a-41ee-81f9-64933f7fb568","arxiv_id":"2502.00761","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"FIRE aligns multiple data quality raters into a common scale and combines them with reliability and orthogonality weights, improving downstream accuracy of pretrained LLMs while cutting training data to under 37.5% of random selection.","lead":"FIRE is a method for scoring the quality of text used to train large language models, combining several existing quality scores into one integrated rating. It selects a small, high-quality subset of the training corpus, and the authors report that models trained on that subset reach the performance of randomly trained models using under 37.5% of the data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: no baseline trains on GPT-4o's generic 'informative signal' prompt directly, so the reported gains may be due to the oracle rather than to FIRE's integration.","rationale":"The paper has real strengths: a clear problem, a plausible alignment scheme, ablations isolating components, and cost analysis. My concern is not that the reported numbers are fabricated or that GPT-4o is necessarily an invalid oracle; it is that the experiment as designed cannot distinguish 'FIRE works' from 'GPT-4o works.' The alignment step deliberately maps every rater into a space defined by GPT-4o's generic pretraining-quality preference, so FIRE's integrated rating is downstream of that same oracle. The closest baselines, QuRating single raters, are trained on dimension-specific prompts, not on the generic informative-signal prompt, and the Comprehensive Rater baseline uses a multi-criteria prompt that the paper itself shows GPT-4o follows poorly (Appendix E.2). Therefore no existing row in Table 1 controls for the oracle itself. I agree partially with the reader's weakest_assumption: the oracle's reliability matters, but even a perfectly reliable oracle would leave the attribution problem open. The proposed control is inexpensive relative to the already-run 20B-token trainings and would settle the issue. Given that the paper's central claim is about integration, this missing control warrants keeping the verdict CONDITIONAL, with the explicit requirement that the generic-oracle baseline be added before the method is accepted as the source of the gains.","tokens_in":23234,"tokens_out":8612,"duration_ms":82984,"concrete_test":"Add a control baseline using one rater trained on the exact Appendix C.2 prompt: collect roughly 500K pairwise comparisons (same scale as QuRating's rater training) where GPT-4o compares randomly sampled SlimPajama sequences with the generic 'informative signal' instruction; train a 1.3B Sheared-Llama reward model on these preferences; score all 627B tokens; select the top 20B; train the same 1.3B model for 10,000 steps under identical hyperparameters and evaluate on the same eight downstream tasks. Compare average accuracy with FIRE (4 Raters) = 50.0 and FIRE Progressive = 50.4. If the generic-oracle-only run matches or exceeds these values, the headline gain is attributable to GPT-4o rather than to FIRE's integration; if it is clearly lower, FIRE's multi-rater alignment and orthogonality provide measurable independent value.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"FIRE's alignment target is GPT-4o's generic 'informative signal' preference, not any of the four QuRating dimensions. In Section 2.2 Step 3 (Eq. 2) and Appendix C.2, the win rate w_ij is the fraction of samples in rater i's j-th interval that GPT-4o judges as more informative for pretraining relative to a random reference. The fitted win-rate-percentile function defines aligned rating A_i(x), and gamma_i = w_i0 is the top-interval win rate. Hence I(x) = A(x)^T (o ⊙ gamma) (Eq. 9) is ultimately a reweighted estimate of GPT-4o's generic quality preferences, with the four QuRating raters serving as proxies. Yet the experiments compare FIRE only against dimension-specific single raters and ad-hoc integration baselines (Table 1); no baseline trains a rater directly on the Appendix C.2 prompt and selects the top 20B tokens by that single signal. Without this control, the 2.9% improvement over Random (Table 1, Figure 1) cannot be attributed to FIRE's alignment, orthogonality, or progressive selection—it could be entirely explained by GPT-4o's oracle. This is the most load-bearing gap because the central claim is about the integration framework, not about GPT-4o's data preferences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FIRE, a framework that integrates multiple quality raters for LLM pretraining data selection. Ratings from individual raters are first aligned into a unified space by partitioning each rater's score distribution into quantiles and estimating, via GPT-4o pairwise comparisons against a random reference subset, the win rate of each quantile. The aligned ratings are then combined using intrinsic-reliability weights and an orthogonality measure derived from a graph-centrality formulation, and a progressive selection scheme is proposed to refine ratings on successively smaller high-quality subsets. Experiments on SlimPajama with Llama-style 1.3B and 3B models report improvements over Random and several single-rater and integration baselines on eight downstream tasks, with ablations showing that alignment, orthogonality, reliability, and progressive selection each contribute to the final average accuracy.","tokens_in":23561,"tokens_out":5353,"duration_ms":53040,"significance":"If the empirical claims hold, FIRE offers a practical and scalable recipe for combining existing quality raters, and the progressive selection scheme is a sensible way to adapt orthogonality estimates to the selected subset. The paper includes useful supporting material: a cost analysis, scaling experiments to a 3B model over 200B tokens, integration of non-QuRating raters, ablations of the orthogonality function, and a human validation of GPT-4o for one rater dimension. However, the central attribution of the reported gains to FIRE's integration mechanisms is currently under-supported by the experimental design, and the headline quantitative claims are based on single training runs. The contribution is potentially valuable for the pretraining data-selection community, but the evidence needs strengthening before the claims can be accepted.","major_comments":[{"comment":"The alignment signal is GPT-4o's generic \"informative signal\" prompt, yet Table 1 contains no control that trains a single rater directly on that generic prompt and selects the top 20B tokens by it. Without such a control, the reported 2.9% improvement over Random (Table 1, Figure 1) cannot be attributed to FIRE's alignment, orthogonality, or progressive integration rather than to GPT-4o's oracle. Please add this baseline, or an equivalent ablation that isolates the oracle effect, before the central claim can be evaluated.","section":"§2.2 Step 3 (Eq. 2), Appendix C.2, Table 1"},{"comment":"The headline numbers (2.9% over Random and less than 37.5% of Random's training tokens) come from single training runs with no error bars, standard deviations, or significance testing. Because the differences among FIRE variants and baselines are fractions of a point (e.g., 50.0 vs 50.4 in Table 1), at least three seeds per condition and a variance report are needed to establish that these differences are not run noise. This is load-bearing for the paper's main quantitative claims.","section":"§4.2, Figure 1"},{"comment":"The proof asserts that the adjacency matrix M is irreducible because the orthogonality graph is complete, but the stated boundary conditions allow zero edge weights (e.g., perfectly correlated raters). A complete graph with zero-weight edges need not be irreducible, and M^alpha o(0) can converge to zero, making the normalization in Eq. (8) undefined. The theorem should be restated under a positivity or irreducibility assumption, or the proof should be modified to handle zero weights explicitly.","section":"Appendix C.6, Theorem 3"},{"comment":"GPT-4o's win-rate judgments are validated against human experts only for the Educational Value rater (Pearson 0.99). Since the aligned ratings for Writing Style, Required Expertise, and Facts and Trivia all depend on GPT-4o judgments through Eq. (2), the reliability of those dimensions is unverified. Please add human spot-checks for the remaining dimensions or explicitly state that the method's validity is contingent on GPT-4o's oracle quality across all four raters.","section":"Appendix C.3"}],"minor_comments":[{"comment":"The table header 'A VG.' should be 'Avg.'; additionally, the rows for FIRE (2 Raters) and FIRE (3 Raters) report averages over combinations, so the number of combinations and the dispersion across combinations should be stated.","section":"Table 1"},{"comment":"The JSON field in the evaluation prompt is spelled 'quesion'; this should be corrected to 'question'.","section":"Appendix D.5"},{"comment":"The variable η is called a 'decay factor' but is used as a retention proportion (e.g., η=60 retains 60% of the data). Clarify the terminology and the description of line 4 to avoid confusion.","section":"Algorithm 1"},{"comment":"The claims '2.9% higher' and 'less than 37.5% of the training data' are stated without noting that they are derived from a single run and interpolated from learning curves; add a caveat or quantitative uncertainty.","section":"Abstract and §4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and contains a substantial amount of experimental work, but the missing oracle control and the lack of repeated runs are significant obstacles to accepting the central attribution claims. I would encourage the editor to invite a revision rather than reject; the required experiments (a direct GPT-4o-prompt baseline and multi-seed training) appear feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FIRE is a sensible, well-made framework for combining multiple quality raters, and the experiments are directionally positive. But the evaluation omits the single most important control—a baseline that uses GPT-4o's generic 'informative signal' prompt directly, without the raters. Without that, the 2.9% gain cannot be attributed to the integration mechanism.\n\nWhat's genuinely new: the win-rate-percentile alignment turns any rater into a comparable signal using a small number of pairwise comparisons against a random reference; the orthogonality-weighted PageRank is a reasonable way to downweight redundant raters; and the progressive selection is a natural improvement. The ablations (Figure 3) show each component contributes, and the cost analysis is honest.\n\nWhere it gets shaky: the stress-test concern holds up on close reading. All four aligned ratings are defined through GPT-4o's generic preference (Appendix C.2), so the final score is a reweighted estimate of that preference. The single-raters (QuRating) are trained on dimension-specific prompts, so they aren't a substitute for a direct GPT-4o baseline. If a simple direct prompt on top-k selection matches FIRE's accuracy, the complexity adds nothing but cost. That's the central gap.\n\nAlso: all main results are single runs; no error bars. The 3B experiment only compares Random vs FIRE, not against single raters. The hyperparameters (η, β) appear tuned on the same evaluation tasks. And the oracle validation covers only one of four dimensions.\n\nNone of this kills the method—it's a practical paper that many pretraining teams would use—but the claims as stated are not yet supported. A revision that adds the missing baseline, repeats runs, and releases code would make it solid.\n\nWho it's for: people building data pipelines for LLMs, especially at the 1B-3B scale. It deserves a serious referee—the idea is useful and the gaps are fixable, not fundamental.\n\nI'd send it out for review, but I'd ask for the direct GPT-4o control and error bars before accepting.","headline":"A practical integration framework with a real evaluation gap: the reported gains could come from the GPT-4o oracle, not from the integration itself.","tokens_in":24106,"tokens_out":3256,"would_cite":true,"duration_ms":32400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FIRE claims that aligning multiple quality raters into win-rate scores, then weighting them by reliability and independence, ranks pretraining data so that selecting the top slice lifts downstream accuracy by 2.9% over random selection…","keywords":["FIRE","data quality rating","rating alignment","rater integration","win rate","orthogonality","progressive data selection","pretraining data selection"],"falsifier":"Replace the GPT-4o oracle with human expert pairwise comparisons on all four rater dimensions, rerun the full alignment, integration, and top-20B selection, and train the same 1.3B model for 20B tokens; if the human-oracle selection does not reproduce a comfortable accuracy margin over Random, then FIRE's reported gains ride on the specific judgment of GPT-4o rather than on its alignment and orthogonality machinery. A complementary check is to train a model on the bottom-rated 20B tokens: if its accuracy is indistinguishable from the top-rated run, the integrated rating is not actually ordering the corpus by pretraining value.","tokens_in":23033,"feed_emoji":"📊","tokens_out":12463,"duration_ms":99872,"temperature":0.7,"pith_summary":"FIRE claims that the conflicting quality scores a corpus receives from different raters can be fused into a single ranking that genuinely improves pretraining. Instead of averaging raw scores, each rater's quantile slices are compared against a random reference sample: the chance that a rater's top slices beat random data, as judged by GPT-4o, becomes a shared 'win rate' currency. Weighting these aligned ratings by each rater's reliability and by how independent the raters are from one another, then re-weighting locally as selection narrows, yields one integrated score per document. On the 627B-token SlimPajama corpus with four quality dimensions, training on the top-rated slice lifts average accuracy over eight tasks to 50.4 versus 47.5 for random selection, and reaches the random baseline's performance level with less than 37.5% of the data. If the claim holds, any scalar quality signal can be aligned, integrated, and converted into measurable pretraining gains.","feed_headline":"Fusing quality raters lifts LLM pretraining accuracy 2.9%","feed_subtitle":"Aligned win rates reach the random baseline's accuracy with under 37.5% of the training data.","key_machinery":"The load-bearing object is the integrated rating $I(x) = A(x)^T (o \\odot \\gamma)$, built in two stages. Rating alignment converts each rater's raw score into a win-rate-percentile curve: the dataset is sorted by the rater, sliced into quantiles, and each slice is compared against a random reference subset by GPT-4o, producing a win rate $w_{ij}$ that a polynomial spline interpolation turns into a continuous function from percentile to aligned rating. Rater integration then sets each rater's weight as the product of $\\gamma_i$, the win rate of the rater's best slice interpreted as intrinsic reliability, and $o_i$, an orthogonality score obtained by iterating a PageRank-style update $o \\leftarrow M o$ on the complete graph whose edge weights are a symmetrized Gaussian of the raters' Pearson correlation (set to 0 for perfectly correlated raters and 0.5 for uncorrelated raters). A progressive selection scheme recomputes the orthogonality weights inside shrinking quantile segments, responding to the paper's observation that rater correlations change across the rating distribution. The integration assumes a linear additive relation between dimensions, a simplification the authors themselves flag as a limitation.","core_discovery":"The paper's central claim is that data-quality ratings from multiple sources, which are incommensurable in their raw form, can be made commensurable and combined into a single pretraining-useful score. Each rater's scores are turned into win rates against a uniformly sampled reference subset: for each quantile of the rater's ranking, GPT-4o judges whether its samples are more informative than samples from the reference set, and a spline curve converts any raw score into an aligned rating. The integrated rating is $I(x) = A(x)^T (o \\odot \\gamma)$: a weighted sum of aligned ratings whose weights are the product of a rater's intrinsic reliability (the win rate of its best slice) and its overall orthogonality, a graph-centrality measure that downweights raters correlated with the others. On the 627B-token SlimPajama corpus, selecting the top slice of this ranking trains a 1.3B model to 50.4 average accuracy on eight tasks versus 47.5 for random selection, and the same selection reaches the random baseline's accuracy with fewer than 37.5% of the tokens; the authors further claim that the ranking improves as raters are added and that the gain persists at larger scale, where FIRE exceeds Random by 1.6% average with a 3B model trained on 200B tokens.","pith_inferences":["Editorial inference: the same alignment-and-orthogonality machinery does not depend on GPT-4o specifically; any oracle able to make relative judgments, such as human experts, a cheaper model, or downstream performance probes, could supply the win rates, so the framework's value could be tested by swapping oracles and checking whether the ranking and its downstream gains survive.","Editorial inference: the progressive scheme encodes a general lesson, namely that redundancy between quality signals is rank-dependent, so computing weights per quantile rather than globally is likely to help any multi-signal ranking task, such as retrieval filtering or dataset distillation, not just pretraining.","Editorial inference: the reported 2.5x token reduction implies a concrete economic metric for data selection, FLOPs saved per unit of accuracy, and since FIRE's integration adds only tens of thousands of pairwise comparisons per rater, its marginal cost is dominated by the single-rater scoring it already consumes.","Editorial inference: the sharp failure of the single comprehensive prompt suggests the four dimensions are genuinely separable for current LLMs, and a judge model that could score all dimensions simultaneously, if one existed, might make FIRE's decomposition unnecessary; that is a testable prediction as judge models improve."],"forward_implications":["Selecting the top-rated slice of SlimPajama lifts average downstream accuracy from 47.5 (Random) to 50.0 (FIRE with four raters) and 50.4 (FIRE Progressive), and adding raters monotonically improves the ranking: 48.4 for two, 49.6 for three, 50.0-50.4 for four.","FIRE-selected data reaches the Random baseline's accuracy with roughly 2.5 times fewer pretraining tokens, under 37.5% of the data, and the advantage persists at the 3B-parameter, 200B-token scale, where FIRE beats Random by 1.6% average.","The framework is rater-agnostic: integrating an unrelated set of raters (Required Expertise, Facts and Trivia, DSIR-Book, DSIR-Wiki) still beats each individual rater and Random (49.0 average), so new quality signals can be plugged in without redesigning the integration.","Each integration component earns its keep: removing rating alignment costs 3.3 points, removing orthogonality costs 2.0, removing intrinsic reliability costs 0.2, and removing all three drops below naive averaging, making alignment the critical step and orthogonality second.","Naive ways of combining criteria fail where FIRE succeeds: a single multi-criteria prompt degrades GPT-4o's agreement with human annotators (down to 0.32 correlation on Required Expertise), whereas FIRE's per-rater comparison and weighted integration keeps quality high across all dimensions."],"supporting_citations":[{"why":"Supplies the four single raters FIRE integrates and the QuRating baselines plus Mix Criteria that FIRE must outperform.","marker":"(Wettig et al., 2024)"},{"why":"The GPT-4o system used as the pairwise comparison oracle that produces every win rate in the alignment step.","marker":"(Islam and Moushi, 2024)"},{"why":"Provides the ASK-LLM and Density baselines and the precedent of asking LLMs to judge pretraining data quality.","marker":"(Sachdeva et al., 2024)"},{"why":"The DSIR importance-resampling baseline that FIRE beats, and whose scores FIRE also demonstrates integrating in Appendix D.4.","marker":"(Xie et al., 2023)"},{"why":"The PageRank centrality iteration that FIRE adapts to aggregate rater orthogonality into a single weight.","marker":"(Page et al., 1999)"},{"why":"The SlimPajama 627B-token corpus that serves as the selection pool and training data for all experiments.","marker":"(Soboleva et al., 2023)"}],"fun_headline_variants":["FIRE fuses quality raters to lift LLM pretraining accuracy by 2.9%","One integrated score from many quality raters cuts pretraining data by 62.5%","FIRE aligns win rates from multiple raters to select better pretraining data","FIRE's progressive selection beats random baseline with 62.5% fewer tokens","FIRE: merged quality ratings train LLMs with under 37.5% of the data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4o's pairwise judgment of which text carries 'more informative signal' for pretraining is a reliable oracle for what actually helps pretraining, across all four quality dimensions, yet only the Educational Value rater's win rates were checked against human experts (Pearson 0.99), leaving Writing Style, Required Expertise, and Facts and Trivia unvalidated.","fun_headline_variants_meta":{"raw":{"variants":["FIRE fuses quality raters to lift LLM pretraining accuracy by 2.9%","One integrated score from many quality raters cuts pretraining data by 62.5%","FIRE aligns win rates from multiple raters to select better pretraining data","FIRE's progressive selection beats random baseline with 62.5% fewer tokens","FIRE: merged quality ratings train LLMs with under 37.5% of the data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000881,"raw_usage":{"total_tokens":3825,"prompt_tokens":983,"completion_tokens":2842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":2728}},"tokens_in":599,"tokens_out":2842,"duration_ms":23042,"temperature":1.0,"reasoning_tokens":2728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:47:38.899518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the GPT-4o oracle with human expert pairwise comparisons on all four rater dimensions, rerun the full alignment, integration, and top-20B selection, and train the same 1.3B model for 20B tokens; if the human-oracle selection does not reproduce a comfortable accuracy margin over Random, then FIRE's reported gains ride on the specific judgment of GPT-4o rather than on its alignment and orthogonality machinery. A complementary check is to train a model on the bottom-rated 20B tokens: if its accuracy is indistinguishable from the top-rated run, the integrated rating is not actually ordering the corpus by pretraining value.","supporting_citations":[],"review_version":1}