Pith. sign in

REVIEW 4 major objections 4 minor 13 references

FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FIRE claims that aligning multiple quality raters into win-rate scores, then weighting them by reliability and independence, ranks pretraining data so that selecting the top slice lifts downstream accuracy by 2.9% over random selection…

desk verdict A practical integration framework with a real evaluation gap: the reported gains could come from the GPT-4o oracle, not from the integration itself. read the letter →

arxiv 2502.00761 v3 pith:2RO4S7UC submitted 2025-02-02 cs.CL

Liangyu Xu , Xuemiao Zhang , Feiyu Duan , Sirui Wang , Rongxiang Weng , Jingang Wang , Xunliang Cai This is my paper · ORCID
classification cs.CL
keywords FIREdataqualityratingalignmentraterintegrationwinrateorthogonalityprogressiveselectionpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FIRE claims that the conflicting quality scores a corpus receives from different raters can be fused into a single ranking that genuinely improves pretraining. Instead of averaging raw scores, each rater's quantile slices are compared against a random reference sample: the chance that a rater's top slices beat random data, as judged by GPT-4o, becomes a shared 'win rate' currency. Weighting these aligned ratings by each rater's reliability and by how independent the raters are from one another, then re-weighting locally as selection narrows, yields one integrated score per document. On the 627B-token SlimPajama corpus with four quality dimensions, training on the top-rated slice lifts average accuracy over eight tasks to 50.4 versus 47.5 for random selection, and reaches the random baseline's performance level with less than 37.5% of the data. If the claim holds, any scalar quality signal can be aligned, integrated, and converted into measurable pretraining gains.

What carries the argument

The load-bearing object is the integrated rating $I(x) = A(x)^T (o \odot \gamma)$, built in two stages. Rating alignment converts each rater's raw score into a win-rate-percentile curve: the dataset is sorted by the rater, sliced into quantiles, and each slice is compared against a random reference subset by GPT-4o, producing a win rate $w_{ij}$ that a polynomial spline interpolation turns into a continuous function from percentile to aligned rating. Rater integration then sets each rater's weight as the product of $\gamma_i$, the win rate of the rater's best slice interpreted as intrinsic reliability, and $o_i$, an orthogonality score obtained by iterating a PageRank-style update $o \leftarrow M o$ on the complete graph whose edge weights are a symmetrized Gaussian of the raters' Pearson correlation (set to 0 for perfectly correlated raters and 0.5 for uncorrelated raters). A progressive selection scheme recomputes the orthogonality weights inside shrinking quantile segments, responding to the paper's observation that rater correlations change across the rating distribution. The integration assumes a linear additive relation between dimensions, a simplification the authors themselves flag as a limitation.

What would settle it

Replace the GPT-4o oracle with human expert pairwise comparisons on all four rater dimensions, rerun the full alignment, integration, and top-20B selection, and train the same 1.3B model for 20B tokens; if the human-oracle selection does not reproduce a comfortable accuracy margin over Random, then FIRE's reported gains ride on the specific judgment of GPT-4o rather than on its alignment and orthogonality machinery. A complementary check is to train a model on the bottom-rated 20B tokens: if its accuracy is indistinguishable from the top-rated run, the integrated rating is not actually ordering the corpus by pretraining value.

Watch

Extended reading notes

Core claim

The paper's central claim is that data-quality ratings from multiple sources, which are incommensurable in their raw form, can be made commensurable and combined into a single pretraining-useful score. Each rater's scores are turned into win rates against a uniformly sampled reference subset: for each quantile of the rater's ranking, GPT-4o judges whether its samples are more informative than samples from the reference set, and a spline curve converts any raw score into an aligned rating. The integrated rating is $I(x) = A(x)^T (o \odot \gamma)$: a weighted sum of aligned ratings whose weights are the product of a rater's intrinsic reliability (the win rate of its best slice) and its overall orthogonality, a graph-centrality measure that downweights raters correlated with the others. On the 627B-token SlimPajama corpus, selecting the top slice of this ranking trains a 1.3B model to 50.4 average accuracy on eight tasks versus 47.5 for random selection, and the same selection reaches the random baseline's accuracy with fewer than 37.5% of the tokens; the authors further claim that the ranking improves as raters are added and that the gain persists at larger scale, where FIRE exceeds Random by 1.6% average with a 3B model trained on 200B tokens.

Load-bearing premise

The load-bearing premise is that GPT-4o's pairwise judgment of which text carries 'more informative signal' for pretraining is a reliable oracle for what actually helps pretraining, across all four quality dimensions, yet only the Educational Value rater's win rates were checked against human experts (Pearson 0.99), leaving Writing Style, Required Expertise, and Facts and Trivia unvalidated.

Editorial extensions

If this is right

  • Selecting the top-rated slice of SlimPajama lifts average downstream accuracy from 47.5 (Random) to 50.0 (FIRE with four raters) and 50.4 (FIRE Progressive), and adding raters monotonically improves the ranking: 48.4 for two, 49.6 for three, 50.0-50.4 for four.
  • FIRE-selected data reaches the Random baseline's accuracy with roughly 2.5 times fewer pretraining tokens, under 37.5% of the data, and the advantage persists at the 3B-parameter, 200B-token scale, where FIRE beats Random by 1.6% average.
  • The framework is rater-agnostic: integrating an unrelated set of raters (Required Expertise, Facts and Trivia, DSIR-Book, DSIR-Wiki) still beats each individual rater and Random (49.0 average), so new quality signals can be plugged in without redesigning the integration.
  • Each integration component earns its keep: removing rating alignment costs 3.3 points, removing orthogonality costs 2.0, removing intrinsic reliability costs 0.2, and removing all three drops below naive averaging, making alignment the critical step and orthogonality second.
  • Naive ways of combining criteria fail where FIRE succeeds: a single multi-criteria prompt degrades GPT-4o's agreement with human annotators (down to 0.32 correlation on Required Expertise), whereas FIRE's per-rater comparison and weighted integration keeps quality high across all dimensions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same alignment-and-orthogonality machinery does not depend on GPT-4o specifically; any oracle able to make relative judgments, such as human experts, a cheaper model, or downstream performance probes, could supply the win rates, so the framework's value could be tested by swapping oracles and checking whether the ranking and its downstream gains survive.
  • Editorial inference: the progressive scheme encodes a general lesson, namely that redundancy between quality signals is rank-dependent, so computing weights per quantile rather than globally is likely to help any multi-signal ranking task, such as retrieval filtering or dataset distillation, not just pretraining.
  • Editorial inference: the reported 2.5x token reduction implies a concrete economic metric for data selection, FLOPs saved per unit of accuracy, and since FIRE's integration adds only tens of thousands of pairwise comparisons per rater, its marginal cost is dominated by the single-rater scoring it already consumes.
  • Editorial inference: the sharp failure of the single comprehensive prompt suggests the four dimensions are genuinely separable for current LLMs, and a judge model that could score all dimensions simultaneously, if one existed, might make FIRE's decomposition unnecessary; that is a testable prediction as judge models improve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces FIRE, a framework that integrates multiple quality raters for LLM pretraining data selection. Ratings from individual raters are first aligned into a unified space by partitioning each rater's score distribution into quantiles and estimating, via GPT-4o pairwise comparisons against a random reference subset, the win rate of each quantile. The aligned ratings are then combined using intrinsic-reliability weights and an orthogonality measure derived from a graph-centrality formulation, and a progressive selection scheme is proposed to refine ratings on successively smaller high-quality subsets. Experiments on SlimPajama with Llama-style 1.3B and 3B models report improvements over Random and several single-rater and integration baselines on eight downstream tasks, with ablations showing that alignment, orthogonality, reliability, and progressive selection each contribute to the final average accuracy.

Significance. If the empirical claims hold, FIRE offers a practical and scalable recipe for combining existing quality raters, and the progressive selection scheme is a sensible way to adapt orthogonality estimates to the selected subset. The paper includes useful supporting material: a cost analysis, scaling experiments to a 3B model over 200B tokens, integration of non-QuRating raters, ablations of the orthogonality function, and a human validation of GPT-4o for one rater dimension. However, the central attribution of the reported gains to FIRE's integration mechanisms is currently under-supported by the experimental design, and the headline quantitative claims are based on single training runs. The contribution is potentially valuable for the pretraining data-selection community, but the evidence needs strengthening before the claims can be accepted.

major comments (4)
  1. [§2.2 Step 3 (Eq. 2), Appendix C.2, Table 1] The alignment signal is GPT-4o's generic "informative signal" prompt, yet Table 1 contains no control that trains a single rater directly on that generic prompt and selects the top 20B tokens by it. Without such a control, the reported 2.9% improvement over Random (Table 1, Figure 1) cannot be attributed to FIRE's alignment, orthogonality, or progressive integration rather than to GPT-4o's oracle. Please add this baseline, or an equivalent ablation that isolates the oracle effect, before the central claim can be evaluated.
  2. [§4.2, Figure 1] The headline numbers (2.9% over Random and less than 37.5% of Random's training tokens) come from single training runs with no error bars, standard deviations, or significance testing. Because the differences among FIRE variants and baselines are fractions of a point (e.g., 50.0 vs 50.4 in Table 1), at least three seeds per condition and a variance report are needed to establish that these differences are not run noise. This is load-bearing for the paper's main quantitative claims.
  3. [Appendix C.6, Theorem 3] The proof asserts that the adjacency matrix M is irreducible because the orthogonality graph is complete, but the stated boundary conditions allow zero edge weights (e.g., perfectly correlated raters). A complete graph with zero-weight edges need not be irreducible, and M^alpha o(0) can converge to zero, making the normalization in Eq. (8) undefined. The theorem should be restated under a positivity or irreducibility assumption, or the proof should be modified to handle zero weights explicitly.
  4. [Appendix C.3] GPT-4o's win-rate judgments are validated against human experts only for the Educational Value rater (Pearson 0.99). Since the aligned ratings for Writing Style, Required Expertise, and Facts and Trivia all depend on GPT-4o judgments through Eq. (2), the reliability of those dimensions is unverified. Please add human spot-checks for the remaining dimensions or explicitly state that the method's validity is contingent on GPT-4o's oracle quality across all four raters.
minor comments (4)
  1. [Table 1] The table header 'A VG.' should be 'Avg.'; additionally, the rows for FIRE (2 Raters) and FIRE (3 Raters) report averages over combinations, so the number of combinations and the dispersion across combinations should be stated.
  2. [Appendix D.5] The JSON field in the evaluation prompt is spelled 'quesion'; this should be corrected to 'question'.
  3. [Algorithm 1] The variable η is called a 'decay factor' but is used as a retention proportion (e.g., η=60 retains 60% of the data). Clarify the terminology and the description of line 4 to avoid confusion.
  4. [Abstract and §4.2] The claims '2.9% higher' and 'less than 37.5% of the training data' are stated without noting that they are derived from a single run and interpolated from learning curves; add a caveat or quantitative uncertainty.

Circularity Check

2 steps flagged · score 1.0 of 10

No significant circularity: the held-out downstream benchmarks are independent of FIRE's construction; only a tautological theorem and a GPT-4o self-confirming case study are minor.

  1. self definitional [Section 2.3, Theorem 1; Appendix B (Theorem 2) proof]
    "The overall orthogonality oi of a rater Ri with other raters can be quantified as weighted degree centrality of the corresponding vertex Vi in the orthogonality graph. ... C(Vi) = Σ_{Vj∈Ai} O(i, j) = Σ_{j=1, j≠i}^n O(i, j). This is consistent with the definition of the overall orthogonality in Equation (4)."

    Equation (4) defines the overall orthogonality of rater j as o_j = Σ_{k≠j} O(j,k). The proof of Theorem 1 defines weighted degree centrality as exactly the same sum, C(V_i)=Σ_{j≠i}O(i,j), and then states it is consistent with Equation (4). The theorem is therefore a restatement of the definition, not an independent derivation. This is a presentational tautology and does not feed the held-out downstream evaluation.

  2. fitted input called prediction [Section 4.3 Case study / Table 3; Appendix D.5; Section 2.2 Step 3 (Eq. 2); Appendix C.2]
    "We pick 1000 data points at random from the top 20B data that are selected by each method. Then we use GPT-4o to assess each dimension six times, taking the average as the final evaluation result. ... Does this text contain an informative signal for pretraining a large-language model? An informative data point should be well-formatted, contain some usable knowledge of the world, and strictly NOT have any harmful, racist, sexist, etc. content."

    FIRE's integrated rating I(x)=A(x)^T(o⊙γ) is constructed from GPT-4o pairwise comparisons of 'informative signal' (Eq. 2; prompt in C.2). Table 3 then validates FIRE's selected data by asking GPT-4o, with a nearly identical criterion, to confirm that the selections are high quality. Because the selection was optimized to track GPT-4o's informativeness preference, high scores in Table 3 are expected by construction; this is self-confirmation rather than an independent check. The external downstream-task results are unaffected, so the main claim is not circular.

full rationale

FIRE's construction is not circular with respect to its headline claim. The integrated rating I(x)=A(x)^T(o⊙γ) is a deterministic function of the four QuRating rater scores, the fitted win-rate-percentile curves, and GPT-4o pairwise judgments against a random reference; no constant is fitted to the reported downstream accuracies. The main results (Table 1, Figures 1 and 4) come from training 1.3B/3B models on selected SlimPajama subsets and evaluating on held-out ARC, SciQ, LogiQA, BoolQ, HellaSwag, PIQA, and WinoGrande, so the comparison against Random and other selectors is genuinely external. The absence of a baseline that directly trains on GPT-4o's generic informativeness prompt is a real experimental-design gap, since it leaves open the possibility that the gains come from the oracle rather than FIRE's integration, but a missing control is a correctness risk, not circularity. Two minor self-referential artifacts are flagged: Theorem 1 restates the definition of overall orthogonality as weighted-degree centrality, and the Table 3 quality case study reuses GPT-4o (with a nearly identical prompt) as the judge of data selected by a method optimized to GPT-4o's preferences. Neither artifact affects the held-out benchmark claims, so the overall circularity score is low.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The framework relies on GPT-4o as an external judge and on several hand-set hyperparameters (eta, beta, alpha) plus unreported constants (k, |D_r|, spline degree, n_max). No new physical or ontological entities are introduced. The orthogonality functional form is selected by downstream performance, which is a form of fitting to the evaluation target.

free parameters (7)
  • k (quantile interval count) = not reported
    Section 2.2 Step 2 partitions sorted data into k intervals; k sets the resolution of the win-rate-percentile spline but is never given a value.
  • |D_r| (reference subset size) = not reported
    Step 3 requires |D_ij| = |D_r|; the size controls sampling noise in win-rate estimates but is not stated.
  • Spline degree n = not reported
    Eq. (10) defines a polynomial spline of degree n; the actual degree used for the win-rate-percentile function is unspecified.
  • eta (progressive decay factor) = 60%
    Section 4.1 sets eta=60; it controls how much data survives each pruning round and directly affects the final selected subset.
  • beta (part multiplication factor) = 20
    Section 4.1 sets beta=20; Figure 5 ablates beta and picks the best value on the downstream evaluation tasks.
  • n_max (maximum parts) = not reported
    Algorithm 1 takes n_max as input, but no value is reported in the experiments.
  • alpha (PageRank iterations) = 50
    Section 2.3 sets alpha=50, justified by stabilization of the orthogonality vector; the convergence threshold is not quantified.
assumptions (5)
  • standard math Perron-Frobenius theorem and spectral theorem for symmetric matrices are used to prove convergence of the M^alpha orthogonality iteration.
    Appendix B, Theorem 3. Requires M non-negative and irreducible; the paper assumes no perfectly correlated rater pairs remain.
  • domain assumption GPT-4o pairwise comparisons are a reliable proxy for pretraining data quality.
    Section 2.2 Step 3 and Eq. (2): win rates are computed from GPT-4o judgments. Appendix C.3 validates only the Educational Value rater against human experts.
  • domain assumption Uniformly sampled reference subset D_r has the same quality distribution as the full pretraining set D_t.
    Section 2.2 Step 1 states the consistency; the win-rate calibration and comparability across raters depend on it.
  • domain assumption Quality dimensions combine linearly, so a weighted sum of aligned ratings equals overall data quality.
    Eqs. (4)-(9) use linear additive integration. Section 7 Limitations admits this linear assumption may miss complex interactions.
  • ad hoc to paper Orthogonality between raters is a function of Pearson correlation with boundary values 0 (fully correlated) and 0.5 (uncorrelated).
    Appendix C.6 sets these boundary conditions "for convenience and without losing rationality"; the functional form is then selected by downstream performance in Appendix E.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training." pith.science (2026). https://pith.science/paper/2RO4S7UC

@misc{pith2026250200761,
  author       = {Pith},
  title        = {Pith review of: FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RO4S7UC}},
  note         = {Machine review of arXiv:2502.00761}
}
read the original abstract

Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5\% of the training data needed by the Random baseline to reach the target performance.

Figures

Figures reproduced from arXiv: 2502.00761 by the authors.

Figure 1
Figure 1. Downstream accuracy with respect to pretrain [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall framework of FIRE, which contains [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Ablation experiments on the impact of differ [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The in-context learning results with respect to [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: (a) The Pearson correlation between differ [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The win rates of samples in different per [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Trends of 3 forms of orthogonality functions [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Trends of orthogonality functions with the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Ablation experiments evaluating the impact [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The impact of the sample temperature co [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 12 canonical work pages

  1. [2]

    The texts might have similar quality, but you should still make a relative judgement and choose the label of the preferred text

    The order in which the texts are presented Note that the texts are cut off, so you have to infer their contexts. The texts might have similar quality, but you should still make a relative judgement and choose the label of the preferred text. [Option A] ... {text a} ... [Option B] ... {text b} ... Now you have to choose between either A or B. Respond only ...

    work page 2024

  2. [3]

    requires greater expertise and prerequisite knowledge to understand it

    work page

  3. [4]

    has a more polished and beautiful writing style

    work page

  4. [5]

    Prefer specific facts and obscure trivia over more common knowledge

    contains more facts and trivia. Prefer specific facts and obscure trivia over more common knowledge

    work page

  5. [7]

    Aspects that should NOT influence your judgement:

    has more educational value, e.g., it includes clear explanations, step−by−step reasoning, or questions and answers. Aspects that should NOT influence your judgement:

    work page

  6. [10]

    The texts might have similar quality, but you should still make a relative judgement and choose the label of the preferred text

    The order in which the texts are presented Note that the texts are cut off, so you have to infer their contexts. The texts might have similar quality, but you should still make a relative judgement and choose the label of the preferred text. [Option A] {text1} [Option B] {text2} Now you have to choose between either A or B. Respond only with a single word...

    work page 2024

  7. [11]

    Rater training (if applicable),

    work page

  8. [12]

    Rater inference over the entire dataset,

    work page

Show all 13 references
  1. [13]

    QuRating (Single Rater)

    Additional computation for FIRE’s win-rate- based integration. QuRating (Single Rater). Each QuRating model is a 1.3B-parameter transformer. It is first fine- tuned on 500K examples (512 tokens each). The 16 Method FLOPs QuRating (single rater) ≈ 8.17× 1020 QuRating (mix of cr...

  2. [14]

    Which language the text is written in

  3. [15]

    The length of the text

  4. [16]

    quesion":

    The order in which the texts are presented Note that the texts are cut off, so you have to infer their contexts. Here is the text: [TEXT BEGIN] {text} [TEXT END] Please follow the question order to respond. For answer, only respond yes or no. Return the results for each questi...

    2024

  5. [2024]

    arXiv preprint arXiv:2402.09668

    How to train data-efficient llms. arXiv preprint arXiv:2402.09668. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: An adver- sarial winograd schema challenge at scale. Commu- nications of the ACM, 64(9):99–106. Daria Soboleva, Faisal ...

    2021 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.