REVIEW 1 major objections 2 minor 5 references
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
T0 review · 1 major / 2 minor · reviewed 2026-06-29 · grok-4.3
Pith's one-line read MIRA discovers source-specific rubrics then distills them into student scorers to filter mid-training data, matching full-corpus results on code benchmarks while using half the tokens.
desk verdict MIRA's rubric discovery plus distillation approach for source-aware mid-training data selection looks practically useful on code data but the abstract supplies no experimental details to back the performance claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
self-anchored rubric discovery: the process of first determining source-group-specific evaluation criteria and then distilling those criteria into scalable student scorers for corpus-wide filtering.
What would settle it
Running the distilled scorers on the same 21 sources but with a different downstream task family (for example, math rather than code) and observing that the selected half-corpus no longer matches full-corpus performance on the new tasks would falsify the central claim.
Extended reading notes
Core claim
MIRA is a source-aware filtering framework that performs self-anchored rubric discovery: for each of five source groups drawn from 21 code sources, it identifies the evaluation criteria that matter for that group, then distills those judgments into student scorers that label the full corpus at scale. In code-oriented mid-training experiments this procedure yields a filtered mixture that outperforms prior selection baselines across nine benchmarks while matching the full-corpus run with only half the tokens.
Load-bearing premise
The rubrics found for each source group capture stable quality signals that survive distillation into student scorers and continue to work for the full corpus.
Editorial extensions
If this is right
- Mid-training mixtures can be reduced to half their original token count without loss of downstream code performance.
- Source-group rubrics allow semantic filtering to scale to heterogeneous data without assuming fixed evaluation criteria.
- Distilled student scorers provide explicit quality signals that outperform both purely model-based and fixed-rubric baselines on nine code benchmarks.
- The same two-stage discovery-plus-distillation pipeline can be applied whenever mid-training data come from multiple sources with distinct formats and roles.
Reading between the lines
- The method could be tested on non-code domains by repeating the rubric-discovery step on math or general-text sources to check whether the same token-reduction benefit appears.
- If the distilled scorers remain effective after further compression, the approach might reduce the teacher-model cost of data selection itself.
- Applying the same anchoring step at the pretraining stage rather than only mid-training would test whether source-aware rubrics help earlier in the pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MIRA, a source-aware data selection framework for LLM mid-training. It performs self-anchored rubric discovery per source group to define evaluation criteria, then distills these rubrics into scalable student scorers for corpus-wide filtering. On a code mid-training setup using 21 sources in 5 groups, MIRA is reported to outperform selection baselines across nine code benchmarks while matching full-corpus performance using only half the tokens.
Significance. If the empirical results hold under rigorous controls, MIRA would address a practical gap in mid-training curation by combining scalable model-based filtering with source-adaptive semantic criteria, enabling more efficient data use without sacrificing downstream capability gains.
major comments (1)
- [Abstract] Abstract: the central performance claims (outperformance on nine benchmarks, parity with full corpus at half tokens) are stated without any description of the experimental setup, baselines, controls, statistical tests, or variance estimates, rendering the primary result unevaluable from the provided text.
minor comments (2)
- Clarify the exact definitions of the 5 source groups and the 21 sources, including any formatting or role differences that motivate the source-aware approach.
- Provide the list of the nine code benchmarks and the specific selection baselines against which MIRA is compared.
Simulated Author's Rebuttal
We thank the referee for the detailed review. The single major comment concerns the level of detail in the abstract; we address it directly below.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central performance claims (outperformance on nine benchmarks, parity with full corpus at half tokens) are stated without any description of the experimental setup, baselines, controls, statistical tests, or variance estimates, rendering the primary result unevaluable from the provided text.
Authors: We agree that the abstract is written at a high level and omits explicit mention of the experimental setup (21 sources in 5 groups, code mid-training), the specific baselines, controls, or variance reporting. This is a deliberate choice to keep the abstract under typical length limits while still conveying the core contribution. All of those details appear in Sections 3 (method) and 4 (experiments), including the nine code benchmarks, token budgets, and comparison to the full-corpus baseline. We are happy to revise the abstract to add one concise sentence referencing the setup and the fact that results are averaged over multiple runs if the editor prefers. revision: partial
Circularity Check
No significant circularity detected
full rationale
The paper describes an empirical data-selection framework (rubric discovery per source group followed by distillation into student scorers) evaluated on code benchmarks. No equations, fitted-parameter predictions, or first-principles derivations appear. Claims rest on external benchmark comparisons rather than any internal reduction of outputs to inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatz smuggling are present in the provided text. The derivation chain is therefore self-contained as a practical method without circularity.
Assumptions & free parameters
Cite this review
Pith. "Pith review of MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection." pith.science (2026). https://pith.science/paper/5I4WSYJS
@misc{pith2026260530288,
author = {Pith},
title = {Pith review of: MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/5I4WSYJS}},
note = {Machine review of arXiv:2605.30288}
}
read the original abstract
Mid-training has become an important stage in modern LLM development, using large-scale curated mixtures to strengthen capabilities before final post-training. Its data selection problem is distinct: the data are optimized under a pretraining-style objective at near-pretraining scale, but are curated toward downstream capabilities and drawn from heterogeneous sources with different formats and training roles. As a result, effective selection requires both scalability and source-adaptive semantic criteria. Existing model-based methods scale well, but provide only implicit quality signals. Semantic selection methods offer stronger judgments, but usually assume fixed rubrics or standardized data formats. To address this mismatch, we propose MIRA, a source-aware filtering framework based on self-anchored rubric discovery. The key idea is to make rubric construction part of data selection: MIRA first discovers what should be evaluated for each source group, then distills those judgments into scalable student scorers for full-corpus filtering. On code-oriented mid-training with 21 sources and 5 source groups, MIRA outperforms selection baselines across nine code benchmarks and matches the full-corpus run while using only half the tokens.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Dolma: an open corpus of three trillion to- kens for language model pretraining research.arXiv preprint. FAIR CodeGen team, Jade Copet, Quentin Carbon- neaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jan- nik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, David Zhang, Kunhao Zheng, Jordi Armengol-Estapé, Pedram Bashiri, Maximilian Beck, Pierr...
-
[2]
ADEPT: Continual pretraining via adaptive expansion and dynamic decoupled tuning.arXiv preprint. Wanru Zhao, Yaxin Du, Nicholas Donald Lane, Siheng Chen, and Yanfeng Wang. 2024. Enhancing data quality in federated fine-tuning of foundation models. arXiv preprint arXiv:2403.04529. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widy...
-
[3]
Name it (specific, not vague)
-
[4]
Give 0-10 integer score
-
[5]
One-sentence justification Format strictly: [A1] <Dimension Name>: <score>/10 -- <justification> [A2] <Dimension Name>: <score>/10 -- <justification> ... [A15] <Dimension Name>: <score>/10 -- <justification> ## Part B -- Summary [Overall Score]: <average of A1-A15, rounded to 1 decimal>/10 [Training Recommendation]: <untrainable | not_trainable | passable...
Reviewed June 29, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.