Pith. sign in

REVIEW 5 major objections 4 minor 33 references

Not All Documents Are What You Need for Extracting Instruction Tuning Data

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EQUAL builds instruction-tuning data from only the useful documents, cutting compute 5-10x while gaining about 2.5% accuracy.

desk verdict EQUAL is a credible framework for cutting the cost of turning web documents into instruction-tuning data, but its headline math gains are not yet trustworthy because AutoMathText was not contamination-filtered against GSM8K even though the selection reward targets GSM8K-train. read the letter →

arxiv 2505.12250 v1 pith:U4TIVLWK submitted 2025-05-18 cs.CL cs.AI

Chi Zhang , Huaping Zhong , Hongtao Li , Chengliang Chai , Jiawei Hong , Yuhao Deng , Jiacheng Wang , Tian Tan
show 6 more authors
Yizhou Yan Jiantao Qiu Ye Yuan Guoren Wang Conghui He Lei Cao
This is my paper · ORCID
classification cs.CLcs.AI
keywords instructiontuningdataextractiondocumentselectionmulti-armedbanditoptimaltransportcontrastivelearningquestion-answerpairswebcorpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that instruction-tuning data can be extracted from large web corpora without paying the cost of reading every document. Its method, EQUAL, clusters documents, samples a small fraction of documents from a chosen cluster, extracts question-answer (QA) pairs from those documents with an LLM, and repeats, using a multi-armed bandit to balance exploiting clusters that already look useful against exploring clusters that have not been sampled. Across math and code corpora with more than a million documents, this cuts computation by 5-10x while improving accuracy by about 2.5% relative to the selection baselines it is compared with. If the claim holds, open web corpora become a practical source of diverse instruction-tuning data rather than a compute sink.

What carries the argument

The machinery is an iterative loop built on three pieces. First, a warm-up stage uses supervised contrastive learning to fine-tune an embedding model so that documents and the QA pairs extracted from them land close together in the same space. Second, the candidate corpus is clustered in that aligned space. Third, each cluster is an arm in a multi-armed bandit: a document-selection score $\mathrm{DS}_j = \widehat{\mathrm{OT}}_j + \alpha \sqrt{2 \ln T_{\mathrm{total}} / T(C_j)}$ combines the estimated optimal transport score $\widehat{\mathrm{OT}}_j$ between the cluster's sampled QA pairs and the reference set with an exploration bonus for under-sampled clusters. The loop extracts QA pairs only from the selected cluster, then updates that cluster's $\widehat{\mathrm{OT}}_j$, so the reward estimate sharpens as more pairs are seen.

What would settle it

Take the cluster that EQUAL scores highest and the cluster it scores lowest on the same corpus, fine-tune the base model separately on QA pairs extracted from each with equal pair counts and equal extraction budgets, and compare accuracy on GSM8K/MATH or HumanEval/MBPP; if the low-scoring cluster performs as well as the high-scoring one, the optimal-transport reward is not carrying the claimed signal.

Watch

Extended reading notes

Core claim

The paper's central claim is that the document is the wrong unit of selection for instruction-tuning data, and so is the individual QA pair; the right unit is a cluster of documents whose extractable QA pairs resemble the distribution of a reference set such as the GSM8K or MBPP training split. EQUAL operationalizes this by aligning document and QA-pair embeddings with a contrastive-learning warm-up, clustering the candidate corpus, and treating each cluster as an arm of a multi-armed bandit whose reward is the optimal transport distance between sampled QA pairs and the reference distribution. Each round selects a cluster, extracts a batch of QA pairs, and updates that cluster's reward estimate, so document selection and data extraction reinforce each other. The reported experiments on AutoMathText and StackOverflow, with over a million documents each, show that using 5% of documents through this loop beats exhaustive all-document extraction on three of four math and code benchmarks and beats the evaluated selection baselines on all of them, at roughly 5-10x lower FLOPs; at 20% of documents it matches or exceeds exhaustive extraction on all four.

Load-bearing premise

The loop assumes that the optimal transport distance between embeddings of extracted QA pairs and the reference training set (GSM8K or MBPP) reliably predicts how much those pairs improve the target benchmark, and it assumes that the math corpus is not already contaminated with near-duplicates of the reference questions, since AutoMathText is not n-gram filtered while StackOverflow is.

Editorial extensions

If this is right

  • A practitioner can skip exhaustive LLM extraction over an entire web corpus; iteratively sampling a few percent of documents yields comparable or better downstream accuracy on math and code benchmarks.
  • Corpora too large for full extraction, such as general web crawls, become feasible instruction-tuning sources instead of being filtered out by budget.
  • Data selection no longer requires computing influence functions or perplexity over every extracted pair; the cost moves to a cheap embedding-space reward computed on sampled clusters.
  • Because the reward is distributional, the method transfers to a new downstream task whenever a small reference set of questions exists, with experiments covering both mathematical and code domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because the machinery needs only an embedding space and a reference set, the same interleaving should apply to expensive non-LLM extractors, such as multimodal captioning pipelines, where the cost per extracted item is even higher.
  • The paper does not test how the optimal transport reward behaves with very small reference sets; a natural extension is to replace the GSM8K and MBPP training splits with a few dozen seed questions and measure whether selection accuracy degrades.
  • Inference: the optimal transport estimate accumulated over rounds could itself serve as a training label for a cheap classifier, potentially removing the need for sampled extraction during selection in later rounds.
  • Inference: because exploration is explicit, EQUAL may be more robust in streaming settings where new document clusters arrive over time and the bandit must keep sampling unknown regions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. This paper addresses the problem of constructing instruction-tuning QA pairs from large document corpora. EQUAL first warms up an embedding model with contrastive learning on sampled documents and their extracted QA pairs, clusters the candidate corpus, and then iteratively selects clusters with a multi-armed bandit whose reward is the optimal-transport distance between the embeddings of extracted QA pairs and a reference set Dr (GSM8K-train or MBPP-train). Extracted pairs are pooled for fine-tuning. Experiments on AutoMathText and StackOverflow with LLaMA-3-8B and Mistral-7B, in full and LoRA settings, compare against random selection, extraction-from-all, rewriting, perplexity- and influence-based selection, and MAB variants, reporting accuracy on GSM8K/MATH and HumanEval/MBPP plus FLOPs. The paper claims a 5-10x cost reduction and about 2.5% accuracy improvement.

Significance. If the claims hold, EQUAL would be a practical contribution: interleaving document selection and QA extraction is a sensible way to avoid the cost of extracting all pairs, and the contrastive warm-up addresses a real distribution mismatch between documents and QA pairs. Strengths include a clear problem formulation, a complete algorithmic description, ablations for each component (CL, MAB, OT), experiments across two domains and four benchmarks, and a release of code and data. However, the central empirical claims are currently under-supported by the measurement and contamination issues detailed below; the claimed gains are plausible but not yet established.

major comments (5)
  1. [Section 4.1, Eq. (4)] Section 4.1 states that StackOverflow is n-gram filtered 'to ensure that our training data is not contaminated by information from the downstream tasks,' but no analogous statement is made for AutoMathText. Because Dr for math is GSM8K-train and the MAB reward is OT distance to Dr (Eq. (4)), any near-duplicate of GSM8K questions in AutoMathText would be precisely the content EQUAL tends to select, so the GSM8K gains in Table 1 and the OT ablations in Table 3 could reflect memorization rather than a learned property of useful instruction data. Please provide overlap analysis (n-gram and/or embedding-level) between AutoMathText documents / extracted pairs and GSM8K train+test and MATH, and rerun the math experiments after removing flagged near-duplicates.
  2. [Appendix B, Table 1] The FLOPs measurement in Appendix B uses torch.profiler on a dummy nn.Linear(1024, 512) with random input, which does not measure the FLOPs of the actual pipeline (Qwen2.5-72B extraction calls, embedding inference, clustering, MAB scoring, and fine-tuning). Since the headline '5-10x reduction' is computed from these numbers, this is load-bearing. Please report the actual profiled operations or state the estimation protocol explicitly; also report FLOPs for the All(Mammoth) baseline against which the 5-10x factor is claimed.
  3. [Algorithm 1, Section 4.2] Algorithm 1's stopping condition is |De| < gamma |Dc|, where |De| counts extracted QA pairs and |Dc| counts documents. Section 4.2 alternately says 'we acquire 5% documents' and 'extracting just 5% of QA pairs.' If each document yields multiple QA pairs, the two interpretations differ by a large factor, and the comparison with a 'Random(5%)' baseline that samples 5% of documents is not apples-to-apples. Please clarify the stopping rule and report both the fraction of documents processed and the number of QA pairs extracted.
  4. [Table 1 and Table 2] Table 1 reports averages over three runs without error bars, while Table 2 lists Random(5%) Full GSM8K as 67.40 and Table 1 lists Random FULL GSM8K as 68.92 for what appears to be the same setting. The claimed ~2.5% accuracy improvement is of the same magnitude as typical run-to-run variance; please provide standard deviations or confidence intervals and reconcile the two tables.
  5. [Section 3.3, Section 4.1] The OT reward is optimized against Dr, which is the training split of the evaluation benchmark (GSM8K-train for math, MBPP-train for code), while the reported accuracies are on GSM8K/MATH and HumanEval/MBPP test splits. This makes GSM8K-test performance partly a measure of alignment with the selection target rather than of general instruction-data utility. Please add a transfer experiment with a reference set disjoint from all evaluation benchmarks, or evaluate on an additional benchmark not used in Dr, to support the general claim that OT-to-Dr predicts downstream accuracy.
minor comments (4)
  1. [Abstract, Section 4.1] The model name is inconsistent: the abstract and Section 4.1 refer to LLaMA-3.1-8B while Table 1 and parts of Section 4.1 say LLAMA-3-8B; please standardize the name.
  2. [Figure 3] Figure 3 would benefit from axis labels and a legend; as printed, it is hard to verify the claimed cluster alignment from the two panels.
  3. [Table 2] Table 2 compares EQUAL and Random at multiple ratios but omits the FLOPs column, making it impossible to see the cost of the All(Mammoth) baseline in the same table; please include FLOPs for the All baseline in Table 1 as well.
  4. [Section 4.1, Figure 4] Section 4.1 says the number of clusters is 'automatically determined by the Elbow method' and later reports k=1000 as the selected value; please state whether k=1000 is the outcome of the method for both datasets or a manually fixed value.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the OT-to-Dr reward is a training-split optimization target, while reported gains are held-out test accuracies, so no prediction reduces to the method's inputs by construction.

full rationale

The derivation chain is self-contained for the purposes of circularity analysis. EQUAL's selection reward is the optimal transport distance between extracted QA-pair embeddings and the reference set Dr (Eq. 4), updated as OT(∪Q_i, Dr) in Eq. 2, and Dr is explicitly the training split of GSM8K and MBPP (Section 4.1). This is an optimization target, not the reported quantity: the headline numbers in Table 1 are held-out test accuracies on GSM8K, MATH, HumanEval, and MBPP, so the metric is not algebraically equal to the reward. No fitted parameter is renamed as a prediction; hyperparameters such as α, the number of clusters, and the extraction ratio are either standard choices, ablations, or stated inputs. No load-bearing claim rests on a self-citation: the components cited for UCB, optimal transport, contrastive learning, and Mammoth extraction are external methods or benchmark definitions, and the paper does not invoke a uniqueness theorem or an ansatz from the authors' own prior work. The strongest concern is data contamination risk rather than circularity: Section 4.1 n-gram filters StackOverflow 'to ensure that our training data is not contaminated by information from the downstream tasks,' but applies no analogous filter to AutoMathText while GSM8K-train is the OT reference; this could inflate math gains if AutoMathText contains near-duplicates of benchmark items. That is an external-validity confound, not a circular reduction, so it does not raise the circularity score under the stated criteria.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or theoretical entities; its load-bearing assumptions are about the representativeness of warm-up samples, the validity of OT as a utility proxy, and the absence of benchmark contamination in the math corpus.

free parameters (4)
  • Cluster count k = 1000 (selected by Elbow/WCSS)
    Documents are clustered into 1,000 clusters, and Figure 4(a) shows accuracy varies by about 2-3% with this choice.
  • Warm-up sampling ratio = 5% of candidate documents
    Contrastive fine-tuning of the embedding model uses 5% of Dc; no sensitivity analysis is reported for this ratio.
  • Extracted data ratio gamma = 5% (main), 10%, 20% (ablation)
    Algorithm 1 stops when extracted pairs reach gamma times |Dc|; gamma is a user-set budget, not derived.
  • UCB exploration coefficient alpha = 1/(sum_c T(C_c)+1)
    Equation 1 uses a time-varying alpha instead of a constant UCB coefficient; it is adopted from Hao et al. 2019, not fitted to downstream accuracy.
assumptions (4)
  • domain assumption Optimal transport distance in the embedding space between extracted QA pairs and the reference set Dr is a reliable proxy for how much those pairs improve fine-tuning on the target tasks.
    Section 3.3 defines OT as the reward; the paper validates indirectly through final accuracy, but never shows that OT rankings correlate with per-cluster utility.
  • domain assumption Contrastive fine-tuning on a 5% warm-up sample of documents transfers document-QA alignment to the full corpus.
    Section 3.2 assumes the sampled documents and extracted QA pairs are representative enough to align the two feature spaces globally.
  • domain assumption The math candidate pool AutoMathText is not contaminated with GSM8K benchmark content.
    The paper applies n-gram contamination filtering only to StackOverflow (Section 4.1), not to AutoMathText, while using GSM8K training as Dr and GSM8K test as an evaluation task.
  • domain assumption Clustering with k-means on aligned embeddings groups documents whose extractable QA pairs are similar.
    Sections 3.1 and 3.2 rely on this to make cluster-level sampling meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Not All Documents Are What You Need for Extracting Instruction Tuning Data." pith.science (2026). https://pith.science/paper/U4TIVLWK

@misc{pith2026250512250,
  author       = {Pith},
  title        = {Pith review of: Not All Documents Are What You Need for Extracting Instruction Tuning Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4TIVLWK}},
  note         = {Machine review of arXiv:2505.12250}
}
read the original abstract

Instruction tuning improves the performance of large language models (LLMs), but it heavily relies on high-quality training data. Recently, LLMs have been used to synthesize instruction data using seed question-answer (QA) pairs. However, these synthesized instructions often lack diversity and tend to be similar to the input seeds, limiting their applicability in real-world scenarios. To address this, we propose extracting instruction tuning data from web corpora that contain rich and diverse knowledge. A naive solution is to retrieve domain-specific documents and extract all QA pairs from them, but this faces two key challenges: (1) extracting all QA pairs using LLMs is prohibitively expensive, and (2) many extracted QA pairs may be irrelevant to the downstream tasks, potentially degrading model performance. To tackle these issues, we introduce EQUAL, an effective and scalable data extraction framework that iteratively alternates between document selection and high-quality QA pair extraction to enhance instruction tuning. EQUAL first clusters the document corpus based on embeddings derived from contrastive learning, then uses a multi-armed bandit strategy to efficiently identify clusters that are likely to contain valuable QA pairs. This iterative approach significantly reduces computational cost while boosting model performance. Experiments on AutoMathText and StackOverflow across four downstream tasks show that EQUAL reduces computational costs by 5-10x and improves accuracy by 2.5 percent on LLaMA-3.1-8B and Mistral-7B

Figures

Figures reproduced from arXiv: 2505.12250 by the authors.

Figure 1
Figure 1. Comparison of Different Solutions. instruction tuning dataset, it uses LLMs to repeatedly scan and analyze all the documents to extract question-answer (QA) pairs, each of which often requires multiple LLM calls (Gilardi et al., 2023; Yue et al., 2024). Consequently, this process is prohibitively expensive, especially when there are a large number of documents to process. Solving this problem requires largely reduci… view at source ↗
Figure 2
Figure 2. The Overall Framework of EQUAL. DSj = OTˆ j + α s 2 lnP Ck∈C T(Ck) T(Cj ) (1) where T(Cj ) denotes the frequency of documents sampled from cluster Cj , P Ck∈C T(Ck) denotes the total sampling times from all clusters. α is set as P 1 Ck∈C T(Ck)+1 (Hao et al., 2019), which provides higher weight to exploration in early stages, but in later stages, it provides higher weight to exploitation (line 9-11). Update the DS Sc… view at source ↗
Figure 3
Figure 3. (a) shows the clusters based on the original embedding model; (b) shows the clusters based on the fine-tuned embedding model. contain many contents irrelevant to the extracted QA pairs, leveraging the document embeddings directly to cluster will lead to the inconsistencies of QA pair embeddings within each cluster. It can be observed that CL effectively aligns the embeddings of documents and their corresponding QA p… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study of cluster numbers and clustering algo￾rithms. Number of Clusters. We use the Elbow (Syakur et al., 2018) method to identify the optimal cluster numbers for AutoMathText and StackOverFlow datasets. In [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 3 canonical work pages

  1. [1]

    Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Mor- cos, A. S. Semdedup: Data-efficient learning at web- scale through semantic deduplication.arXiv preprint arXiv:2303.09540,

    arXiv

  2. [3]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    arXiv

  3. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  4. [7]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021a. Cobbe, K., Kosaraju, V ., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems.Co...

    arXiv

  5. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  6. [10]

    Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Study- ing large language model generalization with influence functions.arXiv preprint arXiv:2308.03296,

    arXiv

  7. [11]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196, 2024a

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y ., Li, Y ., et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196, 2024a. Guo, H., Yao, Y ., Shen, W., Wei, J., Zhang, X., Wang, Z., and Liu, Y . Human-instruction-free llm self-alignment with lim...

    arXiv

  8. [13]

    Unnat- ural instructions: Tuning language models with (almost) no human labor.arXiv preprint arXiv:2212.09689,

    9 Not All Documents Are What You Need for Extracting Instruction Tuning Data Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnat- ural instructions: Tuning language models with (almost) no human labor.arXiv preprint arXiv:2212.09689,

    arXiv

Show all 33 references
  1. [14]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000

  2. [16]

    Rho-1: Not all tokens are what you need.arXiv preprint arXiv:2404.07965,

    Lin, Z., Gou, Z., Gong, Y ., Liu, X., Shen, Y ., Xu, R., Lin, C., Yang, Y ., Jiao, J., Duan, N., et al. Rho-1: Not all tokens are what you need.arXiv preprint arXiv:2404.07965,

    arXiv

  3. [17]

    Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583,

    Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583,

    arXiv

  4. [18]

    When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    Marion, M., ¨Ust¨un, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining llms at scale.arXiv preprint arXiv:2309.04564,

    arXiv

  5. [20]

    Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543,

    2014

  6. [22]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y ., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,

    arXiv

  7. [23]

    Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

    Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y ., et al. Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,

    arXiv

  8. [25]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244,

    arXiv

  9. [26]

    Xu, Z., Jiang, F., Niu, L., Deng, Y ., Poovendran, R., Choi, Y ., and Lin, B. Y . Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464,

    arXiv

  10. [27]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024a. Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., and Liu, Q. Self-distillation bridges distributio...

    arXiv

  11. [28]

    T., Li, Z., Weller, A., and Liu, W

    Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y ., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284,

    arXiv

  12. [29]

    J., Kr- ishna, R., Shen, J., and Zhang, C

    Yu, Y ., Zhuang, Y ., Zhang, J., Meng, Y ., Ratner, A. J., Kr- ishna, R., Shen, J., and Zhang, C. Large language model as attributed training data generator: A tale of diversity and bias.Advances in Neural Information Processing Systems, 36, 2024a. Yu, Z., Das, S., and Xiong, ...

    arXiv

  13. [31]

    Zhang, Y ., Luo, Y ., Yuan, Y ., and Yao, A. C.-C. Automath- text: Autonomous data selection with language models for mathematical texts.arXiv preprint arXiv:2402.07625,

    arXiv

  14. [32]

    Considering that the clustering results are easily affected by the parameters of clustering algorithms, we use different methods to select proper parameters

    and DBSCAN (Ester et al., 1996). Considering that the clustering results are easily affected by the parameters of clustering algorithms, we use different methods to select proper parameters. For DBSCAN, there are 2 key parameters: (1)eps(the radius of a neighborhood w.r.t. som...

    1996

  15. [33]

    to determine the appropriate number of components. B. FLOPs Calculation FLOPs is the number of floating point operations performed by GPUs. Many state-of-the-art methods [1,2,3] use it to measure the consumption of GPU computing resources. In our experiments, FLOPs is collecte...

    2013

  16. [1996]

    and Yang, Q

    Zhang, X. and Yang, Q. Self-qa: Unsupervised knowl- edge guided language model alignment.arXiv preprint arXiv:2305.11952,

    arXiv

  17. [2000]

    Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., Huang, S., Huang, X., Huang, Z., Zhang, D., et al

    Morgan Kaufmann. Li, H., Dong, Q., Tang, Z., Wang, C., Zhang, X., Huang, H., Huang, S., Huang, X., Huang, Z., Zhang, D., et al. Synthetic data (almost) from scratch: Generalized in- struction tuning for language models.arXiv preprint arXiv:2402.13064, 2024a. Li, M., Zhang, Y ....

    arXiv

  18. [2009]

    Step-on-feet tuning: Scaling self-alignment of llms via bootstrapping

    Wang, H., Ma, G., Meng, Z., Qin, Z., Shen, L., Zhang, Z., Wu, B., Liu, L., Bian, Y ., Xu, T., et al. Step-on-feet tuning: Scaling self-alignment of llms via bootstrapping. arXiv preprint arXiv:2402.07610,

    arXiv

  19. [2017]

    R., Mozer, M., Bengio, Y ., Arora, S., et al

    Shah, V ., Yu, D., Lyu, K., Park, S., Yu, J., He, Y ., Ke, N. R., Mozer, M., Bengio, Y ., Arora, S., et al. Ai-assisted generation of difficult math questions.arXiv preprint arXiv:2407.21009,

    arXiv

  20. [2019]

    Measuring math- ematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

  21. [2020]

    Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701,

    Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V ., Tang, Z., Srinivasan, V ., Zhou, T., Huang, H., et al. Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701,

    arXiv

  22. [2021]

    Brown, T. B. Language models are few-shot learners.arXiv preprint arXiv:2005.14165,

    2005 arXiv

  23. [2022]

    The refinedweb dataset for falcon llm: out- performing curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116,

    Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cap- pelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon llm: out- performing curated corpora with web data, and web data only.arXiv preprint arXiv:2306.01116,

    arXiv

  24. [2023]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  25. [2024]

    and Durrett, G

    Divekar, A. and Durrett, G. Synthesizrr: Generating di- verse datasets with retrieval augmentation.arXiv preprint arXiv:2405.10040,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.